libclang.py generator failed

[lpraneis]

Describe the bug

When using doge on C++ comments, message about libclang.py failed. I have libclang.so in a symlink as described in the README.md file, and am able to use it in python, as shown below. Using neovim version 0.5.0 with python support. The comment generation worked fine before #56

>>> from clang.cindex import Index
>>> Index.create()
<clang.cindex.Index object at 0x7fb9fb551f98>

On a generation, the following is opened in a buffer:

[DoGe] libclang.py generator failed.
[DoGe] libclang.py generator failed.
[DoGe] libclang.py generator failed.
Press ENTER or type command to continue

Settings

  let g:doge_doc_standard_cpp='doxygen_javadoc'

Out of vim --version

NVIM v0.5.0-dev
Build type: RelWithDebInfo
Lua 5.1
Compilation: /usr/bin/cc -g -O2 -fdebug-prefix-map=/build/neovim-CyMQsA/neovim-0.5.0+ubuntu1+git201911142143-e3b08a0-00e710e=. -fstack-protector-strong -Wformat -Werror=format-security -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -O2 -g -DMIN_LOG_LEVEL=3 -Og -g -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wshadow -Wconversion -Wmissing-prototypes -Wimplicit-fallthrough -Wvla -fstack-protector-strong -fdiagnostics-color=auto -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FROM -I/build/neovim-CyMQsA/neovim-0.5.0+ubuntu1+git201911142143-e3b08a0-00e710e/build/config -I/build/neovim-CyMQsA/neovim-0.5.0+ubuntu1+git201911142143-e3b08a0-00e710e/src -I/build/neovim-CyMQsA/neovim-0.5.0+ubuntu1+git201911142143-e3b08a0-00e710e/.deps/usr/include -I/usr/include -I/build/neovim-CyMQsA/neovim-0.5.0+ubuntu1+git201911142143-e3b08a0-00e710e/build/src/nvim/auto -I/build/neovim-CyMQsA/neovim-0.5.0+ubuntu1+git201911142143-e3b08a0-00e710e/build/include
Compiled by buildd@lcy01-amd64-004

Features: +acl +iconv +tui

[kkoomen]

The comment generation worked fine before #56

That's logical, because before #56 it tried to use pure regex, but since v1.15.0 it is using clang.

---

To be honest, it's not up to me whether your local environment for C++ works or not, so you'd better ask in an IRC chat or anywhere else, but I'm willing to help you out as far as I can. I'm not a C++ dev, but have knowledge about it to some extend.

To start with, you're using a dev version and I won't test against versions that are not alpha, beta, dev etc. The latest version that NeoVim provides is v4.3. I upgraded the docker versions for Vim to v8.1.2300 and for NeoVim to 4.3 and tests are working as expected.

On a generation, the following is opened in a buffer:
[DoGe] libclang.py generator failed.
[DoGe] libclang.py generator failed.
[DoGe] libclang.py generator failed.
Press ENTER or type command to continue

Mhm, that's weird as well, because if the output is anything else then json it will log the errors. Is this from 1 comment generation? It's weird as well that is says libclang.py generator failed. 3 times which means it was generated 3 times in a row, while it should only generate it once.

You can do some print() statements in generators/libclang.py to see where it fails. For now I can't help you unless you are able to get the error output from the libclang.py.

[mlyapin]

I have the same problem with some functions, where the return type includes the enum or struct keywords.
In some cases, the node.kind is not FUNCTION_DECL, as it should be, but ENUM_DECL or STRUCT_DECL.
For example, I have the following output for a broken function if I insert print(node.kind) at generators/libclang.py:161

[DoGe] libclang.py generator failed.
CursorKind.ENUM_DECL
[DoGe] libclang.py generator failed.
CursorKind.ENUM_DECL
[DoGe] libclang.py generator failed.
CursorKind.ENUM_DECL

The problem seems to be that when clang “sees” enum foo or struct foo for the first time, it assumes that it is declaration of a structure or enumeration.
So this is broken only for the first function with these types, and if these types were declared in some header.
For now, it’s possible to put a declaration of the struct or the enum before the function and it works as expected.

You can try it by yourself with the following code:

// Suppose this header contains the definition of enum Foo
// For example: enum Foo{bar};
#include "fooheader.h"

// Will not work
// node.kind == CursorKind.ENUM_DECL
enum Foo foofunc();

// Will work
// node.kind == CursorKind.FUNCTION_DECL
enum Foo barfunc();

(Sorry if my English is a little bit broken :) )

[kkoomen]

@jbqxo I understand your problem. Is it possible for the enum Foo foofunc(); to also have parameters? For example: enum Foo foofunc(int x, int y);.

Another question is: should this be fixed from my side or this something that you should fix locally? It uses clang and clang requires valid syntax in order to parse it correctly, so I guess this isn't something I can fix from my side.

I can parse ENUM_DECL if needed, but then I like to hear from you the input and the expected generated output with some example scenarios. Then I'll work on it right away.

[mlyapin]

Yes, it works fine if these parameters do not contain structures, enumerations etc. which were not defined, nor declared in the current file:

// Suppose this header contains the definition of enum Foo
// For example: enum Foo{bar};
#include "fooheader.h"

// The output is:
// [DoGe] libclang.py generator failed.
// CursorKind.ENUM_DECL
// [DoGe] libclang.py generator failed.
// CursorKind.ENUM_DECL
// [DoGe] libclang.py generator failed.
// CursorKind.ENUM_DECL
enum Foo foofunc(int x, int y);


// The output is:
// [DoGe] libclang.py generator failed.
// CursorKind.FUNCTION_DECL
// {"name": "barfunc", "returnType": "enum Foo", "parameters": [{"param-type": "param", "name": "x"}, {"param-type": "param", "name": "y"}]}
// [DoGe] libclang.py generator failed.
// CursorKind.FUNCTION_DECL
// [DoGe] libclang.py generator failed.
// CursorKind.FUNCTION_DECL
enum Foo barfunc(int x, int y);

To be honest, I do not see anything that I can do from my side. Only to type in all the declarations necessary for the function before it:

// This declaration of enumeration will fix it.
enum Foo;

// Will work
// node.kind == CursorKind.FUNCTION_DECL
enum Foo foofunc();

// Will work
// node.kind == CursorKind.FUNCTION_DECL
enum Foo barfunc();

But this is tedious for functions that will take several of these arguments.

The problem is that the syntax is valid. This is just an assumption, but during compilation compiler has all the knowledge needed for the preprocessor to set things up.
But when you give to clang contents of a vim buffer, it has no way to do the same.

I don’t think you can simply consider {ENUM, STRUCT, ...}_DECL as function declarations. Please take a look at this example:

// Suppose this header contains the following code:
// typedef enum {Foo} FooEnum;
#include "fooheader.h"

// Then the result will be:
// [DoGe] libclang.py generator failed.
// CursorKind.FUNCTION_DECL
// {“name": "foofunc", "returnType": "int", "parameters": []}
// [DoGe] libclang.py generator failed.
// CursorKind.FUNCTION_DECL
// [DoGe] libclang.py generator failed.
// CursorKind.FUNCTION_DECL
FooEnum foofunc(int x, int y);

// This time it will not work. The output will be the same as for FooEnum foofunc(int x, int y)
FooEnum barfunc(int x, int y);

Note that function arguments have been ignored.

There is no way to “ask” clang to look for headers in given directories?
Thus, you could parse some compile_commands.json and give to clang everything it needs to work.

[kkoomen]

There is no way to “ask” clang to look for headers in given directories?

Mhm, maybe I did something that is not that good. In the libclang.py file I am grabbing the contents of the current vim buffer, then I make a temporarily file (which on MacOS will result in something like/var/folders/sr/dwq02b5x0sj3s_bwn8rgkg100000gn/T/tmp7qeutnbi.cpp or on linux distributions on /tmp/myfile.cpp), but because this file is outside of the project root, clang is probably not able to find the headers. Am I right about this?

If that is the case, then I will adjust the functionality so that it won't create a temporarily file outside of the project. I will then create a temporarily file in the same folder where your current buffer is located at (which would make more sense to me already now that I'm writing this).

[kkoomen]

If that is the case, then I will adjust the functionality so that it won't create a temporarily file outside of the project. I will then create a temporarily file in the same folder where your current buffer is located at (which would make more sense to me already now that I'm writing this).

@jbqxo I do confirm that the above thing I mentioned it fixing it. If I run your code with the current master branch code, it gives me this (same as you mentioned):

Input:

// Suppose this header contains the following code:
// typedef enum {Foo} FooEnum;
#include "fooheader.h"

FooEnum foofunc(int x, int y);

output:

// Suppose this header contains the following code:
// typedef enum {Foo} FooEnum;
#include "fooheader.h"

/**
 * @brief [TODO:description]
 *
 * @return [TODO:description]
 */
FooEnum foofunc(int x, int y);

and if I adjust libclang.py create the temporary file in the same directory as the vim buffer, I do get:

// Suppose this header contains the following code:
// typedef enum {Foo} FooEnum;
#include "fooheader.h"

/**
 * @brief [TODO:description]
 *
 * @param x [TODO:description]
 * @param y [TODO:description]
 * @return [TODO:description]
 */
FooEnum foofunc(int x, int y);

I will push this to master immediately, since this already proved the me that this should be fixed.

[kkoomen]

@jbqxo Can you pull locally to the latest version and test it out? This seems like a good fix.

[mlyapin]

Can confirm that it helped on provided example, but on the project I'm working on now, it gives the same errors (but only 2) and irrelevant results.
Well. I can reproduce it with a simple example, but not quite sure what is going on:

// test.h
enum VeryVeryUniqueEnumName {bar};

// test.c
#include <stdio.h>
#include "test.h"

/**
 * struct _IO_FILE - [TODO:description]
 */
enum VeryVeryUniqueEnumName foofunc(int x, int y);

Output:

[DoGe] libclang.py generator failed.
CursorKind.STRUCT_DECL
[DoGe] libclang.py generator failed.
CursorKind.STRUCT_DECL
{"name": "_IO_FILE", "parameters": []}
[DoGe] libclang.py generator failed.
CursorKind.STRUCT_DECL

I have no idea why.

But about your solution:
I do not think that this will be enough. The problem is that to use a dependency with C/C++, you must specify the paths where a preprocessor can find its headers.
So compiler usually invoked with some flags like:

-I../.deps/usr/include/luajit-2.1 -Isrc/nvim/auto

With these flags, the compiler will also look for unknown headers in these paths.
So, I suspect that if some function will contain a type from these headers, it will not work again.

The function clang.cindex.Index.parse(...) accepts a list of command-line arguments. I suspect that you could give it all the necessary arguments, and it will work as expected.
But there is a problem with figuring out these parameters, as there is no general way, and, unfortunately, it will depend on a user.

And thank you for your time and patience :)

[mlyapin]

About the last part of my previous message.
Consider the following quick change at generators/libclang.py:158

tu = index.parse(filename, args=['-I', '/tmp/tc'])

Then it works for this code:

// In file /tmp/tc/test.h
enum Foo {bar};

// In file /tmp/t/test.c
#include "test.h"

  /**
   * @brief [TODO:description]
   *
   * @param x [TODO:description]
   * @param y [TODO:description]
   * @return [TODO:description]
   */
enum Foo foofunc(int x, int y);

This will usually be compiled with:

clang -I /tmp/tc test.c

[kkoomen]

Well. I can reproduce it with a simple example, but not quite sure what is going on:

// test.h
enum VeryVeryUniqueEnumName {bar};

// test.c
#include <stdio.h>
#include "test.h"

/**

• struct _IO_FILE - [TODO:description]
  */
  enum VeryVeryUniqueEnumName foofunc(int x, int y);

Output:

[DoGe] libclang.py generator failed.
CursorKind.STRUCT_DECL
[DoGe] libclang.py generator failed.
CursorKind.STRUCT_DECL
{"name": "_IO_FILE", "parameters": []}
[DoGe] libclang.py generator failed.
CursorKind.STRUCT_DECL

I have no idea why.

I do get different results. I added the test.h and simply generated the results. These are my results, it properly detects it's an enum, but I haven't added ENUM_DECL yet to be supported, but atleast it detects it properly. How come it can't recognize it in your local environment?

[DoGe] libclang.py generator failed.
CursorKind.ENUM_DECL VeryVeryUniqueEnumName
[DoGe] libclang.py generator failed.
CursorKind.ENUM_DECL VeryVeryUniqueEnumName
[DoGe] libclang.py generator failed.
CursorKind.ENUM_DECL VeryVeryUniqueEnumName

---

And thank you for your time and patience :)

I should say this to you ;)

[mlyapin]

Sorry, seems like a problem with my system (ArchLinux, glibc, clang 9.0.0, nvim 0.4.3).
Can't reproduce the problem on another system (MacOS 10.15.1, apple's libc, clang 9.0.0, nvim 0.4.3).
Well, never mind then, I will look into it later.

[kkoomen]

Okay. I will close this issue for now. If you experience new problems, feel free to make a new issue.

[mlyapin]

And about the indexing problem:
Would it be difficult to add something like an array (e.g. g: doge_clang_extra_args) of additional arguments for clang that would be passed directly to clang.cindex.Index.parse(args=[...])?

Together with plugins like Localvimrc, this will be a good and easy workaround without much hassle.

[kkoomen]

That is indeed a useful option. I'll work on it right away.

[kkoomen]

@jbqxo Merged it into master. You can use g:doge_clang_args.

[TomzBench]

im unable to generate docs for enums. its hard to follow what the solution is. Should this still be open? From comments above, It seems like this was closed because it works on @mlyapin Mac but still doesnt work on his linux. (or my linux, (perhaps all linux?))