New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

main: revise ptags for client tools #2459

Merged

masatake merged 17 commits into universal-ctags:master from masatake:revise-pseudo-tags

Mar 22, 2020

Member

masatake commented Mar 7, 2020 •

edited

Loading

[SELF-INCOMPATBILE] emit TAG_KIND_DESCRIPTION ptag only for enabled kinds,
introduce TAG_FIELD_DESCRIPTION ptag,
introduce TAG_EXTRA_DESCRIPTION ptag,
introduce TAG_PROC_CWD ptag,
attach "extras:ptag" fields to ptags, and
accept ptag names wrapped by '{' and '}'.

TAG_PARAM_DESCRIPTION still reamains in "not-impelemented".
However, the changes in this pull request may be enough for 6.0.

Member Author

masatake commented Mar 7, 2020 •

edited

Loading

These are should be explaiend in ctags-for-client-developers(5) man page.
format(5) man page must be updated a bit, too.

We must extend readtags API to support these new ptags.

coveralls commented Mar 7, 2020 •

edited

Loading

Coverage increased (+0.04%) to 86.839% when pulling 5842bcc on masatake:revise-pseudo-tags into 30cd8e0 on universal-ctags:master.

codecov bot commented Mar 7, 2020 •

edited

Loading

Codecov Report

Merging #2459 into master will increase coverage by 0.03%.
The diff coverage is 97.6%.

@@            Coverage Diff             @@
##           master    #2459      +/-   ##
==========================================
+ Coverage    86.7%   86.74%   +0.03%     
==========================================
  Files         177      177              
  Lines       36500    36572      +72     
==========================================
+ Hits        31646    31723      +77     
+ Misses       4854     4849       -5

Impacted Files	Coverage Δ
main/field.c	`95.85% <100%> (+0.02%)`	⬆️
main/writer.c	`97.72% <100%> (ø)`	⬆️
parsers/cxx/cxx_parser_variable.c	`90.56% <100%> (ø)`	⬆️
main/ptag.c	`93.33% <100%> (+0.37%)`	⬆️
main/entry.c	`87.88% <100%> (ø)`	⬆️
main/writer-json.c	`93.67% <100%> (ø)`	⬆️
main/xtag.c	`94.11% <100%> (+0.15%)`	⬆️
main/writer-ctags.c	`98.7% <100%> (+0.04%)`	⬆️
main/options.c	`83.19% <94.59%> (+0.44%)`	⬆️
main/parse.c	`95.06% <97.91%> (+0.18%)`	⬆️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 30cd8e0...5842bcc. Read the comment docs.

masatake mentioned this pull request

Cxx: fill end fields of variable kind tags #2456

Merged

masatake force-pushed the revise-pseudo-tags branch 4 times, most recently from fb287dd to fd1d0ee Compare

March 8, 2020 14:44

Member Author

masatake commented Mar 8, 2020 •

edited

Loading

I have written the initial version of ctags-client-tools(7) man page. See e61022a.

Update tags(5)
Describe each ptags in ctags-client-tools(7)

I'm also working on readtags side.

With both side changes (ctags and readtags), you can write a linker, a tool merging more than two tags file into one. It is really needed to apply ctags and tags files to large codebases.

The changes in pseudo tags related code are needed to make ctags support multi-pass parsing.

masatake force-pushed the revise-pseudo-tags branch 3 times, most recently from 558cde2 to 41c505b Compare

March 9, 2020 19:52

Member Author

masatake commented Mar 9, 2020

I will add TAG_PARAM_DESCRIPTION after releasing 6.0.0.

!_TAG_PARAM_DESCRIPTION!C      includePath      /\/usr\/include/

masatake mentioned this pull request

Release plans ? #1038

Closed

jwhitley reviewed

View reviewed changes

jwhitley left a comment

It looks really good @masatake-san. Thank you for your hard work on universal-ctags! I have tried to make small suggestions to improve the flow in English. Please review them.

docs/man/ctags-client-tools.7.rst Outdated

Comment on lines 21 to 22

		Client tool here is a tool running ctags command
		and/or reading tags file generated by ctags command.

jwhitley Mar 9, 2020

Suggested change

      
            **Client tool** here is a tool running ctags command
          
            and/or reading tags file generated by ctags command.
          
            **Client tool** means a tool running the ctags command
          
            and/or reading a tags file generated by ctags.

docs/man/ctags-client-tools.7.rst Outdated

Comment on lines 47 to 48

		put to the field for input file. The description, "Derived from
		Exuberant Ctags", is put to the field for pattern.

jwhitley Mar 9, 2020

Suggested change

      
            put to the field for input file. The description, "Derived from
          
            Exuberant Ctags", is put to the field for pattern.
          
            used in the field for input file. The description, "Derived from
          
            Exuberant Ctags", is used in the field for pattern.

docs/man/ctags-client-tools.7.rst Outdated

+              Exuberant Ctags", is put to the field for pattern.
+              Universal-ctags extends the naming scheme of the classical pseudo-tags
+              available in Exuberant-ctags for emitting a language specific

jwhitley Mar 9, 2020

Suggested change

      
            available in Exuberant-ctags for emitting a language specific
          
            available in Exuberant-ctags for emitting language specific

docs/man/ctags-client-tools.7.rst Outdated

Comment on lines 65 to 66

		pseudo tag with the option, don't include neither the prefix ("!_")
		nor the suffix ("!C"). Specify the body of the name, here

jwhitley Mar 9, 2020

Suggested change

      
            pseudo tag with the option, don't include neither the prefix ("!_")
          
            nor the suffix ("!C"). Specify the body of the name, here
          
            pseudo tag with the option, don't include the prefix ("!_")
          
            or the suffix ("!C"). Specify the body of the name, here

docs/man/ctags-client-tools.7.rst Outdated

+              	ctags emits pseudo-tags by default when writing tags
+              	to a regular fie (like "tags' file.) However, when specifying ``-o -``
+              	or ``-f -`` for writing tags to standard output,
+              	ctags doesn't emits pseudo-tags. ``--extras=+p`` or

jwhitley Mar 9, 2020

Suggested change

      
            	ctags doesn't emits pseudo-tags. ``--extras=+p`` or
          
            	ctags doesn't emit pseudo-tags. ``--extras=+p`` or

docs/man/ctags-client-tools.7.rst Outdated

+              	``--pseudo-tags=...`` option for enabling them.
+              ``--pseudo-tags=[+|-]names|*``
+              	Specifies a list of kinds of pseudo-tag to include in the output.

jwhitley Mar 9, 2020

Suggested change

      
            	Specifies a list of kinds of pseudo-tag to include in the output.
          
            	A list of kinds of pseudo-tags to include in the output.

docs/man/ctags-client-tools.7.rst Outdated

Comment on lines 92 to 95

+              	The parameter names is a set of names listed with
+              	``--list-pseudo-tags``. Surround each name of the set by braces
+              	like "{TAG_PROGRAM_AUTHOR}". "!_", the prefix representing
+              	pseudo-tags is for emitting. You don't have to include the prefix
+              	when specifying a name in the option argument for ``--pseudo-tags=``
+              	option.

jwhitley Mar 9, 2020

Suggested change

      
            	The parameter names is a set of names listed with
          
            	``--list-pseudo-tags``. Surround each name of the set by braces
          
            	like "{TAG_PROGRAM_AUTHOR}". "!_", the prefix representing
          
            	pseudo-tags is for emitting. You don't have to include the prefix
          
            	when specifying a name in the option argument for ``--pseudo-tags=``
          
            	option.
          
            	The parameters are a set of pseudo tag names. Valid pseudo tag names
          
            	can be listed with ``--list-pseudo-tags``. Surround each name in the set 
          
            	with braces, like "{TAG_PROGRAM_AUTHOR}". You don't have to include the "!_" 
          
            	pseudo tag prefix when specifying a name in the option argument for ``--pseudo-tags=``
          
            	option.

docs/man/ctags-client-tools.7.rst Outdated

+              	  ...
+              	A client tool can know "{anonymous}", "{fileScope}", "{pseudo}",
+              	and "{subparser}" extars are enabled from the output.

jwhitley Mar 9, 2020

Suggested change

      
            	and "{subparser}" extars are enabled from the output.
          
            	and "{subparser}" extras are enabled from the output.

docs/man/ctags-client-tools.7.rst Outdated

Comment on lines 154 to 155

		If your tool relies on some fields , refer to the pseudo-tags of
		this kind. A tool can reject the tags file that doesn't include

jwhitley Mar 9, 2020

Suggested change

      
            	If your tool relies on some fields , refer to the pseudo-tags of
          
            	this kind.  A tool can reject the tags file that doesn't include
          
            	If your tool relies on some fields, refer to the pseudo-tags of
          
            	this kind.  A tool can reject a tags file that doesn't include

docs/man/ctags-client-tools.7.rst Outdated

+              	TBW
+              ``TAG_PROC_CWD`` (new in Universal-ctags)
+              	Indicates the working directory of ctags was at.

jwhitley Mar 9, 2020

Suggested change

      
            	Indicates the working directory of ctags was at.
          
            	Indicates the working directory of ctags during processing.

Member Author

masatake commented Mar 10, 2020

Another review. emptyflask@2f3f88afed69929f88ea31a393b706f76c6ba531from @emptyflask.

I have never imaged I could get more than two reviewers. An issue here; it is hard for me to decide which comments I should choose. I will try to do cherry-pick.

Member

AmaiKinono commented Mar 11, 2020 •

edited

Loading

My question after reading the added manpage is: Is it safe to say that tags start with !_ are pseudo tags? (seems not.) If not, what's the recommended way to deal with pseudo tags for client tools? (will be readtags, right?) Seems readtags ignores psuedo tags in its output, what method does it use?

Member Author

masatake commented Mar 11, 2020

We already handled your concerns:-)

ctags can handle "!_" appeared in input well.

input.html

<html><body><h1>!_HELLO</h1></body></html>

$ ctags -o input.tags input.html
$ cat input.tags
!_TAG_FILE_FORMAT	2	/extended format; --format=1 will not append ;" to lines/
!_TAG_FILE_SORTED	1	/0=unsorted, 1=sorted, 2=foldcase/
!_TAG_OUTPUT_FILESEP	slash	/slash or backslash/
!_TAG_OUTPUT_MODE	u-ctags	/u-ctags or e-ctags/
!_TAG_PROGRAM_AUTHOR	Universal Ctags Team	//
!_TAG_PROGRAM_NAME	Universal Ctags	/Derived from Exuberant Ctags/
!_TAG_PROGRAM_URL	https://ctags.io/	/official site/
!_TAG_PROGRAM_VERSION	0.0.0	/30783d15/
\x21_HELLO	/tmp/input.html	/^<html><body><h1>!_HELLO<\/h1><\/body><\/html>$/;"	h	language:HTML
[yamato@slave]~/var/ctags-github% readtags -t input.tags  -l 
readtags -t input.tags  -l 
!_HELLO	/tmp/input.html	/^<html><body><h1>!_HELLO<\/h1><\/body><\/html>$/

I cannot remember where this behaviour, escaping the first !, is documented.
If I cannot find it, we have to wrtite it to our man pages.

For users directly use readtags, the un-escaping feature of readtags is good enough. However, people writing a tool for manipulating tags files may want the way to access raw-escaped data in tags file from the tool. I am the one of the such people. I'm working on this topic in #2468.

Member Author

masatake commented Mar 21, 2020 •

edited

Loading

I added commits reflecting the reviewing done by @jwhitley and @emptyflask.

Member Author

masatake commented Mar 21, 2020

Though I use "squash!" prefix in the commit logs, I will not squash to allow me to re-read what kind of changes were needed.

masatake force-pushed the revise-pseudo-tags branch 2 times, most recently from 64b8fe3 to 93073dc Compare

March 22, 2020 22:42

masatake added 8 commits

March 23, 2020 07:44


          [SELF-INCOMPATIBLE]main: emit TAG_KIND_DESCRIPTION ptags only for ena…

fd0af0c

…bled kinds

Signed-off-by: Masatake YAMATO <yamato@redhat.com>


          main: extend pseudo tags related functions to take language parameter…

aa53c96

… explictily

Signed-off-by: Masatake YAMATO <yamato@redhat.com>


          main: insert pattern starting and ending markers (/) for PTAG_KIND_* …

2ea4388

…in ctags writer

The makers are parts of tags output. So ctags writer should insert
them when emitting ptags.

Signed-off-by: Masatake YAMATO <yamato@redhat.com>


          main: allocate vString each time when emitting KIND_DESCRIPTION

02cadfe

...instead of allocating it as function static variable.

This change is for suppressing "still reachable leak" reported
by valgrind.

Signed-off-by: Masatake YAMATO <yamato@redhat.com>


          main: introduce TAG_FIELD_DESCRIPTION ptag

9ab0e07

Signed-off-by: Masatake YAMATO <yamato@redhat.com>


          main: introduce TAG_EXTRA_DESCRIPTION ptag

726cac0

Signed-off-by: Masatake YAMATO <yamato@redhat.com>


          main: introduce TAG_PROC_CWD ptag

3692a8c

This records the current working directory of ctags process.
This may help client tools solve the file path for input
source which is recorded in tags file in relative path.

Signed-off-by: Masatake YAMATO <yamato@redhat.com>


          Tmain: rename pseudo tags related test cases

e350c1e

Signed-off-by: Masatake YAMATO <yamato@redhat.com>

masatake added 9 commits

March 23, 2020 07:44


          Tmain: rename a test case about exttas field

9a1b9fd

Signed-off-by: Masatake YAMATO <yamato@redhat.com>


          Tmain: rename some test cases

5db6ae8

Signed-off-by: Masatake YAMATO <yamato@redhat.com>


          main: fill extras field of pseudo tags

545d142

Consider a fragment of html file:

	<h1>!_TAG_PROGRAM_VERSION</h1>

our HTML parser captures it well with escaping as:

	\21_TAG_PROGRAM_VERSION ...

So readtags is not confused; readtags understands it is not a pseudo
tag. However, once the tag is stored to tagEntry data struct of
readtags, the client code cannot distinguish it with a pseudo tag
because the name of the tag is escaped when readtags stores it to the
data structure.

This change attaches "extras:pseudo" to pseudo tags.  This helps hits
whether a tag is a pseudo tag or not when a tag is stored to the data
structure; if "extras:pseudo" is not attached, the tag is not a pseudo
tag.

Signed-off-by: Masatake YAMATO <yamato@redhat.com>


          main: allow to use {...} notation when enabling/disabling a pseudo tag

e0653da

Signed-off-by: Masatake YAMATO <yamato@redhat.com>


          docs(man,web): add an initial version of ctags-client-tools(7)

00e5952

Signed-off-by: Masatake YAMATO <yamato@redhat.com>


          docs(man): use proper page titles

b28d530

Signed-off-by: Masatake YAMATO <yamato@redhat.com>


          squash! docs(man,web): add an initial version of ctags-client-tools(7)

9cee5f4

Reflect the result of reviewing done by @jwhitley.


          squash! docs(man,web): add an initial version of ctags-client-tools(7)

12d83e6

Use "pseudo tag types" instead of "pseudo tag kinds" to avoid
confusion with kinds in "kinds in language objects".


          squash! docs(man,web): add an initial version of ctags-client-tools(7)

5842bcc

Reflect the result of reviewing done by @emptyflask.

masatake force-pushed the revise-pseudo-tags branch from 93073dc to 5842bcc Compare

March 22, 2020 22:45

Member Author

masatake commented Mar 22, 2020

We should not add TAG_PARAM_DESCRIPTION. If it is needed, individual parsers should emit their own ptags.

TAG_ROLE_DESCRIPTION, this one is needed when we add an option for enabling/disabling roles.

masatake merged commit 1ee982c into universal-ctags:master

Member Author

masatake commented Mar 22, 2020

@jwhitley and @emptyflask, thank you very much.
Please, allow me to call you again when I write more documents at #1038.

jwhitley commented Mar 23, 2020

I would be glad to help!

masatake mentioned this pull request

[LOOKING FOR VOLUNTEERS] Language consultants/advisers #2494

Open

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet