👌 IMPROVE: Store rawtext in AST nodes (for gettext)

[jpmckinney]

If you have Markdown like:

* My **Awesome** markdown

Sphinx needs to be passed My **Awesome** markdown.

The code does not do that.

This is the same bug as in recommonmark: readthedocs/recommonmark#187 which had originally been fixed by a Sphinx maintainer, and which I more recently fixed (though recommonmark is very behind on merging PRs).

As you can see in the code for paragraphs, sometimes the original Markdown content is not where you expect it. So far, I've only tested a few tokens.

cc @choldgraf

[codecov]

Codecov Report

Merging #301 (2875f5e) into master (86df2ea) will increase coverage by 0.05%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #301      +/-   ##
==========================================
+ Coverage   90.60%   90.65%   +0.05%     
==========================================
  Files          14       14              
  Lines        1809     1809              
==========================================
+ Hits         1639     1640       +1     
+ Misses        170      169       -1     

Flag	Coverage Δ
pytests	90.65% <100.00%> (+0.05%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
myst_parser/docutils_renderer.py	91.85% <100.00%> (ø)
myst_parser/html_to_nodes.py	91.66% <0.00%> (+2.08%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 86df2ea...0112d62. Read the comment docs.

[jpmckinney]

Actually, the problem is lower-level. I'll open an issue.

[jpmckinney]

Sorry, nevermind - this branch actually does address #304 (at least for the node types I tested).

[chrisjsewell]

Sorry, nevermind

lol yeh I was just going to say it does capture the "bold" syntax etc.

you can use https://markdown-it.github.io/ (and the debug tab) to see what gets stored where in the tokens

[chrisjsewell]

Is there any other additions you want to make before merging?

[chrisjsewell]

I see you removed all the changes for list items, etc. Was that intended?

[jpmckinney]

@chrisjsewell I added a bunch more test cases in https://github.com/jpmckinney/myst-parser-tests and they all extract correctly now with only the changes to these two lines. I can still add a table test case, and maybe some other block-level elements I missed. I don't really know how to test this without looking in Sphinx's code to see what it does when the gettext builder is run.

[jpmckinney]

I've now added a table test case, and it extracts incorrectly, so I'll add another commit.

[jpmckinney]

The new commit fixes terms (dt) and table cells.

[chrisjsewell]

I think we should add a test for this, to https://github.com/executablebooks/MyST-Parser/tree/master/tests/test_sphinx/sourcedirs:

• Add a gettext folder with an index.md and conf.py similar to the ones in https://github.com/jpmckinney/myst-parser-tests
• Add a test to https://github.com/executablebooks/MyST-Parser/blob/master/tests/test_sphinx/test_sphinx_builds.py, that builds that folder with the gettext builder and then reads+checks the _build/gettext/index.pot is as expected in some way (probably similar to

  MyST-Parser/tests/test_sphinx/conftest.py

  Lines 58 to 59 in 702a982

  	@pytest.fixture
  	def get_sphinx_app_output(file_regression):

  )

[chrisjsewell]

thats great thanks