Feat/wiki scraper #51
Conversation
|
I think it's a good idea to replace Wikipedia's custom mathematics syntax with LaTeX. Does it make sense to do it at this stage of the pipeline, or later? |
|
@craffel Yeah it should be easy to parallelize this, it runs off files which list page titles (one per line) so you can parallelize over the files (and we are already parallelizing over wiki's) and we can split the inputs pretty easily for more parallelism. @StellaAthena I think it would best to have that happen later. I was thinking after this export there would be a step that converts from these xml to dolma which would have raw wiki markup as the |
blester125
force-pushed
the
feat/wiki-scraper
branch
from
d2dadf0
to
a2c7008
Compare
blester125
force-pushed
the
feat/wiki-scraper
branch
3 times, most recently
from
227138a
to
517bca9
Compare
blester125
force-pushed
the
feat/wiki-scraper
branch
4 times, most recently
from
05d64f2
to
f364ea8
Compare
blester125
force-pushed
the
feat/wiki-scraper
branch
from
450e6b1
to
77a1b68
Compare
blester125
force-pushed
the
feat/wiki-scraper
branch
from
3e31f23
to
e7d562c
Compare
Add tools to scrape mediawiki wikis that don't publish dumps Add tool that exports the xml based on the list of pages. Add the ability to convert wikis to dolma Download and extract script supports multiworker Create WTF Wikipedia parsing server which uses a worker pool to allow for timeouts Creation of script that removes html tags we found in many wiki dumps. Added Shadow Paging to the creation of wikitext dolma files Added Shadow Paging to dolma preprocessing. Added script that remove `None` lines from dolma files. Added script that can combine dolma shards while tracking what was used where to allow for aligned combinations of later versions.
blester125
force-pushed
the
feat/wiki-scraper
branch
from
59590a8
to
4e571b6
Compare
|
Datasets have been uploaded to https://huggingface.co/datasets/blester125/wiki-dolma WikiMedia + Talk pages are cleaner and have 14.6 Billion tokens. |
This PR adds scripts that can be used to get an xml export of mediawiki sites that don't provide dumps. The resulting dump will contain a list of
<page>, one for each exported page. Each page has multiple<revision>which can be used to create an author list. The most recent<revision>'s<text>can be used to get the mediawiki markup representation of the page to use as the document text.An index of pages is built using the
Special:AllPagesquery url and then exports are made usingSpecial:Export.