Commits on Sep 24, 2024

1. Tooling to download and process Wikis …

   Add tools to scrape mediawiki wikis that don't publish dumps
   
   Add tool that exports the xml based on the list of pages.
   
   Add the ability to convert wikis to dolma
   
   Download and extract script supports multiworker
   
   Create WTF Wikipedia parsing server which uses a worker pool to allow for timeouts
   
   Creation of script that removes html tags we found in many wiki dumps.
   
   Added Shadow Paging to the creation of wikitext dolma files
   
   Added Shadow Paging to dolma preprocessing.
   
   Added script that remove `None` lines from dolma files.
   
   Added script that can combine dolma shards while tracking what was used
   where to allow for aligned combinations of later versions.

   

   blester125 committed Sep 24, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 4e571b6
   • Browse repository at this point

   Copy the full SHA

   4e571b6 View commit details

   Browse the repository at this point in the history