Big Data Datasets

Curated list of Publicly available Big Data datasets. Uncompressed size in brackets. No Blockchains.

Structured

• CelebFaces Attributes - 1.2 GB
  • Over 200k images of celbrities with 40 binary attribute annotations

Text

• CommonCrawl (AWS) - A corpus of web crawl data composed of over 25 billion web pages.
  • Semi-Structured (includes Metadata): 250 TB
• DBpedia - curated wikipedia data
• Freebase
  • Freebase: 22 GB (250 GB)
  • Freebase Deleted Triples: 2 GB (8 GB)
  • Freebase/wikidata Mappings: 22 MB (243 MB)
• StackOverflow Data (BigQuery) - 182 GB

Image

• Landsat 8 (AWS)
• Uber Self Driving Car Challenge
  • 200 GB+ (compressed)

Audio

• Million Song Dataset (AWS)

Bonus: API / Streamdata / "Self-Service"

• Facebook Social Graph API

Bonus: Opendata / Census / Government data

• EU Open Data Portal
• data.gov
• US Census
• data.gov.uk
• CIA World Factbook
• healthdata.gov
• digital.nhs.uk
• Gapminder
• National Centers for Environmental Information

Meta / Lists / Sources

These pages might link to datastes which are already in the list.

• Data Science Central List of Big Data Sets
• AWS Open Data
  • Usage examples
• Kaggle Datastes
• Data Science Central: Big data sets available for free
• KDNuggets