Data Sets for Quantitative Research: Public Use Datasets

There are many research organizations making data available on the web, but still no perfect mechanism for searching the content of all these collections. The links below will take you to data search portals which seem to be among the best available. Note that these portals point to both free and pay sources for data, and to both raw data and processed statistics.

• PEW Research Center
• Open Access Directory (OAD) Data Repositories
• Data.gov
• UK Data Archive
• Socioeconomic Applications Data Center
• Council of European Social Science Data Archives (CESSDA)
• NTIS Federal Computer Products Center.*  Includes databases, data files, CD-ROM, etc. available for purchase.  
• Harvard DataVerse
• r3Data.org Registry of Research Data Repositories
• Open Data: European Commission Launches European Data Portal (over 1 million datasets From 36 countries)
• Awesome Public Datasets (on github)*. Includes a mix of free and pay resources.
• SNAP (Stanford Network Analysis Project)
• Statistics, Resources and Big Data on the Internet, 2020 *

 * Resources that are not entirely free are marked with an asterisk.

Have you found fantastic numeric information in a less-than-ideal format, such as PDF or HTML?   Here are some software products that may help you transform those formats into numbers that you can read into a spreadsheet or statistical software program.  Some of these are free or offer limited time, free trials:

• Convert PDF charts and tables into machine-readable, numeric datasets
  • Spark OCR: Find tables in images, visually identify rows and columns, and extract data from cells into data frames. Turn scans from financial disclosures, academic papers, lab results and more into usable data. 
  • PDFTables : PDF to Excel Converter
  • Tabula : Extract tables from PDFs
  • table-ocr: For those who know Python
  • Abbyy FineReader : Access and modify information locked in paper-based documents and PDF files
  • OCR Space: This free service transforms PDFs into plain text files directly in your browser.  Rows and columns are preserved, making it easier to import the file into Excel using the Import Text Wizard.  See further explanation and instructions here: Table recognition with OCR.
     
• Web scraping tools
  • Parsehub : Data mining tool for data scientists and journalists
  • Data Streamer : Index weblogs, mainstream news, and social media
  • Outwit : Turn websites into structured data

Feeling intrigued, but unsure how to leverage web-based data for your own research?  Here are some how-to guides:

• Intro to Data and Code by Lena Groeger, Propublica
  • Data Journalism: What it is and why should I care?
  • How to get data from the Web
  • Manipulating data
• Library books available for checkout
  • Data for journalists: a practical guide for computer-assisted reporting by Brant Houston (2019)
• E-Books
  • Scraping for journalists by Paul Bradshaw (2013)
  • Data Journalism Heist by Paul Bradshaw (2013)

These are some of the most significant datasets available on the internet, arranged by topic.  Almost everything here is freely available. The few that do involve fees are marked with asterisks (*). Note that some of the listings below are also available in ICPSR.

Political Science/Public Policy

• American National Election Studies
• Conflict and Peace Data Bank, 1948-1978 available through ICPSR
• Correlates of War
• Cross-National Time-Series Data Archive (available as a library item on CD-ROM)
• International Country Risk Guide (ICRG) Table 3B: Political Risk Points by Component, 1984-2009 (available to MU affiliates)
• Polidata Presidential Results By Congressional District 1992-2004 (available to MU affiliates)
• Record of American Democracy, 1984-1990
• Survey of Income and Program Participation 

Demographics

• IPUMS: Integrated Public Use Microdata Series
• Geocorr -- Geographic Correspondence Engine
• Missouri Census Data Center UEXPLORE/Dexter (explanation)
• National Historical Geographic Information System

Business and Economics

• Consumer Expenditure Surveys microdata
• National Bureau of Economic Research data
• National Longitudinal Surveys from the Bureau of Labor Statistics
• Panel Study of Income Dynamics
• World Bank – Poverty and Equity Data
• International industrial development (manufacturing, mining, utilities, etc.) data from the United Nations (UNIDO)

Health

• Biologic Specimen and Data Repository Information Coordinating Center (bioLINCC)
• Demographic and Health Surveys (mainly 3rd world countries)
• Global Health Observatory data repository from the World Heath Organization
• ICPSR Health and Medical Care Archive 
• ICPSR National Addiction and HIV Data Archive Program
• National Center for Health Statistics Public Use Data Files from the U.S. CDC
• Missouri Information for Community Assessment (MICA) health datasets
• National Longitudinal Study of Adolescent Health
• National Cancer Institute SEER data

Science

• DataONE  Earth and environmental data
• EPA Environmental dataset gateway

Sociology

• General Social Survey
• National Longitudinal Surveys (U.S. Bureau of Labor Statistics)
• National Survey of Households and Families
• Pew Internet & Technology
• World Values Survey

Education

• National Center for Education Statistics DataLab
• The National Survey of College Graduates (NSCG)
• NCES Public Elementary & Secondary Schools Universe Survey Data
• The Survey of Doctorate Recipients (SDR)

Miscellaneous

• American Religion Data Archive
• National Household Travel Survey
• Roper Opinion Polls* 
• Eurostat*

*Resources that are not entirely free are marked with an asterisk