Sometimes data needs fixing before it can be used. Data Cleaning is the preparation of data that has been corrupted, is incorrect, or needs reformatting before loading into analytical software.
Depending on the type of cleaning that needs to be performed, there are various methods and tools used for data cleaning. One of the most common is spreadsheet software such as Excel.
Common points to consider when looking to clean your data include looking for:
Reformatting your data is one of the most common cleaning needs for a new data set. The most widely accepted data format is the American Standard Code for Information Interchange or ASCII and includes common file extensions of *.csv, *.txt., and *.dat.
Introductory Resource: Data Preparation & Descriptive Statistics
Examples:
Cleaning Data is often a cyclical event as data is used, reused, and added to. Below represents a common perception of the Data Cleaning Cycle.
Excel is not statistical software, but it is often the software of choice for data cleaning and inputting into statistical software.The library has many resources available for learning what capabilities excel has to offer for data cleaning.
There are a myriad of Data Cleaning Software choices available for purchase. It is always wise to consult with reviews or a trusted entity with experience with the software to make sure it applies to your application.
Opensource software is also available:
Best Practices in Data Cleaning by
Provides a clear, step-by-step process of examining and cleaning data in order to decrease error rates and increase both the power and replicability of results.
Data Cleaning by
In this book, we first discuss the goals of data cleaning. Often, the goals of data cleaning are not well defined and could mean different solutions in different scenarios. Toward clarifying these goals, we abstract out a common set of data cleaning tasks that often need to be addressed. This abstraction allows us to develop solutions for these common data cleaning tasks. We then discuss a few popular approaches for developing such solutions.
The Data Wrangler's Handbook by
This book demystifies the process of working with data, familiarizing readers with a small number of simple tools, and easily digestible but powerful concepts. Using tools that come with desktop computers, readers will learn to extract, manipulate, and analyze data (and metadata) of any size and complexity.
