Data Munging: A Process Overview in Python

How does one transform a massive, inconsistent spreadsheet of transactions riddled with typos and bad delimiters into structured and trusted input for use in sophisticated analytics? Worse yet - what if it’s not even a spreadsheet, but a raw webpage, a thousand emails, a text file containing a billion error logs, or a collection of unstructured documents stored haphazardly in a cloud?

The answer is data munging. Data munging is a set of concepts and a methodology for taking data from unusable and erroneous forms to the new levels of structure and quality required by modern analytics processes and consumers. 

What is data munging?

Sometimes confused with data wrangling, data munging is the initial process of refining raw data into content or formats better-suited for consumption by downstream systems and users. 

The term ‘Mung’ was coined in the late 60s as a somewhat derogatory term for actions and transformations which progressively degrade a dataset, and quickly became tied to the backronym “Mash Until No Good” (or, recursively, “Mung Until No Good”).

But as the diversity, expertise, and specialization of data practitioners grew in the internet age, ‘munging’ and ‘wrangling’ became more useful generic terms, used analogously to ‘coding’ for software engineers. 

With the rise of cloud computing and storage, and more sophisticated analytics, these terms evolved further, and today refer specifically to the initial collection, preparation, and refinement of raw data.

The data munging process: An overview

With the wide variety of verticals, use-cases, types of users, and systems utilizing enterprise data today, the specifics of munging can take on myriad forms. 

1. Data exploration: Munging usually begins with data exploration. Whether an analyst is merely peeking at completely new data in initial data analysis (IDA), or a data scientist begins the search for novel associations in existing records in exploratory data analysis (EDA), munging always begins with some degree of data discovery. 
2. Data transformation: Once a sense of the raw data’s contents and structure have been established, it must be transformed to new formats appropriate for downstream processing. This step involves the pure data scientist, for example un-nesting hierarchical JSON data, denormalizing disparate tables so relevant information can be accessed from one place, or reshaping and aggregating time series data to the dimensions and spans of interest. 
3. Data enrichment: Optionally, once data is ready for consumption, data mungers might choose to perform additional enrichment steps. This involves finding external sources of information to expand the scope or content of existing records. For example, using an open-source weather data set to add daily temperature to an ice-cream shop’s sales figures. 
4. Data validation: The final, perhaps most important, munging step is validation. At this point, the data is ready to be used, but certain common-sense or sanity checks are critical if one wishes to trust the processed data. This step allows users to discover typos, incorrect mappings, problems with transformation steps, even the rare corruption caused by computational failure or error.

Data munging in python

When it comes to actual tools and software used for data munging, data engineers, analysts, and scientists have access to an overwhelming variety of options.

The most basic munging operations can be performed in generic tools like Excel or Tableau —from searching for typos to using pivot tables, or the occasional informational visualization and simple macro. But for regular mungers and wranglers, a more flexible, powerful programming language is far more effective. 

Python is often lauded as the most flexible popular programming language, and this is no exception when it comes to data munging. With one of the largest collections of third-party libraries, especially rich data processing and analysis tools like Pandas, NumPy, and SciPy, Python simplifies many complex data munging tasks. Pandas in particular is one of the fastest-growing and best-supported data munging libraries, while still only a tiny part of the massive Python ecosystem. 

Python is also easier to learn than many other languages thanks to simpler, more intuitive formatting, as well as a focus on legible english-language-adjacent syntax. Thanks to Python’s wide applicability, rich libraries, and online support, new practitioners will additionally find the language useful far beyond data processing use cases, everywhere from web development to workflow automation.

The cloud and the future of data munging

Cloud computing and cloud data warehouses have generally contributed to a massive expansion of enterprise data’s role throughout organizations, and across markets. Data munging is only a relevant term today thanks to the importance of fast, flexible, but carefully governed information, all of which have been the primary benefits of modern cloud data platforms.

Now, concepts such as the data lake and NoSQL technologies have exploded the prevalence, and utility, of self-service data and analytics. Individual users everywhere have access to vast raw data, and are increasingly trusted to transform and analyze that data effectively. These specialists must know how to clean, transform, and verify all of this information themselves.

Whether in modernizing existing systems like data warehouses for better reliability and security, or empowering users such as data scientists to work on enterprise information end-to-end, data munging have never been more relevant concepts. 

Getting started with data munging

Data munging is the general procedure for transforming data from erroneous or unusable forms, into useful and use-case-specific ones. Without some degree of munging, whether performed by automated systems or specialized users, data cannot be ready for any kind of downstream consumption. 

But powerful and versatile tools, like Python, are making it increasingly easy for anyone to munge effectively. Talend Data Fabric, integrated with the Python data ecosystem, does most of the munging for you by collecting, transforming, and sharing well-governed data all through a single suite of applications. Try Talend Data Fabric today to begin preparing your data.

Try Talend Data Fabric today