Data lake vs data warehouse

Data lakes and data warehouses are both widely used to store data for analytics, but they are not interchangeable terms. A data lake tends to include large amounts of raw data, the purpose for which may not yet be defined. A data warehouse is a repository for structured, filtered data that has already been processed for a specific purpose. There is a newer but established data management architecture trend called the data lakehouse, which sets out to combine data lake with the data management capabilities of a data warehouse.

These first two types of data storage are often confused, but are much more different than they are alike. The distinction is important because they serve different granular purposes and require different sets of eyes to be properly optimised. While a data lake works for one company, a data warehouse may be a better fit for another. Some companies may need both.

Four key differences between a data lake and a data warehouse

There are several differences between a data lake and a data warehouse. Data structure, processing methods, ideal users, and the overall purpose of the data are the key differentiators.

Data structure: raw vs. processed

Raw data is data that has not yet been processed for a purpose and tends to be unstructured (think of a video file) or semi-structured (for instance, images with metadata attached). Perhaps the greatest difference between data lakes and data warehouses is the varying structure of raw vs. processed data. Data lakes primarily store raw, unprocessed data, often including multimedia files, log files, and other very large files, while data warehouses mostly store structured, processed, and refined data that tends to be text and numbers.

Because of this, data lakes typically require much larger storage capacity than data warehouses. Additionally, raw, unprocessed data is malleable, can be quickly analysed for any purpose, and is ideal for machine learning. The risk of all that raw data, however, is that without appropriate data quality and data governance measures in place, data lakes may become data swamps.

Purpose: undetermined vs. in-use

The purpose of individual data pieces in a data lake is not fixed. Raw data flows into a data lake, sometimes with a specific future use in mind and sometimes just to have on hand. This means that data lakes have less organisation and less filtration of data than data warehouses.

Processed data is data that has been put to a specific use. Since data warehouses only house processed data, all of the data in a data warehouse has been used for a specific purpose within the organisation and is more likely to be queried in the future. This means that storage space is not wasted on data that is less likely to be used.

Users: data scientists vs. business professionals

Data lakes are often difficult to navigate by those unfamiliar with unprocessed data. Raw, unstructured data usually requires a data scientist and specialised tools to process and translate for any specific business use.

However, it’s important to note that there is growing momentum behind data preparation tools that create self-service access to the information stored in data lakes.

Learn more at, “What is Data Preparation?” →

Processed data is used in charts, spreadsheets, tables, and more, so most business users can read it. Processed data like that stored in data warehouses only requires that the user be familiar with the topic represented.

Accessibility: flexible vs. secure

Accessibility and ease of use refer to the use of the data repository as a whole, not the data within it. Data lake architecture has less structure and therefore, data lakes have very few limitations.

Data warehouses are more structured by design. One major benefit of data warehouse architecture is that the processing and structure of data makes the data itself easier to decipher, while the limitations of structure make data warehouses difficult and costly to manipulate.

Data lake vs. data warehouse: which is right for me?

Organisations often need both. Data lakes were born out of the need to harness big data and benefit from raw, unprocessed data for machine learning. Yet there is still a need to create data warehouses for analytics use by business users.

Healthcare: data lakes store unstructured information

Data warehouses have been used for many years in the healthcare industry, but their use has not been hugely successful. Because of the unstructured nature of much of the data in health care (physicians’ notes, clinical data, etc.) and the need for real-time insights, data warehouses are generally not an ideal model.

Data lakes allow for a combination of structured and unstructured data, which tends to be a better fit for health care companies.

Learn more about how Talend helped AstraZeneca build a global data lake. →

Education: data lakes offer flexible solutions

In recent years, the value of big data in education has become enormously apparent. Data about student grades, attendance, and more can not only help failing students get back on track, but can actually help predict potential issues before they occur. Flexible big data solutions have also helped educational institutions streamline billing, improve fundraising, and more.

Much of this data is vast and very raw, so many times, institutions in the education sphere benefit best from the flexibility of data lakes.

Finance: data warehouses appeal to the masses

In finance, as well as other business settings, a data warehouse is often the best storage model because it can be structured for access by the entire company rather than a data scientist.

Big data has helped the financial services industry make enormous strides, and data warehouses have been a significant player in that forward progress. Financial services companies may also benefit from machine learning and AI, however, so they may need data lakes as well.

Transportation: data lakes help make predictions

Much of the benefit of data lake insight lies in the ability to make predictions after the data is processed for predictive analytics, machine learning, and AI.

In the transportation industry, especially in supply chain management, the prediction capability that comes from flexible data in a data lake can have a huge upside, namely cost-cutting benefits realised by examining data from reports generated within the transport pipeline.

The importance of choosing a data lake or data warehouse

The “data lake vs. data warehouse” conversation has likely just begun, but the key differences in structure, process, users, and overall agility make each model unique. Depending on your company’s needs, developing the right data lake and/or data warehouse will be instrumental in growth.

As we mentioned above, a new model, the data lakehouse, is emerging. Only time will tell whether this is simply a refinement of data lakes or whether it becomes a “best of both worlds” alternative that can meet a wide range of needs.

Learn more about cloud data lakes, or try Talend Data Fabric to begin harnessing the power of big data today.