Two of the more popular options for data architecture include data lake and data warehouses. A data warehouse is a lot like a filing cabinet to put it simply. Imagine a whole bunch of folders storing structured data that is also pre-sorted into formats that database software can use.
In contrast, data lakes are more like the disorganized box of files and papers many of us are guilty of having in storage. It can sometimes be a disorganized mess and where one file stops and the next one begins can be unclear.
Additionally, while data warehouses only contain structured data, data lakes contain both structured and unstructured data. The unstructured data is just that. It lacks any form of structure and is often referred to as the messy digital information such as pdf’s, audio and video files, and images.
So, now we will delve a bit more into the debate of a data lake vs. data warehouse.
Inside the Data Warehouse and Data Lake
When a company begins to load data into their data warehouse, they have to give it shape and structure, so it can be modeled prior to being placed in data warehousing; this process is often referred to as a schema-on-write.
However, with a data lake, the company is able to upload the information as raw data. It isn’t until you need to actually use the data when it will begin to take form and structure, also known as schema-on-read.
It is also important to note a date lake is in no way a data warehouse, nor should it be considered a replacement for one. You must determine the needs of the company to decide which is best because they are both optimized for different purposes, and each needs to be used for exactly what it was designed to do.
For example, company leadership wants to analyze sales figures across a specific timeframe including the number of inquiries they received about a certain product or the overall performance on various marketing campaigns. For these purposes, a data warehouse is the ideal storage choice for these kinds of applications because all the associated figures are stored as structured data.
However, for most companies launching new and expansive data initiatives, structured data is only a small part of the whole story. Each year, businesses are generating a substantial quantity of unstructured data. In fact, 451 Research found that 63 percent of enterprise data and service providers are keeping at least 25 petabytes of unstructured data.
(For those that don’t know, a petabyte is a unit of measurement used to describe digital information. One petabyte is equivalent to one quadrillion bytes.)
Furthermore, data lakes allow analysts the ability to go far beyond the traditional descriptive analytics, and they can explore the realm of predictive analytics. Predictive analytics use current and historical data available to predict future trends which could have an impact on various aspects of the company; for example, the following year’s revenue or potential risks and opportunities the company may face.
Before choosing either a data lake or a data warehouse, think about who will be conducting these data analyses and what sort of data sources they’ll need to do so. Data warehouses used to only be accessible to IT teams, while data lakes can be configured for access by analysts and business personnel across the company.
An enterprise data warehouse is a unified database that stores all the information about a company and then makes it accessible across the company. It features a more unified approach when it comes to organization which also leads to a better way to represent data. An enterprise data warehouse allows for the ability to categorize certain data analytics according to subject so that it can be classified or permissible for only specific divisions within a company.
Finding Value in the Data Lake
When it comes to making the most of the data lake your company has chosen, you will want to ensure the analytic platform is designed for a data lake. The platform needs to be able to embrace the loose structure data lakes are known for, and if the company is unable to take advantage of the data lake’s versatility, then they are not using it to their full advantage and see far less value than they otherwise could.
Because a data lake has the ability to store both types of data and its sustainability for the company’s future analytical needs, it’s certainly tempting to think data lakes are the most obvious answer. However, due to their loose structure, they are sometimes considered to be a “swamp” rather than a lake. Proper data governance and lineage are required to make the most of the data management and keep it in ready form for a team of analysts.
Starting the journey toward a more data-informed business is important. Executives decades ago may even remember data wasn’t even a topic readily discussed outside of IT. But now, with the magnitude of analytic needs and the diversity of tools currently available, it is the leadership teams turn to lead the conversation on the value of data storage.
