What is a Lakehouse Anyway?

The podcast discusses the concept of lake houses in data architecture. A lake house integrates a data warehouse with a data lake, providing a more efficient and versatile means of storing and querying data. It follows a Medallion architecture with three layers: bronze for raw data, silver for cleaned and enriched data, and gold for aggregated data suitable for analytics. This structure helps prevent the data swamp issue common in traditional data warehouses.

Using a lake house can be cost-effective, allowing the storage of various data types quickly and inexpensively. It also supports data science applications by retaining raw data needed for detailed analysis. However, it requires additional steps for data transformation, especially for raw data.

Switching from an existing data warehouse to a lake house should be part of a larger transformation plan, done incrementally to ensure a smooth transition. For new setups, a lake house should be considered if large data volumes are expected, while traditional data warehouses might still be suitable for smaller organizations with existing SQL skills.

Modern tools like Azure Data Lake and Fabric can streamline the process, helping organizations achieve a single version of the truth. The discussion emphasizes the importance of evaluating organizational needs and available resources before deciding on the best data architecture solution.

You can download Power BI Desktop from here.

If you already use Power BI, or are considering it, we strongly recommend you join your local Power BI user group here.

To find out more about our services and the help we can offer, contact us at one of the websites below:
UK and Europe: https://www.clearlycloudy.co.uk/
North America: https://clearlysolutions.net/

Transcript

Andy
We're going to discuss lake houses—whether they exist or not. Think of it as a myth like elves or Eskimos. Today will be a quick one, and apologies for the microphone quality—I'm using a wireless headset in a client's office. So, today's topic is lake houses. What is a lake house, anyway? We'll give a technical description from Tom and discuss the pros and cons. So, let's start. What is a lake house, Tom?

Tom
A lake house is essentially a house on a lake without any pie charts. Seriously, it's about data warehousing backed by a data lake instead of a traditional transactional database. Typically, you'd use a SQL instance or Azure for a data warehouse. In a lake house, you store data in a data lake, usually in the delta format. Delta format helps with efficient querying, unlike CSV files which can be unwieldy.

The lake house structure follows the Medallion architecture: bronze, silver, and gold layers. The bronze layer ingests raw data without transformations. The silver layer involves deduplication, data cleansing, and enrichment. This layer is good for data science applications needing raw data. The gold layer aggregates data for analytics, suitable for dashboards in tools like Power BI. This layered approach makes data querying more efficient and versatile.

Andy
That was thorough, Tom. Let's move to Shailan. Why would someone use a lake house?

Shailan
There are several reasons. First, storage types: a lake house can handle various types of data, including CSV files, quickly and cost-effectively. Second, the cost is relatively low. Using pricing calculators, we see that adding terabytes is inexpensive.

The architecture's layered approach—bronze, silver, gold—optimizes data storage and retrieval. However, it requires extra steps for data transformation, especially for raw data. Yet, it helps achieve a single version of the truth, centralizing data effectively.

Tom
By the silver and gold layers, headers and other issues should be sorted out. Raw data might need more processing initially.

Shailan
True. While transforming raw data may require effort, the benefits like cost-effectiveness and data versatility make it worthwhile. Tools like Azure Data Lake and Fabric streamline the process, preventing the data swamp issue common in traditional data warehouses.

Andy
If someone already has a data warehouse, should they switch to a lake house?

Tom
A switch to a lake house should be part of a bigger transformation. It’s a significant task but offers long-term advantages. Gradually migrate core data, mirroring the current data warehouse structure in the gold layer, running parallel systems initially.

Andy
What about setting up a new data warehouse?

Tom
For new setups, consider both options. A lake house is better for handling large data volumes, but a traditional data warehouse might be simpler for smaller organizations with existing SQL skills.

Andy
Anything else to add?

Shailan
Consider modern architecture options like Fabric, with updated pricing. We can help assess your data needs and recommend solutions.

Andy
Thanks, everyone. This podcast will be out on December 18th. Enjoy the festive season and see you next time.

Tom & Shailan
Cheers! Have a great Christmas.