Design a Data Lake

Problem Statement

Design a scalable Data Lake for a large e-commerce platform that ingests 10TB of data daily from various sources, including relational databases, application logs, and third-party APIs.

High Level Architecture

Here is the interactive architecture diagram. You can view the different components and how they interact.

Sample Analytics Query

Here is an example query:

-- Find the top 5 most active users in the data lake log
SELECT 
  user_id, 
  COUNT(*) as event_count 
FROM raw_events 
GROUP BY user_id 
ORDER BY event_count DESC 
LIMIT 5;

The architecture relies on cloud-native object storage, separated compute and storage, and a robust data catalog.

1. Ingestion Layer: Kafka for streaming, Airflow for batch.
2. Storage Layer: Amazon S3 / Google Cloud Storage.
3. Processing Layer: Apache Spark / Flink.
4. Serving Layer: Presto / Trino for ad-hoc querying.

Storage Formats

We will use columnar formats to optimize analytical queries.

-- Example queries run faster on Parquet
SELECT user_id, count(*) 
FROM page_views 
WHERE date = '2026-06-14' 
GROUP BY user_id;

Deep Dive: Partitioning

Partitioning strategies are critical to avoid full table scans. We partition data by date and event_type.

Handling Late Data