[Avg. reading time: 2 minutes]
Medallion Architecture
This is also called as Multi-Hop architecture.
Bronze Layer (Raw Data)
- Append-only ingestion
- No business logic
- Schema minimally enforced
- Supports replay / backfill
Silver Layer (Cleansed and Conformed Data)
- Deduplication
- Joins / normalization
- Schema enforcement
- Basic data quality checks
Gold Layer (Curated Business-level tables)
- Business logic
- Aggregations
- KPI tables
- Semantic-ready datasets
(Many Inputs)
Kafka APIs Files DBs Streams
\ | | | /
\ | | | /
\ | | | /
\ | | | /
▼ ▼ ▼ ▼ ▼
╔══════════════════════╗
║ BRONZE ║ ← Wide ingest funnel
║ (Raw / Append-only)║
╚══════════════════════╝
│
│ (filter, dedupe, schema fix)
▼
╔══════════════════════╗
║ SILVER ║ ← Compression layer
║ (Clean / Conformed) ║
╚══════════════════════╝
│
│ (business logic, joins)
▼
╔══════════════════════╗
║ GOLD ║ ← High-value core
║ (Aggregated / KPI) ║
╚══════════════════════╝
│
┌────────────┼────────────┬────────────┬────────────┐
▼ ▼ ▼ ▼ ▼
BI / SQL ML Features APIs Reverse ETL Real-time Apps
(Dashboards) (Feature Store) (Serving) (Salesforce) (Streams)
Different Personas involved
- Data Engineer
- Data Analysts
- Data Scientists