Data Intensive
Data-intensive / data-platform architecture: warehouse vs lakehouse, OLTP vs OLAP, batch vs strea…
by claude-dev-suite · MIT
The skill
Data-intensive / data-platform architecture: warehouse vs lakehouse, OLTP vs OLAP, batch vs streaming (lambda/kappa), change-data-capture, and data mesh. Architect-level platform-shape decisions, not SQL or ETL code. USE WHEN: designing a data platform/pipeline architecture, "data warehouse", "lakehouse", "OLAP vs OLTP", "lambda/kappa", "streaming vs batch", "CDC", "data mesh", "medallion", analytics platform, columnar store, ingestion topology. DO NOT USE FOR: writing SQL/ORM (use database skills); a specific ETL tool (use data/data-processing skills); storage-engine internals (use `storage-engines`).
Data-Intensive Architecture
OLTP vs OLAP — separate the workloads
- OLTP: many small row-oriented transactions, low latency, normalized.
- OLAP: few large column-oriented scans/aggregations, throughput-oriented. Don't run heavy analytics on the OLTP store — replicate/ETL/CDC into an analytics store. Columnar formats (Parquet/ORC) + columnar engines win OLAP.
Storage architecture choice
| Option | Idea | Fits |
|---|---|---|
| Warehouse (Snowflake, BigQuery, Redshift) | Managed columnar SQL store | Structured BI, governance, SQL-first |
| Data lake (object storage + files) | Cheap, schema-on-read, any format | Raw/varied data, ML feature sources |
| Lakehouse (Delta/Iceberg/Hudi on object storage) | Lake economics + ACID tables + time travel | Unified BI + ML, open formats, avoids lock-in |
Lakehouse (open table formats) is the common 2026 default when you want one copy of data serving both BI and ML without warehouse lock-in.
Batch vs streaming topology
- Batch: periodic, simple, high-latency. Streaming (Kafka/Flink): continuous, low-latency, more complex (exactly-once, watermarks, state).
- Lambda: batch layer (accurate) + speed layer (fresh) merged — two codebases, complexity. Kappa: stream-only, reprocess from the log — simpler; prefer it unless you truly need a separate batch layer.
- CDC (Debezium): stream row changes out of OLTP without dual-writes — the clean way to feed lake/warehouse/search in near-real-time.
Modeling & ownership
- Medallion (bronze/silver/gold) layering for lake/lakehouse refinement.
- Data mesh: domain-owned data products + federated governance — organizational, for large orgs; overkill for small teams (start centralized).
When to recommend what
- BI on structured data, SQL team → warehouse (or lakehouse w/ SQL engine).
- Mixed BI + ML, open formats, scale → lakehouse (Iceberg/Delta) + Kappa + CDC.
- Real-time decisions → streaming (Kafka + Flink), exactly-once where it matters.
- Don't reach for data mesh until org scale forces decentralization.
Use this skill
Connect your agent to the Skill Store MCP server and ask for it by ref: claude-dev-suite/data-intensive-310314681.
Details
- Publisher
- claude-dev-suite
- Category
- Data & Analytics › Data pipelines & ETL
- License
- MIT
- Updated
- 2026-10-19
References
More from claude-dev-suite
- Axe Core axe-core - accessibility testing engine for automated WCAG compliance checks.
- Wcag WCAG 2.2 Web Content Accessibility Guidelines.
- Anthropic Python Anthropic Python SDK for Claude API integration.
- Decision Model Calibration Turning a model's class probabilities into numbers you can set a threshold on: building the label…