What is Apache Iceberg?
Apache Iceberg is an open table format designed for huge analytic datasets. Originally developed at Netflix, it's now an Apache top-level project used by Apple, LinkedIn, Adobe, and hundreds of enterprises.
For data archiving, Iceberg solves fundamental problems that have plagued enterprises for decades.
Why Iceberg Matters for Archiving
The Legacy Archive Problem
Traditional archive solutions suffer from:
- Proprietary formats — data locked in vendor-specific formats
- No query access — archives are "write once, read never"
- Schema rigidity — can't evolve schema without rewriting data
- Vendor lock-in — expensive migrations if vendor EOLs product
The Iceberg Solution
Iceberg provides:
- Open format — your data, your control, forever
- SQL queryable — query archives with standard SQL
- Schema evolution — add/remove columns without rewriting
- Engine agnostic — works with Spark, Trino, Flink, Athena, and more
Key Iceberg Features for Archiving
1. Time Travel
Query data as it existed at any point in time:
```sql
-- Query orders as they were on Jan 1, 2024
SELECT * FROM orders
FOR TIMESTAMP AS OF '2024-01-01 00:00:00';
-- Query using snapshot ID
SELECT * FROM orders
FOR VERSION AS OF 1234567890;
```
Use cases:
- Audit and compliance queries
- Point-in-time reporting
- Debugging data issues
- Legal discovery
2. Schema Evolution
Add, drop, rename, or reorder columns without rewriting data:
```sql
-- Add a new column
ALTER TABLE orders ADD COLUMN region STRING;
-- Rename a column
ALTER TABLE orders RENAME COLUMN cust_id TO customer_id;
-- Queries on old data still work!
```
Why it matters for archives:
- Source systems evolve over time
- Archives span years or decades
- No expensive data migrations
3. Partition Evolution
Change partitioning without rewriting data:
```sql
-- Originally partitioned by year
-- Add month partitioning for recent data
ALTER TABLE orders
ADD PARTITION FIELD month(order_date);
```
Benefits:
- Optimize query performance over time
- Handle changing access patterns
- No downtime or data movement
4. Hidden Partitioning
Users don't need to know partitioning scheme:
```sql
-- Iceberg automatically prunes partitions
SELECT * FROM orders
WHERE order_date BETWEEN '2024-01-01' AND '2024-03-31';
-- Only reads relevant partitions
```
Why it matters:
- Business users write simple queries
- No partition key in WHERE clause required
- Optimizer handles efficiency
5. ACID Transactions
Full ACID compliance for archive operations:
- Atomic writes — all-or-nothing archive batches
- Consistent reads — no partial results during writes
- Isolated operations — concurrent queries don't conflict
- Durable storage — data persists reliably
Iceberg vs. Other Formats
Building Archives with Iceberg
Architecture Overview
```
Source Systems → Ingestion → Iceberg Tables → Query Engines
Databases CDC/Batch S3 + Metadata Athena
Files Spark Parquet files Trino
APIs Spark
```
Storage Layout
Iceberg stores data and metadata separately:
```
s3://archive-bucket/
├── orders/
│ ├── data/
│ │ ├── year=2023/
│ │ │ ├── 00001.parquet
│ │ │ └── 00002.parquet
│ │ └── year=2024/
│ │ └── 00001.parquet
│ └── metadata/
│ ├── v1.metadata.json
│ ├── v2.metadata.json
│ └── snap-123456.avro
```
Benefits:
- Metadata tracks all changes
- Data files are immutable
- Easy backup and recovery
Implementing Retention
```sql
-- Remove snapshots older than 90 days
CALL system.expire_snapshots(
table => 'orders',
older_than => TIMESTAMP '2024-01-01'
);
-- Remove orphan files
CALL system.remove_orphan_files(
table => 'orders'
);
```
Cost Benefits of Iceberg Archives
Storage Efficiency
Query Efficiency
Cost impact: 90%+ reduction in query costs
Getting Started with Iceberg
Step 1: Create a Catalog
Use AWS Glue, Hive Metastore, or Iceberg REST catalog:
```sql
CREATE DATABASE archive_db;
```
Step 2: Create Archive Table
```sql
CREATE TABLE archive_db.orders (
order_id BIGINT,
customer_id BIGINT,
order_date DATE,
total_amount DECIMAL(10,2),
status STRING,
archived_at TIMESTAMP
)
USING iceberg
PARTITIONED BY (year(order_date))
LOCATION 's3://archive-bucket/orders/';
```
Step 3: Archive Data
```sql
INSERT INTO archive_db.orders
SELECT *, current_timestamp() as archived_at
FROM production.orders
WHERE order_date < date_sub(current_date(), 365);
```
Step 4: Query Archives
```sql
-- Simple query
SELECT * FROM archive_db.orders
WHERE customer_id = 12345;
-- Time travel query
SELECT * FROM archive_db.orders
FOR TIMESTAMP AS OF '2024-06-01'
WHERE customer_id = 12345;
```
Conclusion
Apache Iceberg transforms data archiving from a necessary evil into a strategic asset. With time travel, schema evolution, and broad engine support, archives become queryable, evolvable, and future-proof.
ArchiveHouse uses Iceberg as its foundation, giving you all these benefits without the complexity of managing Iceberg infrastructure yourself.
Ready to build a modern archive? Request a demo to see Iceberg-powered archiving in action.