Data Engineering Basics

Batch vs Streaming Processing Explained Simply If you’ve read Data Warehouse vs Data Lake vs Lakehouse, you know where data can live once it’s ready for analysis. But before it gets there, it has to be processed, and that raises another fundamental question, just as important as OLTP vs OLAP: when does that processing happen? That’s where batch processing and streaming processing come in. Think of it like doing laundry Imagine you’re in charge of laundry for a household. Batch processing is doing laundry once a week. You wait until there’s a full load, throw it all in the machine together, and process it in one go. Efficient, predictable, and you know exactly when it’ll be done, but if someone needs a clean shirt right now, they’re out of luck until the next load. Streaming processing is washing each item the moment it gets dirty. Nothing waits around: a shirt gets dirty, it gets washed immediately, one item at a time. Nobody waits for laundry day, but running the washing machine constantly, for one item at a time, is a lot less efficient than doing a full load. Same idea with data: do you wait and process it in chunks, or handle each piece the instant it arrives? ...

Batch vs Streaming Processing Explained Simply If you’ve read Data Warehouse vs Data Lake vs Lakehouse, you know where data can live once it’s ready for analysis. But before it gets there, it has to be processed — and that raises another fundamental question, just as important as OLTP vs OLAP: when does that processing happen? That’s where batch processing and streaming processing come in. Think of it like doing laundry Imagine you’re in charge of laundry for a household. Batch processing is doing laundry once a week. You wait until there’s a full load, throw it all in the machine together, and process it in one go. Efficient, predictable, and you know exactly when it’ll be done — but if someone needs a clean shirt right now, they’re out of luck until the next load. Streaming processing is washing each item the moment it gets dirty. Nothing waits around — a shirt gets dirty, it gets washed immediately, one item at a time. Nobody waits for laundry day, but running the washing machine constantly, for one item at a time, is a lot less efficient than doing a full load. Same idea with data: do you wait and process it in chunks, or handle each piece the instant it arrives? ...

Data Warehouse vs Data Lake vs Lakehouse Explained Simply If you’ve read OLTP vs OLAP and ELT vs ETL, you know that data eventually needs to land somewhere it can be analyzed. But “somewhere” isn’t one thing: there are actually three common answers: a data warehouse, a data lake, or a lakehouse. These terms get thrown around a lot, often interchangeably, which makes them confusing. Let’s fix that with a simple analogy. Think of it like storing food Imagine you’re in charge of storing food for a restaurant. A data warehouse is like a pantry with labeled shelves. Everything is pre-sorted, cleaned, and organized into containers. You know exactly where the flour is, and it’s always in the same jar, in the same format. Easy to grab and use, but someone had to do the work of sorting it first, and you can only store what fits the pantry’s shelving system. A data lake is like a giant walk-in cooler where you just throw in whatever arrives: whole vegetables, sealed meat, unlabeled boxes from a supplier. Nothing is sorted. It’s flexible and cheap to just dump things in, but finding what you need, or trusting what condition it’s in, takes more work. A lakehouse is like a walk-in cooler that also has some shelving and labeling built in. You get the flexibility of storing anything, but with enough structure that you can still find and trust what’s there. Now let’s translate that into actual data engineering terms. ...

Data Warehouse vs Data Lake vs Lakehouse Explained Simply If you’ve read OLTP vs OLAP and ELT vs ETL, you know that data eventually needs to land somewhere it can be analyzed. But “somewhere” isn’t one thing — there are actually three common answers: a data warehouse, a data lake, or a lakehouse. These terms get thrown around a lot, often interchangeably, which makes them confusing. Let’s fix that with a simple analogy. Think of it like storing food Imagine you’re in charge of storing food for a restaurant. A data warehouse is like a pantry with labeled shelves. Everything is pre-sorted, cleaned, and organized into containers. You know exactly where the flour is, and it’s always in the same jar, in the same format. Easy to grab and use — but someone had to do the work of sorting it first, and you can only store what fits the pantry’s shelving system. A data lake is like a giant walk-in cooler where you just throw in whatever arrives — whole vegetables, sealed meat, unlabeled boxes from a supplier. Nothing is sorted. It’s flexible and cheap to just dump things in, but finding what you need — or trusting what condition it’s in — takes more work. A lakehouse is like a walk-in cooler that also has some shelving and labeling built in. You get the flexibility of storing anything, but with enough structure that you can still find and trust what’s there. Now let’s translate that into actual data engineering terms. ...

ELT vs ETL: What’s the Difference? If you’ve read about OLTP vs OLAP, you know that data usually starts life in an OLTP system (like the database behind an app) and needs to make its way into an OLAP system (like a data warehouse) so people can analyze it. The question is: how does that data get there? That’s where ETL and ELT come in. They’re two strategies for moving and preparing data, and the difference comes down to one thing: when you transform the data. The three steps Both approaches share the same three ingredients: Extract: pull the data out of the source system (a database, an API, a file, etc.) Transform: clean it, reshape it, join it, aggregate it, turn raw data into something useful Load: put the data into its destination (usually a data warehouse) The letters are the same. The order is not. And that small difference in order changes a lot about how the pipeline works. ETL: Extract, Transform, Load In ETL, you transform the data before it reaches the warehouse. flowchart LR A[Source System] --> B[Extract] B --> C[Transform<br/>separate processing engine] C --> D[Load] D --> E[(Warehouse)] The transformation happens in a separate system (often a dedicated ETL tool or a processing cluster) sitting between the source and the destination. By the time data lands in the warehouse, it’s already clean, structured, and ready to query. ...

ELT vs ETL: What’s the Difference? If you’ve read about OLTP vs OLAP, you know that data usually starts life in an OLTP system (like the database behind an app) and needs to make its way into an OLAP system (like a data warehouse) so people can analyze it. The question is: how does that data get there? That’s where ETL and ELT come in. They’re two strategies for moving and preparing data — and the difference comes down to one thing: when you transform the data. The three steps Both approaches share the same three ingredients: Extract — pull the data out of the source system (a database, an API, a file, etc.) Transform — clean it, reshape it, join it, aggregate it — turn raw data into something useful Load — put the data into its destination (usually a data warehouse) The letters are the same. The order is not. And that small difference in order changes a lot about how the pipeline works. ETL: Extract, Transform, Load In ETL, you transform the data before it reaches the warehouse. flowchart LR A[Source System] --> B[Extract] B --> C[Transform<br/>separate processing engine] C --> D[Load] D --> E[(Warehouse)] The transformation happens in a separate system — often a dedicated ETL tool or a processing cluster — sitting between the source and the destination. By the time data lands in the warehouse, it’s already clean, structured, and ready to query. ...

OLTP vs OLAP Explained Simply If you’re new to data engineering, you’ll hear these two acronyms constantly: OLTP and OLAP. They sound like jargon, but the idea behind them is simple. Once you get it, a lot of other things in data engineering (why we build data warehouses, why we copy data out of production databases, why star schemas exist) will start to make sense. Let’s break it down. Two very different jobs Imagine two people working at a bank. The first person is a teller. Someone walks up, deposits $200, and walks away. Then the next person withdraws $50. Then someone opens a new account. Each of these is a small, quick transaction. The teller doesn’t care about the bank’s history; they care about doing this one transaction, correctly, right now. The second person is an analyst. At the end of the month, they ask questions like: “What was our average account balance across all branches in Amsterdam over the last year?” This isn’t one small task; it’s a question that touches millions of past transactions to produce one answer. These two jobs need different tools. That’s the whole idea behind OLTP and OLAP. OLTP: Online Transaction Processing OLTP systems are built for the teller’s job: many small, fast operations happening constantly. ...

OLTP vs OLAP Explained Simply If you’re new to data engineering, you’ll hear these two acronyms constantly: OLTP and OLAP. They sound like jargon, but the idea behind them is simple. Once you get it, a lot of other things in data engineering — why we build data warehouses, why we copy data out of production databases, why star schemas exist — will start to make sense. Let’s break it down. Two very different jobs Imagine two people working at a bank. The first person is a teller. Someone walks up, deposits $200, and walks away. Then the next person withdraws $50. Then someone opens a new account. Each of these is a small, quick transaction. The teller doesn’t care about the bank’s history — they care about doing this one transaction, correctly, right now. The second person is an analyst. At the end of the month, they ask questions like: “What was our average account balance across all branches in Amsterdam over the last year?” This isn’t one small task — it’s a question that touches millions of past transactions to produce one answer. These two jobs need different tools. That’s the whole idea behind OLTP and OLAP. OLTP: Online Transaction Processing OLTP systems are built for the teller’s job: many small, fast operations happening constantly. ...