Fundamentals

Structured vs Semi-Structured vs Unstructured Data In the last post, we talked about data pipelines and how data flows from source to storage. But here’s something important: not all data looks the same. Some data is clean and organized, some is messy but has a pattern, and some is just… chaos. That’s where structured, semi-structured, and unstructured data come in. And this distinction is the entire reason why data warehouses and data lakes are built differently. Think of it like organizing a filing system Imagine you’re in charge of filing documents for a company. Structured data is like a perfectly organized file cabinet. Every file has the same format: name, date, department, and amount. You know exactly where everything goes, and you can quickly find information. “How much did we spend in sales last month?” — you can answer that in seconds by looking at your organized files. Semi-structured data is like a folder of emails. Emails have a subject, sender, date, and content — but some emails have attachments, some don’t. Some have multiple recipients, some don’t. There’s a structure, but it’s flexible. Unstructured data is like a box of printed photographs and handwritten notes. Sure, they all came from your company, but there’s no consistent format. Some photos have dates written on the back, some don’t. Some notes are one line, others are pages long. You can read them, but you can’t instantly summarize them. Same company, same filing system, three very different types of information. ...

What Is a Data Pipeline, Really? By now, you’ve probably heard the terms ETL vs ELT, you know about data warehouses and data lakes, and you understand batch vs streaming processing. But here’s the thing: those are all just pieces. A data pipeline is how all these pieces fit together into one complete picture, from the moment data is born all the way to the moment someone uses it to make a decision. Think of it like a pizza delivery system Imagine you’re running a pizza restaurant, and you want to track everything that happens to every pizza. Raw data is all the orders coming in: what toppings, delivery address, time ordered. Extraction is grabbing those orders from the phone, email, and app: getting all the information out of wherever it lives. Transformation is organizing all that information into a standard format: customer name, address, toppings list, price, delivery time. Storage is putting that organized information in a safe place, maybe a database where you can look it up later. Analysis is looking at all the pizza data and asking questions: “What’s our most popular topping? Are we getting faster at deliveries? Which neighborhoods order the most?” That entire journey, from someone calling to order a pizza to you understanding your business better, that’s a data pipeline. It’s the whole assembly line, not just one machine in it. ...

What Is a Data Pipeline, Really? By now, you’ve probably heard the terms ETL vs ELT, you know about data warehouses and data lakes, and you understand batch vs streaming processing. But here’s the thing: those are all just pieces. A data pipeline is how all these pieces fit together into one complete picture — from the moment data is born all the way to the moment someone uses it to make a decision. Think of it like a pizza delivery system Imagine you’re running a pizza restaurant, and you want to track everything that happens to every pizza. Raw data is all the orders coming in: what toppings, delivery address, time ordered. Extraction is grabbing those orders from the phone, email, and app — getting all the information out of wherever it lives. Transformation is organizing all that information into a standard format: customer name, address, toppings list, price, delivery time. Storage is putting that organized information in a safe place — maybe a database where you can look it up later. Analysis is looking at all the pizza data and asking questions: “What’s our most popular topping? Are we getting faster at deliveries? Which neighborhoods order the most?” That entire journey — from someone calling to order a pizza to you understanding your business better — that’s a data pipeline. It’s the whole assembly line, not just one machine in it. ...

How text gets turned into numbers that capture meaning, and why ‘similar meaning = nearby numbers’ is the foundation of search, recommendations, and RAG.

Why specificity beats cleverness, and how to talk to a model that takes everything you say literally.

What a token actually is, why every model has a memory limit, and why this quietly affects cost, speed, and what you can even ask a model to do.

No math-heavy transformer internals, just the core idea: an LLM is a very sophisticated autocomplete engine trained on huge amounts of text.

What You’ll Learn What a data engineer actually does, in plain terms Why companies need this role at all How it’s different from a data analyst or data scientist What a “day in the life” roughly looks like Why This Matters Imagine a company collects millions of orders, clicks, and support tickets every day, scattered across apps, databases, and spreadsheets. Someone has to gather all of that, clean it up, and organize it so the rest of the company can actually use it. That “someone” is a data engineer. Without this role, every report, dashboard, and AI model would be built on messy, unreliable data. The Simple Explanation A data engineer builds and maintains the systems that move data from where it’s created to where it’s used. Think of raw data as ingredients scattered across a farm, a warehouse, and a truck. A data engineer is like the person who collects those ingredients, cleans them, and delivers them to the kitchen, ready for the chef (a data analyst or data scientist) to cook something useful with them. 💡 Analogy: If a company’s data is a city’s water supply, the data engineer builds and maintains the pipes, treatment plants, and pumps. Analysts and scientists are the people who turn on the tap and use the water, they rarely think about the plumbing, but it has to work perfectly for them to do their job. ...

What You’ll Learn What a data engineer actually does, in plain terms Why companies need this role at all How it’s different from a data analyst or data scientist What a “day in the life” roughly looks like Why This Matters Imagine a company collects millions of orders, clicks, and support tickets every day — scattered across apps, databases, and spreadsheets. Someone has to gather all of that, clean it up, and organize it so the rest of the company can actually use it. That “someone” is a data engineer. Without this role, every report, dashboard, and AI model would be built on messy, unreliable data. The Simple Explanation A data engineer builds and maintains the systems that move data from where it’s created to where it’s used. Think of raw data as ingredients scattered across a farm, a warehouse, and a truck. A data engineer is like the person who collects those ingredients, cleans them, and delivers them to the kitchen — ready for the chef (a data analyst or data scientist) to cook something useful with them. 💡 Analogy: If a company’s data is a city’s water supply, the data engineer builds and maintains the pipes, treatment plants, and pumps. Analysts and scientists are the people who turn on the tap and use the water — they rarely think about the plumbing, but it has to work perfectly for them to do their job. ...

The simple difference between predictive AI and generative AI, explained without the jargon.