010203040506070809101112131415
ALL MODULESmodule_15.py

Working with JSON, CSV & Structuring Data

~45 minTheoryLive CodingChallengeQuiz
theory.md

12. Data Formats, Serialization Engines & Enterprise ETL Pipelines

12.1Data Serialization Foundations

When an application wants to send data across a network (to a mobile app or server) or save it to a file, Python objects living in memory (like dictionaries or custom classes) cannot travel directly through network cables.

  • Data Serialization: Converting live in-memory Python objects into a flat, standardized text or binary format (like JSON or CSV) that any programming language can read.
  • Data Deserialization: The reverse process—parsing flat JSON/CSV text back into live Python memory objects!
  • Analogy Time: Think of serialization like taking apart a wooden chair and flat-packing it into a cardboard box for shipping. Deserialization is assembling the chair back together when it arrives at your house!

12.2JSON Mechanics (RFC 8259) & Custom Encoders

JSON (JavaScript Object Notation) is the universal text standard for web APIs worldwide. Python provides the built-in json module (json.dumps() to serialize to text, json.loads() to deserialize from text).

  • Handling Custom Objects: Standard JSON only understands primitives (int, float, str, bool, list, dict). Objects like datetime raise a TypeError. To serialize them, we create a custom json.JSONEncoder class!

12.3Safe CSV Processing (csv.DictReader / csv.DictWriter)

CSV (Comma-Separated Values) files store tabular spreadsheet data in plain text.

  • Why avoid string splitting? Never parse CSVs by manually splitting lines by commas (line.split(","))! If a cell contains a quoted comma (e.g. "Austin, TX"), string splitting will slice the data incorrectly.
  • Always use Python's built-in csv.DictReader and csv.DictWriter modules to handle quotes, commas, and headers safely!

12.4Enterprise ETL Pipeline Architecture

ETL stands for Extract, Transform, Load—the foundational pipeline architecture used in data engineering:

  1. Extract: Fetch raw data from external sources (CSV files, database queries, web APIs).
  2. Transform: Clean, validate, filter, and format the data.
  3. Load: Write the clean processed data into its final destination (JSON file, data warehouse, database).

12.5Code Demo 1: JSON Serialization with Custom Serializer

Serialize Python datetime objects cleanly into JSON:

PYTHON

12.6Code Demo 2: Complete CSV-to-JSON ETL Data Pipeline

A complete pipeline extracting CSV rows, transforming data, and loading JSON output:

PYTHON
GOTCHA 1

Confusing JSON syntax with Python dict syntax: writing true or null inside Python code (raises NameError), or expecting JSON string keys to be unquoted.

GOTCHA 2

Attempting to serialize non-standard Python objects (like datetime, set, or custom class instances) using json.dumps() without providing a custom default serialization handler. Raises TypeError: Object of type X is not JSON serializable.

GOTCHA 3

Parsing CSV files manually using line.split(',') instead of csv.reader or csv.DictReader. Manual splitting fails on quoted strings containing commas.

GOTCHA 4

Forgetting that csv.reader and csv.DictReader return ALL values as strings. You must explicitly cast numerical fields using int() or float().

WHERE THIS SHOWS UP IN REAL CODE

Modern web APIs communicate via JSON payloads, while business intelligence platforms export analytics data as CSV files. Python developers spend significant time building ETL (Extract, Transform, Load) pipelines to move data between these formats.

TRY THIS YOURSELF

Modify Demo 1 to add a custom key 'verified': True and serialize using json.dumps(..., sort_keys=True).

live_coding — JSON Configuration Merger & CSV Report Generator

Notice how easy it is to bridge JSON objects into CSV rows using DictWriter.

Python not loaded yet.
demo — Demonstrate JSON parsing/serialization, csv.DictReader/DictWriter, and ETL pipelines.

Demonstrate JSON parsing/serialization, csv.DictReader/DictWriter, and ETL pipelines.

challenge_1

Let's merge two JSON data strings into one object!

Steps to Complete:

  1. Given json1 = '{"a": 10, "b": 20}' and json2 = '{"b": 30, "c": 40}'.
  2. Parse both into dicts using json.loads(), merge dicts (json2 updates json1).
  3. Serialize merged dict back to JSON text with json.dumps() and print Merged JSON: {"a": 10, "b": 30, "c": 40}.

Expected Output:

TEXT
Python not loaded yet.
challenge_2

Let's parse a CSV string and calculate total scores!

Steps to Complete:

  1. Given `csv_data = 'name,score

Alice,90 Bob,80'`.

  1. Use csv.DictReader to parse lines and sum scores as integers.
  2. Print Total Score: 170.

Expected Output:

TEXT
Python not loaded yet.
challenge_3

Let's convert raw CSV records into a typed JSON array!

Steps to Complete:

  1. Given `csv_text = 'id,active

1,true 2,false'`.

  1. Parse rows with csv.DictReader, converting id to int and active to boolean.
  2. Dump to JSON string and print JSON Output: [{"id": 1, "active": true}, {"id": 2, "active": false}].

Expected Output:

TEXT
Python not loaded yet.
quiz — 6 questions
30s question timers are paused. Start individually or all together.
# question 1

What is the difference between `json.loads()` and `json.load()` in Python?

# question 2

How is the JSON boolean `true` deserialized into Python by default?

# question 3

Why should you use `csv.DictReader` instead of manual `line.split(',')` when parsing CSV files?

# question 4

What error is raised when trying to serialize a custom class instance with `json.dumps()` without a custom encoder?

# question 5

What data type do `csv.reader` and `csv.DictReader` return for all row field values by default?

# question 6

Which parameter in `json.dumps(data, indent=2)` controls pretty-printing for human readability?

discussion

# comments · data-formats

  • @priya · 2h ago

    The typewriter hero is such a nice touch.

    • @alex · 1h ago

      Totally agree! Made it super engaging.

  • @sam · 1d ago

    Finally a course where I can run code without leaving the page.

# further_reading