Memory-Efficient Data Streaming with Generators in Python

Question

Explain how to read a 20GB log file in Python on a machine with only 4GB of RAM without crashing. Write a generator function that yields parsed JSON logs one by one.

Accepted Answer

Optimal Solution:
import json
from typing import Generator

def stream_json_logs(file_path: str) -> Generator[dict, None, None]:
    with open(file_path, 'r', encoding='utf-8') as f:
        for line in f:
            line = line.strip()
            if line:
                try:
                    yield json.loads(line)
                except json.JSONDecodeError:
                    continue

Technical Explanation:
The `yield` statement turns the function into a generator. File iterators read one line into memory at a time, maintaining constant O(1) memory usage regardless of file size.