List comprehensions vs generator expressions — when doe...

The difference is memory, and it’s a bigger deal in ML than it sounds. A list comprehension evaluates everything up front and holds it in memory, which means you can index into it and iterate over it as many times as you like. A generator expression produces one value at a time, on demand, and once you’ve consumed it, it’s gone — you can’t loop over it twice.

That constraint is exactly why generators matter for data loading: nobody has enough RAM to materialize 10GB of images as a Python list. Instead you write a generator that yields one batch at a time, and the training loop consumes it lazily, batch by batch. For small, reusable arrays — say, a quick feature transform you’ll inspect and reuse — a list comprehension is simpler and just as fast. The pattern to internalize: comprehensions for in-memory feature work, generators for anything that has to stream.