Mid-level (2-5 years)Python
What are generators and the yield keyword in Python?
Quick answer
A generator is a function that uses yield to produce values one at a time and remembers its state between calls, so it can process large or infinite sequences without holding everything in memory.
Calling a generator function does not run it; it returns an iterator. Each next() call resumes the function until the next yield. A list of ten million numbers takes hundreds of megabytes, while a generator producing them one by one uses almost none.
A generator expression, (x * x for x in range(10**7)), is the lazy counterpart of a list comprehension. Generators can only be consumed once. They are ideal for reading big files line by line, streaming API pages and building data pipelines.
def read_large_file(path):
with open(path) as f:
for line in f:
yield line.strip()
for row in read_large_file("big.csv"):
process(row) # one line in memory at a time
total = sum(x * x for x in range(1_000_000)) # no list createdKey points
- yield pauses the function and keeps local state
- Lazy evaluation saves memory
- A generator can be iterated only once