Generator

Generators in Python are essentially functions that using “yield” keyword instead of “return”. The trick is that “yield” does not end the function and a function can “yield” infinitely many things. The main down side is that we can’t simply call the generator to access all the output. Instead, we can iterate over the output inside a loop. The main advantage of a generator is the fact that its output is generated on demand. Normally, the variables that we defined need a place in memory to store it. Imagine a huge list contains millions of items inside, That requires a huge amount of memory to store and it will wake a long time to iterate it. But the output of a generator is not stored in memory by default, therefore, it would save tons of memory when dealing with large amounts of data.

Create a generator

In Python, a generator works like a function. So we can simply use “def” to define a generator. Inside the function, we need to use “yield” instead of “return” to indicate the items that we want to yield. If the body of a def contains yield, the function automatically becomes a Python generator function. The syntax of a generator is:

def generator_name():
    yield statement

For example, if we want to yield the first positive 5 even integers. Instead of calling the generator directly to get the output, we need to iterate it.

# define a generator
def my_gen():
  for i in range(1, 6):
    yield 2 * i

# access the output
for i in my_gen():
  print(i)

Output:

2
4
6
8
10

Instead of returning a specific value, the generator returns a iterable object. That means we can also iterate the generator using next() function.

# define a generator
def my_gen():
  for i in range(1, 6):
    yield 2 * i

# iterate using next()
gen = my_gen()
print(next(gen))
print(next(gen))
print(next(gen))

Output:

2
4
6

When dealing with an extremely larger data set and trying to access it, the program might be interrupted by Python. However, we can make a generator which will generate the same numbers on demand. Since the numbers aren’t stored in memory, python doesn’t complain about the giant range being used.

# an example of a list that is possibly too large to fit in memory
L = [2*n for n in range(2**40)]

# using generator
def my_gen():
    for n in range(2**40):
        yield 2*n

Generators can also be used for recursive sequences. For example, we can use generator to generate all Fibonacci Numbers. Compare with recursion, generator will be much faster.

# a generator for all Fibonacci numbers
def fib_gen():
    a0 = 1
    a1 = 1
    while True:
        yield a0
        a0, a1 = a1, a0+a1

We can actually turn any list comprehension into a generator, using parenthesis instead of square rackets.

# list comprehension:
L = [2*n for n in range(100)]

# generator expression:
G = (2*n for n in range(100))