Another tef gem on pipelines


tef has written another great post about pipelines, and in particular why (and how!) not to use message queues to implement them:

how (not) to write a pipeline [tef, cohost]

The tl;dr is that your “background job” is really a state machine. Even the simplest possible job:

def do_something(inputs) -> outputs:
    # do something

Is a state machine with 4 possible states:

  • enqueued
  • started
  • succeeded
  • failed

And a simple pipeline based on a message broker almost certainly does not support:

  • navigating that state machine correctly, including error detection
  • back pressure
  • easy-to-use smoke detectors and fire extinguishers for operating the service

So bite the bullet and implement this using a proper database table (or equivalent) to track the state of each job!

You might still end up with a queue, but as an optimization, not as a load-bearing part of the design:

the queue buffers the results of a more expensive database query


Distributed systems » Pipelines