Function Chains vs. Pipelines

2025-02-03

Feb 04, 2025

Overview

There is a deep, fundamental difference between programming with functions and programming with pipelines of isolated components.

In an attempt to illustrate this difference in an understandable form, I used an LLM (Claude 3.5 Sonnet). I posed a series of questions and copied the answers into an appendix (a separate PDF file) to this article.

The key conclusion is that what we call “programming” today should not be done with programming languages which are based solely on the concept of functions, i.e. currently popular programming languages, like Rust, Python, Javascript, etc. These languages are not suitable for programming in the modern day. We see this mismatch leaking out in the form of syntactic baubles added to these languages, like ‘await’, ‘.then’, ‘try...except’, etc.

I stick to discussing “pipelines” in this article and appendix, but, complete isolation1 allows more architectural configurations than just simple, sequential pipelines. One can imagine multiple inputs and multiple outputs and feedback.

TL;DR

Below, I’ve listed some key points to consider. These points have been directly cut/pasted from Claude’s responses. The full text of the responses can be found in the Appendix.

A chain of functions is not the same as a pipeline, though they can appear superficially similar.
Function-based ordering can be a major source of bugs.
A pipeline isn't just syntactic sugar for function composition - it's a fundamentally different abstraction for data flow.
The key insight is that pipelines are about modelling data flow, while function chains are about control flow - they're different abstractions serving different purposes.
The key difference isn't that pipelines lack control flow - it's that they encapsulate it within a standardized pattern.
A function call passes both control flow and data.
In traditional function calls, the caller blocks and waits for the callee to return - this is indeed synchronous operation.
A pipeline only passes data, not control flow.
While both pipelines and chained functions compose operations sequentially, there are key structural differences.
The individual pipeline stages don't directly control program flow like function calls do.
Pipeline nodes do NOT determine which code runs next [pt: whereas functions do]
The key difference is that pipeline order depends solely on queue state, while function order depends on the call stack at each point.
The key insight is that real-world systems tend to be naturally asynchronous and concurrent. Function-based control flow, which is inherently sequential, often fights against this reality. Pipeline-based data flow better matches how these systems actually work.
While C didn't invent functions, it did standardize a particular model of function calls that became pervasive in programming. This may have made it harder for programmers to think in terms of alternative models like pipelines, even when they might be more appropriate.
Ordering is critically important in modern programming for several reasons.
Most mainstream programming languages are fundamentally built around functions, which is becoming increasingly misaligned with modern problem spaces.
This suggests we might need fundamental changes in programming language design to better match contemporary computing realities.
Textual representation becomes inadequate for complex architectures.
We're largely still programming with a model designed for sequential, single-machine computation from the 1970s.

Appendix

I include the Q&A with Claude as a PDF.

2025 02 03 Appendix For Function Chains Vs

232KB ∙ PDF file

Download

Programming Simplicity

Discussion about this post

Ready for more?