Codestinger

What pyramids, bridges and beavers teach software architects

2026-08-30T16:33:00+00:00

Software architecture is a young discipline. Engineers have been building pyramids and bridges for thousands of years and nature has been refining structures for millions. When a design decision gets hard, I step outside software and ask how a beaver, a bridge builder or a submarine designer would solve it. History helps too, often most where things went badly wrong.

Lessons from nature#

Beavers: shape the environment, then maintain it. A beaver doesn't fight the river; it dams it into a pond that keeps its lodge safe. And it never stops maintaining: the sound of running water, a sign of a leak, sends it straight back to patch the dam. Design with the environment your system runs in (its rate limits, maintenance windows and quirks) and keep listening for leaks. Small, regular repairs outlast heroic rebuilds.

Bears: prepare for the lean season. A bear builds reserves in autumn and survives the winter at a slower pace. Build headroom, caches, queues and runbooks in calm times, so that during an outage your system can run in a reduced mode, protect what matters most and recover fully afterwards.

Otters: know your keystones. Sea otters keep a favourite stone as a tool and, as a keystone species, protect whole kelp forests. Master a small set of tools. And know the small components everything silently depends on (certificates, DNS records, scheduled jobs): they deserve the most care, not the least.

Orcas and whales: roles and experience. Orcas hunt as coordinated pods guided by experienced elders; humpback whales hunt together with bubble nets, each playing a part. Clear roles and shared protocols let a team achieve what no individual can.

Kingfishers: change the shape of the problem. Japan's bullet trains boomed as they left tunnels until engineers modelled the train's nose on the kingfisher's beak. They didn't add power; they changed the shape so the problem went away. Good code works the same way: rethink the problem until the special case disappears.

Lessons from the builders before us#

The pyramids: logistics wins. The Great Pyramid's real achievement was organisation: quarries, boats, crews and supplies coordinated for decades. One of the oldest papyri ever found is a work log from its construction. In software, the supply lines behind the code (pipelines, environments and clear ownership) decide whether it gets built at all. And write things down.

Japan: renew and bend. The Ise Grand Shrine is rebuilt every twenty years so each generation learns the craft. Pagodas survive earthquakes by swaying rather than resisting. Rebuild environments from code and rehearse recovery so knowledge stays alive. Design systems that bend and recover rather than break.

Bridges: leave margins. The Brooklyn Bridge's cables were designed six times stronger than needed. When substandard wire was found already woven in, that margin kept the bridge safe. It still carries traffic today. Software receives bad inputs all the time; headroom, timeouts and validation are its margins.

Submarines: contain the failure. Watertight compartments let a flooded section be sealed off while the boat survives. Software borrowed the idea as the bulkhead pattern: isolate each dependency so one failure can't sink the whole system.

Soviet engineering: beauty and proven designs. Moscow's metro stations were built like palaces for everyday commuters, a reminder that tools people use daily deserve care and beauty. The Soyuz rocket has flown for decades by improving one proven design carefully rather than replacing it.

The Second World War: strategy at scale. Studying the war is not admiring any side of it, but its strategy holds clear lessons. Mission command gave units the goal and trusted them with the how, which is how I like to lead teams. The Blitzkrieg principles translate well: combine every skill in one team, focus on the decisive point, go around the fortress instead of attacking it head-on (replace legacy systems piece by piece) and never outrun your supply lines. And the simpler tank that could be built and repaired in the tens of thousands beat the complex one built in the hundreds.

Chernobyl: never switch off the safeguards. The 1986 disaster happened during a rushed test with safety systems disabled and a design flaw operators had not been told about. Never test in production without safeguards, test the failure path, share known weaknesses and be honest after incidents.

The short version#

Design with your environment and keep listening for leaks.
Build reserves before the lean season.
Know your keystones and give them the most care.
Give everyone a clear role.
Change the shape of the problem before adding more force.
Invest in logistics and write things down.
Rebuild regularly and bend rather than break.
Leave margins and contain failures.
Make everyday tools beautiful and evolve proven designs.
Lead with intent, focus on the decisive point and never outrun your supply lines.
Never switch off the safeguards.

Architecture is a people job

2026-07-19T21:54:00+00:00

When people imagine a software architect, they usually picture diagrams: boxes, arrows, a cloud icon or two. Those matter. But the most important architecture work I do happens before the diagram exists, in conversations with the people who will build, deploy, sell, configure and live with the system.

A design that is technically elegant but awkward to deploy, hard to configure or impossible to explain has failed. It just hasn't found out yet.

Everyone sees a different system#

Put five people in a room and ask them what the product is. You'll get five honest and different answers.

Deployers see installers, configuration files, upgrade paths and rollbacks. Their question is: will this go in cleanly at a customer site and stay up?
Solution designers see building blocks. Their question is: can I shape this to fit this customer's process without asking for new code?
Product owners see value, priorities and deadlines. Their question is: what do we get, what does it cost and what are we giving up?
Developers see modules, dependencies and technical debt. Their question is: can we build this well and still understand it next year?
Clients see outcomes. Their question is: does it solve my problem reliably, every day?

None of these views is wrong and none is complete. An architect's job is to hold all of them at once and find the design that serves them together.

Listen before you draw#

Before I propose anything, I try to spend time with each group and ask questions that surface what they really need:

What slows you down today?
What would you never want to do by hand again?
Where do things usually go wrong: at install, at configuration or months later?
What do you have to explain to someone else? What makes it hard to explain?
If you could change one thing about how this works, what would it be?

The answers are rarely about technology. They're about time, confidence and risk. Those are exactly the things good architecture should improve.

Know the strengths in the room#

Teams are not interchangeable resources. Every person has areas where they shine and areas where they're still growing. A good design takes that into account.

If one developer has a gift for clear APIs, give them the boundaries between components. If nobody enjoys operations, that's a signal to invest early in automation. Designing for the team you actually have is not a compromise. It's what makes a design deliverable.

Carry the heaviest parts yourself#

Designing for the team you have doesn't mean avoiding what the team hasn't mastered yet. Sometimes the right answer really is Rust, C or a hard piece of cryptography while the team is strongest in Python. Giving up on the right technology would weaken the product. Handing an unfamiliar, high-risk component to people still learning it would put both them and the deadline under pressure.

In those cases I take the heavy lifting on myself. An architect who still writes code can build the hardest part personally and deliver it as a building block the team can safely rely on:

A small, stable interface in the language the team already uses, so adopting it feels like using any other library.
Thorough tests, including the failure cases, so its behaviour is predictable under pressure.
Clear documentation of what it does, what it guarantees and what it doesn't.
Packaging and automation that make it install and upgrade like everything else in the project.

The team keeps its momentum and delivers features on top of a foundation it can trust, without being slowed down by the complexity underneath. Over time, pairing and code reviews open the internals up to anyone who wants to learn them, so the building block never becomes a mystery owned by one person.

Strengths grow when someone invests in them. Mentoring is part of an architect's job: pairing on a hard design, explaining the reasoning behind a decision and giving people ownership of something slightly beyond their current reach. Much of what I know was passed on to me that way and passing it on is how a team outgrows its architect.

Design features around people, not modules#

It's tempting to organise features the way the code is organised. Users don't care how the code is organised. They care about the job they're trying to do.

So I like to think in terms of the people who will use each feature:

For deployers: one clear place for configuration, sensible defaults, pre-flight checks that fail early with a helpful message, upgrades that can be rolled back.
For solution designers: configuration instead of custom code, templates they can copy and adapt, previews that show the effect of a change before it goes live.
For product owners: features that can be switched on gradually plus demos that show real behaviour rather than slides.
For clients: interfaces that speak their language, not ours.

Great technology is not yet a product#

Engineers, me included, can fall in love with an elegant algorithm or a clever engine. But technology on its own solves nothing for anyone. It becomes a product only when people can install it, trust it, understand it and rely on it day after day. It also needs a team that can support it for years. Teams with brilliant technology still fail when they forget the people around it.

Keeping deployers, designers, product owners and clients in the conversation is what turns good engineering into something people actually use.

Abstraction is a form of empathy#

Underneath, the system can be as sophisticated as it needs to be: concurrency, integrations, encryption, retries, caching. The people using it shouldn't have to know.

Every time a deployer has to understand our internal architecture to install the product (or a solution designer has to read code to configure it), the complexity has leaked. Good abstraction keeps it contained. It means clear names, honest error messages, safe defaults and interfaces that make the right thing easy and the wrong thing hard.

Hiding complexity is not about dumbing things down. It's about respecting other people's time and expertise.

Habits that help#

A few practices I keep coming back to:

Shadow a real deployment before designing the installer or the configuration model.
Demo early and often to the people who'll use the feature, not just to the people who asked for it.
Write decisions down in plain language so product owners and clients can follow the reasoning, not just the team.
Take the riskiest part yourself when the team hasn't mastered it yet. Hand it over as a tested, documented building block.
Invite disagreement. The person who spots a problem in a design review has saved you weeks.
Close the loop. When feedback changes a design, tell the person who gave it. It builds trust and it brings more feedback next time.

The short version#

Architecture is the art of making many people's needs fit into one system. That starts with understanding them: their perspective, their pressures and their strengths. It also means being willing to carry the heaviest parts yourself, so the team can build on solid ground. Get that right and the diagram almost draws itself. Get it wrong and no amount of clever engineering will save it.

What doing DevOps taught me about architecture

2026-06-14T10:26:00+00:00

I'm a software architect by profession, but I rarely stop at the diagram. I develop and manage teams. When a product needs it, I do the DevOps myself: pipelines, infrastructure and releases. For a while design and operations felt like two different jobs: one drew the boxes, the other kept the boxes running. Doing both changed how I design almost everything.

These are the lessons that changed my designs the most, with the concrete practices behind each one.

1. The deployment is part of the design#

If nobody has decided how a component is built, versioned, deployed and rolled back, the design isn't finished. I ask "how does this ship? How does it un-ship?" in the first architecture discussion, not the last.

Two practices follow from that question.

Build once, promote everywhere. The artefact that passed testing must be the exact artefact that reaches production. Build a wheel or image once, give it an immutable version and promote that same artefact from test to staging to production. Environment differences belong in configuration, never in rebuilt code.

Make every change reversible. Rolling back code is easy; rolling back a database is not. So schema changes follow the expand and contract pattern, where each step is backward compatible with the version before it:

-- Release 1 (expand): add the new column; old code simply ignores it
ALTER TABLE orders ADD COLUMN lot_number TEXT;

-- Release 1 (migrate): backfill from the old column, in batches on large tables
UPDATE orders SET lot_number = batch_code WHERE lot_number IS NULL;

-- Release 2: code reads and writes lot_number only

-- Release 3 (contract): once no deployed version uses the old column
ALTER TABLE orders DROP COLUMN batch_code;

At every point, the previous release still works against the current schema, which means rollback is always one deployment away.

2. Configuration is an interface, so validate it#

Most production incidents I've been pulled into were not code bugs. They were configuration: a missing variable, a typo in a URL, a timeout in milliseconds where seconds were expected. Treat configuration as a typed interface and validate all of it at start-up, reporting every problem at once:

import os
from dataclasses import dataclass, fields


class ConfigError(Exception):
    pass


@dataclass(frozen=True)
class Settings:
    erp_url: str
    batch_size: int
    request_timeout: float
    dry_run: bool

    @classmethod
    def from_env(cls, prefix="APP_", env=os.environ):
        values, problems = {}, []
        for field in fields(cls):
            name = prefix + field.name.upper()
            raw = env.get(name)
            if raw is None:
                problems.append(f"{name} is not set")
                continue
            try:
                if field.type is bool:
                    if raw.lower() not in {"true", "false"}:
                        raise ValueError("expected true or false")
                    values[field.name] = raw.lower() == "true"
                else:
                    values[field.name] = field.type(raw)
            except ValueError as exc:
                problems.append(f"{name}={raw!r} is invalid: {exc}")
        if problems:
            raise ConfigError("invalid configuration:\n  " + "\n  ".join(problems))
        return cls(**values)

A service that refuses to start with a clear list of what's wrong is far cheaper than one that starts and fails an hour later on the first real request.

3. Operability is a feature, so give it numbers#

"The system should be reliable" is not a requirement. A service level objective is. Agree on what "working" means (for example, 99.9% of order messages processed within five minutes) and the target turns into an error budget: the amount of failure you can afford before reliability work takes priority over features.

def error_budget_minutes(slo, days=30):
    return (1 - slo) * days * 24 * 60


def burn_rate(bad_minutes, elapsed_days, slo, days=30):
    """1.0 means the budget will run out exactly at the end of the period."""
    budget = error_budget_minutes(slo, days)
    return (bad_minutes / budget) * (days / elapsed_days)


error_budget_minutes(0.999)  # 43.2 minutes per 30 days
burn_rate(bad_minutes=20, elapsed_days=5, slo=0.999)  # ≈ 2.8: alert, the budget is burning too fast

Alert on the burn rate rather than on every individual error. The on-call rotation stops drowning in noise. Pair it with health checks that mean something: a liveness check says the process is running; a readiness check says it can actually reach its database, queue and downstream systems. Orchestrators need both. They should never be the same endpoint.

4. Boring technology wins, by explicit criteria#

Every new tool has a cost that isn't on the pricing page: learning, upgrading, securing and debugging it at night. I'd rather run a few well-understood things superbly than many interesting things badly. To keep that from becoming "we never adopt anything new", I ask the same questions of every proposal:

Which problem does it solve that our current stack cannot? How do we measure that?
Who on the team can operate it at 3 a.m.? Who is the second person?
How is it upgraded, backed up, monitored and secured?
What is the exit plan if it's abandoned or relicensed?

Technology that answers all four earns its place. Novelty alone doesn't.

5. The pipeline is the product's immune system#

Security and quality checks that depend on someone remembering a checklist will eventually be skipped. In the pipeline they run on every change. Nothing reaches production without passing them. A condensed version of the shape I use in GitLab CI:

stages: [test, security, build, deploy]

variables:
  UV_FROZEN: "1"  # never resolve new versions in CI: the lockfile is the truth

test:
  stage: test
  image: python:3.12
  script:
    - pip install uv
    - uv sync
    - uv run pytest --junitxml=report.xml
  artifacts:
    reports:
      junit: report.xml

audit:
  stage: security
  image: python:3.12
  script:
    - pip install uv
    - uv export --no-hashes --no-emit-project --format requirements-txt -o requirements.txt
    - uvx pip-audit -r requirements.txt
    - uvx bandit -r src -ll
    - uvx --from cyclonedx-bom cyclonedx-py requirements requirements.txt -o sbom.json
  artifacts:
    paths: [sbom.json]

build:
  stage: build
  image: python:3.12
  script:
    - pip install uv
    - uv build
  artifacts:
    paths: [dist/]

deploy-production:
  stage: deploy
  id_tokens:
    CLOUD_ID_TOKEN:
      aud: https://cloud.example.com
  script:
    - ./deploy.sh dist/  # exchanges CLOUD_ID_TOKEN for short-lived cloud credentials
  environment: production
  rules:
    - if: $CI_COMMIT_TAG
      when: manual

A few details carry most of the value:

The lockfile is enforced, so every build is reproducible and a surprise dependency upgrade can't sneak in.
Dependencies are audited and code is scanned on every change. A software bill of materials (SBOM) records exactly what shipped, which turns "are we affected by this new vulnerability?" into a search instead of a guess.
The artefact is built once. The deploy job uses that same dist/.
No long-lived cloud secrets exist. id_tokens gives the job a signed OIDC token that the cloud provider trades for short-lived credentials, scoped to this project and this environment.
Production needs a tag and a human, so every release is deliberate and traceable.
People get the least access they need. Routine changes go through the pipeline. Direct access to production is temporary, approved and logged. Power over production is a responsibility and the process should make that visible.

6. Own your supply chain#

Every dependency is code you run but didn't write. Beyond the pipeline checks, a few habits make that manageable: pin everything through lockfiles, host approved packages on an internal index so builds don't depend on the public internet being available and friendly, build native wheels once per platform in CI rather than on customer machines and sign release artefacts so a deployment can prove where it came from.

7. Design for the team you have, then pave the road#

The best architecture is one your team can actually own. A design that quietly depends on a specialist nobody has hired is a liability, however clever it is. Match the complexity to the people and grow both together. When a system genuinely needs something specialised, someone has to own it openly. Often I build that part myself and deliver it as a well-tested building block with a simple interface, so the team can rely on it without carrying its complexity.

The most effective way I've found to do that is a golden path: a project template that already contains the pipeline above, logging, configuration validation, health checks and documentation skeletons. Starting a new service the right way becomes the easiest option. Good practice spreads by default instead of by memo.

8. Write decisions down#

Most painful conversations in a codebase's life start with "why on earth did we do it this way?" A short written record answers that question for years. On my teams, every significant change gets a one-page Architecture Decision Record:

# ADR-012: Use a message queue between the order connector and the ERP

## Status
Accepted on 2026-03-18

## Context
The ERP accepts about 20 requests/second and goes down for maintenance weekly.
Order bursts reach several hundred per minute.

## Decision
Publish orders to a durable queue; a single consumer writes to the ERP with
retries and idempotency keys.

## Consequences
+ Bursts and ERP downtime no longer lose or delay orders at the source.
+ The consumer can be paused, replayed and scaled independently.
- One more component to deploy and monitor.
- Orders reach the ERP eventually, not instantly.

It takes ten minutes to write and saves hours of archaeology. The Consequences section is the most important part: every decision has downsides and naming them honestly is what makes the record trustworthy.

The short version#

Architecture and operations aren't separate disciplines. Build once and promote the same artefact. Make every change reversible. Validate configuration like an interface. Give reliability numbers and alert on budgets, not noise. Adopt technology by criteria, not fashion. Let the pipeline guard quality and the supply chain. Pave a golden path for the team. Write decisions down. Your future self (probably on call) will thank you.

Simulate before you integrate

2026-01-25T20:15:00+00:00

Some systems are easy to test: call a function, check the result. Others only come alive when the real world feeds them: machines producing events, upstream systems pushing orders, sensors streaming readings at thousands of messages a minute. You can't book a production line for a Tuesday afternoon just to run your tests.

That's why so much of my tooling work is simulation: software that behaves like the real environment closely enough that the platform consuming it can't tell the difference. Done well, a simulator becomes the most valuable test asset a team owns.

What a good simulator gives you#

End-to-end tests without the hardware. The whole pipeline runs against realistic input, on any laptop or CI runner.
Performance testing on demand. Turn the speed up to ten times a real shift and watch where things queue.
Failures on purpose. Real systems send duplicates, drop messages and deliver them late. A simulator can do all of that deliberately and repeatably.
Replay. Record a scenario once, replay it after every change and compare results.
Demos that don't lie. Stakeholders see the product working on data that looks real, because it behaves like real data.

Rule one: reproducibility#

A simulator that produces different data every run turns every failing test into a mystery. Seed the randomness and the same scenario always produces the same stream:

import random
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone


@dataclass(frozen=True)
class Event:
    seq: int
    source: str
    timestamp: datetime
    value: float


def generate(sources, start, interval, seed=42):
    rng = random.Random(seed)  # a private RNG: nothing else can disturb the sequence
    seq, now = 0, start
    while True:
        for source in sources:
            seq += 1
            yield Event(seq, source, now, round(rng.gauss(50.0, 4.0), 2))
        now += interval

It's a generator, so it can run forever without holding anything in memory and itertools.islice takes exactly as many events as a test needs.

Rule two: simulated time, not wall-clock time#

Tie the simulator to the real clock and a one-hour scenario takes an hour. Generate timestamps from a simulated clock instead, then decide separately how fast to emit them: real time for a demo, as fast as possible for a load test:

import time


def paced(events, speed=1.0):
    """Emit events at `speed` times real time. speed=None means as fast as possible."""
    first_sim = first_wall = None
    for event in events:
        if speed:
            first_sim = first_sim or event.timestamp
            first_wall = first_wall or time.monotonic()
            due = first_wall + (event.timestamp - first_sim).total_seconds() / speed
            time.sleep(max(0.0, due - time.monotonic()))
        yield event

Note time.monotonic(): the wall clock can jump (NTP corrections, daylight saving); a monotonic clock can't.

Rule three: inject the failures production will send#

The happy path is the least interesting thing to test. Wrap the clean stream in a layer that misbehaves in controlled, seeded ways:

def with_faults(events, seed=7, duplicate=0.01, drop=0.005, delay=0.02):
    rng = random.Random(seed)
    held = []
    for event in events:
        roll = rng.random()
        if roll < drop:
            continue  # lost in transit
        if roll < drop + delay:
            held.append(event)  # will arrive late, out of order
            continue
        yield event
        if rng.random() < duplicate:
            yield event  # delivered twice
        if held and rng.random() < 0.5:
            yield held.pop(0)
    yield from held

Now the consumer's deduplication, ordering and gap detection get exercised on every run, long before a real network does it at 2 a.m.

Putting it together#

from itertools import islice
import json

start = datetime(2026, 1, 1, 6, 0, tzinfo=timezone.utc)
stream = with_faults(generate(["line-1", "line-2", "line-3"], start, timedelta(seconds=1)))

with open("scenario.jsonl", "w") as out:
    for event in islice(paced(stream, speed=None), 10_000):
        out.write(json.dumps({**event.__dict__, "timestamp": event.timestamp.isoformat()}) + "\n")

Swap the file for a message queue, an HTTP endpoint or a database and the same generator drives a real integration test.

Beyond the basics: what makes simulators hard#

A generator of plausible-looking events is where most simulators stop. The ones that genuinely de-risk a platform go further in five directions.

1. Model the domain, not just the data#

Random values that happen to have the right types will pass a schema check and still be nonsense: an order completed before it was released or a batch cancelled twice. Real systems produce events that follow a lifecycle, so the simulator should model that lifecycle explicitly as a state machine.

import random

TRANSITIONS = {
    "created": ["released", "cancelled"],
    "released": ["in_production", "cancelled"],
    "in_production": ["completed", "on_hold"],
    "on_hold": ["in_production", "cancelled"],
}


def lifecycle(order_id, rng):
    state = "created"
    yield order_id, state
    while state in TRANSITIONS:
        state = rng.choice(TRANSITIONS[state])
        yield order_id, state


rng = random.Random(42)
for order_id, state in lifecycle("PO-1001", rng):
    print(order_id, state)

Every generated sequence is now valid by construction. When you do want invalid sequences (to test how the consumer rejects them), inject them deliberately as a named fault, just like duplicates and delays.

2. Respect backpressure#

A simulator that pushes events faster than the system under test can accept doesn't test the system. It floods it and hides the real bottleneck. Put a bounded queue between the generator and the sink, so a slow consumer slows the producer down and you can measure how far behind it falls:

import asyncio


async def run(events, sink, rate, max_in_flight=100):
    """Emit events at `rate` per second into a bounded queue and report the peak backlog."""
    queue = asyncio.Queue(maxsize=max_in_flight)
    peak_backlog = 0

    async def produce():
        nonlocal peak_backlog
        for event in events:
            await queue.put(event)  # blocks whenever the consumer falls too far behind
            peak_backlog = max(peak_backlog, queue.qsize())
            await asyncio.sleep(1 / rate)  # the simulated arrival rate
        await queue.put(None)

    async def consume():
        while (event := await queue.get()) is not None:
            await sink(event)

    await asyncio.gather(produce(), consume())
    return peak_backlog

A consumer that keeps up holds the backlog near zero. One that can't will climb until the backlog reaches max_in_flight, at which point the producer is throttled to the consumer's real speed. That peak is the clearest signal you'll get that the consumer cannot sustain the rate you asked for. Pacing the producer matters: without the sleep, the producer would fill the queue before the consumer ever ran and even an instant consumer would look overloaded.

3. Record, replay and compare with golden files#

Seeding makes a scenario reproducible; recording it makes it portable. Save every emitted event, then replay the same file against each new build and compare the consumer's output with a known-good "golden" result. Any difference is either a bug or an intended change that needs a new golden file.

import json
from pathlib import Path


def record(events, path):
    with Path(path).open("w") as out:
        for event in events:
            out.write(json.dumps(event, sort_keys=True) + "\n")


def replay(path):
    with Path(path).open() as source:
        for line in source:
            yield json.loads(line)


def matches_golden(actual, golden_path):
    expected = json.loads(Path(golden_path).read_text())
    return actual == expected

A library of recorded scenarios (a normal shift, a network outage, the worst peak of last year) becomes a regression suite that runs in CI and protects the platform for years.

4. Keep the event schema a shared contract#

The simulator is only useful while it speaks exactly the same language as the real systems. If the real schema gains a field and the simulator doesn't, every test is quietly testing the past. Define the schema once, version it and have both the consumer and the simulator validate against it.

from dataclasses import dataclass, fields

SCHEMA_VERSION = 2


@dataclass(frozen=True)
class OrderEvent:
    schema: int
    order_id: str
    state: str
    timestamp: str


def validate(payload):
    expected = {f.name for f in fields(OrderEvent)}
    if set(payload) != expected:
        raise ValueError(f"fields {sorted(set(payload) ^ expected)} do not match the contract")
    if payload["schema"] != SCHEMA_VERSION:
        raise ValueError(f"schema {payload['schema']} is not the supported version {SCHEMA_VERSION}")
    return OrderEvent(**payload)

A contract test that validates a sample of simulated events on every build catches drift the day it happens, not the day a customer's system sends the new field.

5. Load profiles and latency percentiles#

Averages hide exactly the behaviour you care about. Drive the platform through profiles that mirror reality (a normal shift, a sustained peak, a sudden spike after an outage) and report latency as percentiles:

import statistics

PROFILES = {
    "normal_shift": [(8 * 3600, 20)],  # (duration in seconds, events per second)
    "peak": [(3600, 20), (1800, 120), (3600, 20)],
    "recovery_spike": [(600, 0), (120, 500), (1800, 20)],
}


def latency_report(latencies_ms):
    cuts = statistics.quantiles(latencies_ms, n=100)
    return {"p50": cuts[49], "p95": cuts[94], "p99": cuts[98], "max": max(latencies_ms)}

The p99 tells you what your unluckiest users experience. The recovery spike tells you whether the platform can catch up after an outage or will fall further behind forever. Those two numbers have saved more go-lives than any average ever has.

The payoff#

A simulator built this way becomes one of the most valuable assets a team owns. It enables end-to-end tests without hardware, performance tests on demand, deliberate chaos, regression suites of recorded scenarios and demos that behave like the real thing. Every one of them grows from the same foundations: seeded, simulated-time, deliberately faulty, domain-accurate and contract-bound. They pay for themselves the first time a bug is caught on a laptop instead of on a line.

Encrypting secrets at rest in Python with AES-GCM and Argon2

2025-12-06T11:48:00+00:00

Sooner or later, every on-prem tool needs to remember a secret: a database password, an FTPS credential, an API token. Cloud apps have managed secret stores. A desktop application or a service in a locked-down intranet often doesn't.

This is the approach I use when secrets have to live in a local file: authenticated encryption with AES-256-GCM, a key derived from a passphrase with Argon2id and (the part most examples skip) a versioned format so the design can change later without stranding existing files.

Don't invent cryptography

Everything below uses well-reviewed primitives from the cryptography and argon2-cffi packages. The only thing we design is the file layout. Never implement the algorithms yourself.

pip install cryptography argon2-cffi

The two building blocks#

AES-GCM is authenticated encryption: besides hiding the data, it detects any modification. Flip a single bit in the file and decryption fails loudly instead of returning garbage. It needs a 256-bit key and a 96-bit nonce that must never repeat for the same key.

Argon2id turns a human passphrase into that 256-bit key. It's deliberately slow and memory-hungry, so an attacker who steals the file can't try billions of guesses per second on a GPU. It needs a random salt per file so identical passphrases produce different keys.

A versioned file format#

Everything needed to decrypt, except the passphrase, goes into a small header:

Field	Size	Purpose
magic	4 bytes	Identifies the file type (`CSEC`)
version	1 byte	Lets the format evolve
time cost	4 bytes	Argon2 iterations
memory cost	4 bytes	Argon2 memory, in KiB
parallelism	1 byte	Argon2 lanes
salt	16 bytes	Random, per file
nonce	12 bytes	Random, per encryption

Then the ciphertext, with GCM's 16-byte authentication tag on the end.

Storing the Argon2 parameters in the header means you can make them stronger next year while old files still decrypt with the parameters they were written with.

The code#

import os
import struct

from argon2.low_level import Type, hash_secret_raw
from cryptography.hazmat.primitives.ciphers.aead import AESGCM

MAGIC = b"CSEC"
VERSION = 1
HEADER = struct.Struct(">4sBIIB16s12s")  # magic, version, t, m, p, salt, nonce

# RFC 9106's second recommended profile: 3 passes, 64 MiB, 4 lanes
TIME_COST, MEMORY_COST, PARALLELISM = 3, 64 * 1024, 4

# What we're willing to run when *reading* a file (see "Why the details matter")
MAX_TIME_COST, MAX_MEMORY_COST, MAX_PARALLELISM = 10, 1024 * 1024, 16


def _derive_key(passphrase: str, salt: bytes, t: int, m: int, p: int) -> bytes:
    return hash_secret_raw(
        passphrase.encode(), salt, time_cost=t, memory_cost=m, parallelism=p, hash_len=32, type=Type.ID
    )


def encrypt(plaintext: bytes, passphrase: str) -> bytes:
    salt, nonce = os.urandom(16), os.urandom(12)
    header = HEADER.pack(MAGIC, VERSION, TIME_COST, MEMORY_COST, PARALLELISM, salt, nonce)
    key = _derive_key(passphrase, salt, TIME_COST, MEMORY_COST, PARALLELISM)
    # The header is passed as "associated data": not encrypted, but authenticated
    return header + AESGCM(key).encrypt(nonce, plaintext, header)


def decrypt(blob: bytes, passphrase: str) -> bytes:
    header, ciphertext = blob[: HEADER.size], blob[HEADER.size :]
    magic, version, t, m, p, salt, nonce = HEADER.unpack(header)
    if magic != MAGIC:
        raise ValueError("not a codestinger secrets file")
    if version != 1:
        raise ValueError(f"unsupported format version {version}")
    if not (1 <= t <= MAX_TIME_COST and 8 * p <= m <= MAX_MEMORY_COST and 1 <= p <= MAX_PARALLELISM):
        raise ValueError("key-derivation parameters out of range")
    key = _derive_key(passphrase, salt, t, m, p)
    return AESGCM(key).decrypt(nonce, ciphertext, header)  # raises InvalidTag on tampering or wrong passphrase

Using it:

import json

blob = encrypt(json.dumps({"ftps_password": "hunter2"}).encode(), passphrase)
with open("secrets.bin", "wb") as f:
    f.write(blob)

secrets = json.loads(decrypt(open("secrets.bin", "rb").read(), passphrase))

Why the details matter#

The header is authenticated. Passing it as associated data means an attacker can't quietly lower the Argon2 cost or swap the salt. Any change to the header makes decryption fail.

Parameters are checked before they're used. The header is only authenticated after the key has been derived, so a malicious file could ask for a billion Argon2 passes or terabytes of memory and freeze the application before the tag check ever runs. Bounding the parameters first turns that denial-of-service into an instant error.

A fresh salt and nonce every time. Each encryption derives a new key from a new salt and uses a new random nonce. Nonce reuse is the one mistake that truly breaks GCM and this design makes it practically impossible.

Wrong passphrase and tampering look the same. Both raise InvalidTag. That's intentional: don't give an attacker a signal about which one happened.

Versioning is your escape hatch. When the format needs to change (a new KDF, a key file instead of a passphrase or per-entry encryption), write version 2, keep reading version 1 and migrate files the next time they're saved.

What this doesn't solve#

Encryption at rest protects a stolen file. It doesn't protect a compromised machine: if malware can read your process memory or log your keystrokes, it gets the passphrase too. And the passphrase itself has to come from somewhere: a prompt, the operating system's keyring or a hardware-backed store. Choosing that source is the real security decision; the file format above just makes sure that decision isn't undone by a weak container.

Finally, remember where most secrets actually leak. It is rarely through broken cryptography. It is through people: a passphrase pasted into a chat, written on a note beside the screen or read out to a convincing caller. Strong encryption has to be paired with simple procedures, sensible access rules and people who know why they matter.

Building Python connectors that last

2025-10-19T21:37:00+00:00

A lot of my work is integration: code that moves data between an ERP, a manufacturing system, a warehouse system and whatever API sits in the middle, over REST, SOAP, SFTP or FTPS. None of these systems were designed with each other in mind.

The first version of a connector usually works on the developer's machine. The version that matters is the one that is still running two or more years later, at a customer site, with very little maintenance. Getting there takes three kinds of work: understanding the environment before writing code, building habits into the runtime and investing in everything that keeps the code stable over time.

Part 1: Before writing any code#

Most connector problems I've seen were decided before the first line of code: the wrong assumption about volumes, an operating system nobody asked about, a firewall rule nobody requested. So I start with questions:

Infrastructure: which operating systems and Python versions, which proxies, firewalls and internal certificate authorities? Can we install packages or must everything ship as a self-contained bundle?
Volume and latency: records per hour at a normal peak and at the worst peak of the year, message sizes and the latency the business actually needs. These decide polling or streaming, batching or single calls.
The slowest system: if the ERP accepts twenty requests per second, the design respects that from day one.
Timelines and lifetime: a smaller connector that is tested and documented on time beats a complete one held together by hope. And a connector expected to run for years in a regulated environment is designed very differently from a one-off migration script.

Part 2: Habits that survive production#

1. Every network call gets a timeout#

Many HTTP clients will wait forever by default. One hung connection and your nightly job is still "running" at lunch.

import httpx

client = httpx.Client(
    base_url="https://erp.example.com/api",
    timeout=httpx.Timeout(10.0, connect=5.0),  # 5s to connect, 10s for everything else
)

Create the client once and reuse it; it keeps connections alive and makes timeouts consistent.

2. Retry the right failures, with backoff and jitter#

Some failures are temporary: dropped connections, 429 Too Many Requests, 502/503/504. Retry those. Others are permanent (400, 401, 404) and retrying them just hammers the other side.

import random
import time

import httpx

RETRYABLE_STATUS = {429, 500, 502, 503, 504}


def request_with_retry(client, method, url, *, attempts=5, base_delay=0.5, max_delay=30.0, **kwargs):
    for attempt in range(1, attempts + 1):
        try:
            response = client.request(method, url, **kwargs)
        except httpx.TransportError:
            if attempt == attempts:
                raise
            retry_after = None
        else:
            if response.status_code not in RETRYABLE_STATUS or attempt == attempts:
                response.raise_for_status()
                return response
            header = response.headers.get("Retry-After", "")
            retry_after = float(header) if header.isdigit() else None

        # Exponential backoff with full jitter, unless the server told us how long to wait
        delay = retry_after or random.uniform(0, min(max_delay, base_delay * 2**attempt))
        time.sleep(delay)

The jitter matters more than it looks. Without it, fifty workers that failed together retry together and knock the recovering server over again.

3. Make writes idempotent#

Retries create a new problem: did the first POST succeed before the connection dropped? If you retry blindly you may create the same order twice.

Give each logical operation a stable key and let the receiver deduplicate on it. Many APIs accept an Idempotency-Key header; for systems that don't, keep a table of processed message IDs on your side.

import uuid

key = str(uuid.uuid5(uuid.NAMESPACE_URL, f"order:{order.number}:{order.revision}"))
request_with_retry(client, "POST", "/orders", json=order.payload(), headers={"Idempotency-Key": key})

Deriving the key from business data (rather than a random UUID) means a re-run of the whole job produces the same keys.

4. Stream pages and keep secrets outside#

Treat paginated APIs as a generator that yields items page by page, so memory stays flat and work starts immediately. Keep credentials in the environment or a secret manager and check them all at start-up, so a missing one fails loudly before the run begins rather than halfway through it.

5. Log like you'll debug it at 3 a.m.#

Every log line should answer which run, which record, which system. A correlation ID per run and the business key per record turn a wall of text into something you can search:

import logging
import uuid

log = logging.getLogger("connector.orders")
run_id = uuid.uuid4().hex[:8]

log.info("order synced", extra={"run_id": run_id, "order": order.number, "target": "erp"})

Pair it with a formatter that prints those extra fields (or a JSON formatter) and never log payloads that contain personal data or secrets.

6. Validate at the boundary#

The other system will eventually send you something unexpected: a missing field, a date in the wrong format, a quantity as a string. Parse and validate everything at the edge, quarantine bad records with a clear reason and keep the rest flowing. One malformed message should never stop a batch of thousands.

from dataclasses import dataclass
from datetime import date
from decimal import Decimal, InvalidOperation


class RecordError(ValueError):
    """A record that can't be processed. The message says why."""


@dataclass(frozen=True)
class Order:
    number: str
    quantity: Decimal
    due: date


def parse_order(raw):
    try:
        number = str(raw["number"]).strip()
        quantity = Decimal(str(raw["quantity"]))
        due = date.fromisoformat(raw["due"])
    except KeyError as exc:
        raise RecordError(f"missing field {exc.args[0]!r}") from None
    except (InvalidOperation, ValueError, TypeError) as exc:
        raise RecordError(f"invalid value: {exc}") from None
    if not number:
        raise RecordError("empty order number")
    if not quantity.is_finite() or quantity <= 0:
        raise RecordError(f"quantity must be positive, got {quantity}")
    return Order(number, quantity, due)


good, quarantined = [], []
for raw in records:
    try:
        good.append(parse_order(raw))
    except RecordError as exc:
        quarantined.append((raw, str(exc)))

Note the is_finite() check: Decimal("NaN") parses happily and then raises on comparison. Edge cases like that are exactly what the tests in Part 3 are for.

Part 3: Built to last#

Compatibility, including the legacy#

Integrations often run where the newest tools can't. I have shipped connectors to servers where the only interpreter was Python 2.7, because the surrounding system could not be upgraded. Respect that: avoid syntax the target doesn't support, isolate the few imports that differ between versions and test on every interpreter you promise to support. Write down what you discover about the other systems too (SOAP versions, accepted TLS versions, date formats and undocumented encodings). It is the first thing someone needs when that system is finally upgraded.

Secure by default#

A connector holds credentials to two or more business systems, which makes it an attractive target. Keep TLS verification on (trust an internal CA explicitly instead of disabling checks), use service accounts with minimal permissions, pin and scan dependencies with pip-audit and bandit in CI and never write secrets or personal data to logs or quarantine files.

Compiling with Cython#

For connectors that ship to customer servers, I often compile the core modules with Cython. The same Python source becomes a native extension module. Hot loops such as mapping, parsing and transformation usually run faster while the distributed package contains compiled binaries rather than readable source.

# setup.py
import os

from Cython.Build import cythonize
from setuptools import setup
from setuptools.command.build_py import build_py

COMPILED = ["erp_connector/mapping.py"]


class build_py_without_compiled_sources(build_py):
    """Ship the compiled extension instead of the readable .py for compiled modules."""

    def find_package_modules(self, package, package_dir):
        compiled = {os.path.normpath(path) for path in COMPILED}
        modules = super().find_package_modules(package, package_dir)
        return [(pkg, mod, path) for pkg, mod, path in modules if os.path.normpath(path) not in compiled]


setup(
    name="erp-connector",
    packages=["erp_connector"],
    ext_modules=cythonize(COMPILED, compiler_directives={"language_level": "3"}),
    cmdclass={"build_py": build_py_without_compiled_sources},
)

Declare setuptools and cython under [build-system] requires in pyproject.toml so the build also works in a clean, isolated environment. pip wheel . --no-deps then produces a platform-specific wheel. Build one per target operating system and Python version in CI (Windows runners included) and check that the wheel contains the compiled module but not its .py source. Compiled code is harder to read and modify, but it is not encryption, so keep secrets out of it all the same.

Tests for the good, the bad and the ugly#

A connector earns trust through tests that cover both positive and negative cases. The happy path proves the feature works. The negative cases prove it fails safely when the world misbehaves.

from decimal import Decimal

import pytest

from erp_connector.orders import RecordError, parse_order


def test_valid_order_is_parsed():
    assert parse_order({"number": " PO-2 ", "quantity": 2.5, "due": "2026-10-01"}).quantity == Decimal("2.5")


@pytest.mark.parametrize(
    "raw, reason",
    [
        ({"quantity": "5", "due": "2026-10-01"}, "missing field 'number'"),
        ({"number": "PO-3", "quantity": "five", "due": "2026-10-01"}, "invalid value"),
        ({"number": "PO-5", "quantity": "NaN", "due": "2026-10-01"}, "must be positive"),
        ({"number": "PO-6", "quantity": "1", "due": "01/10/2026"}, "invalid value"),
    ],
)
def test_invalid_orders_are_rejected(raw, reason):
    with pytest.raises(RecordError, match=reason):
        parse_order(raw)

Beyond unit tests, integration tests exercise the connector against a stand-in for the real system. httpx ships a mock transport that makes this easy, including the failures that are hard to reproduce on demand:

import httpx

from erp_connector.http import request_with_retry


def test_retries_until_the_server_recovers(monkeypatch):
    responses = iter([httpx.Response(503), httpx.Response(503), httpx.Response(200, json={"ok": True})])
    client = httpx.Client(base_url="https://erp.test", transport=httpx.MockTransport(lambda request: next(responses)))
    monkeypatch.setattr("time.sleep", lambda seconds: None)

    assert request_with_retry(client, "GET", "/health").json() == {"ok": True}

For bigger scenarios I run the connector end to end against a simulator that produces realistic volumes and faults. All of it runs automatically in CI on every change and on every Python version the connector supports, together with a performance test at realistic volume so a slowdown is caught before a customer sees it.

Documentation that outlives the project#

Two years from now, the person supporting the connector may not be the person who wrote it. They should find a README explaining what it connects and how data flows, a configuration reference, a runbook for deploying, upgrading, rolling back and fixing common errors and a changelog that traces every version on a customer site.

The checklist#

Before a connector goes live, I ask:

Understanding

Do we know the client's infrastructure, operating systems and Python versions?
Are the volumes, peak loads and required latency agreed and written down?
Does the design respect the speed limits of the slowest system?
Is the scope realistic for the timeline?

Runtime

Does every call have a timeout, with only temporary failures retried using backoff and jitter?
Can the job be re-run safely after a crash halfway through?
Is memory flat no matter how much data comes back?
Are all secrets external and checked at start-up?
Can I trace one record from source to target and does a bad record get quarantined instead of killing the run?

Longevity

Is it tested on every interpreter and system version we promise to support?
Is TLS verified, are permissions minimal and are dependencies scanned?
Are compiled builds produced for every target platform?
Do negative cases get as much testing as positive ones, with integration and performance tests on every change?
Could someone else deploy, support and upgrade it from the documentation alone?

When every answer is yes, the connector is ready for production. More importantly, it's ready to keep running for years with very little attention.

When Python needs Rust: from a Python 2 FTPS library to PyO3

2025-04-06T18:02:00+00:00

Python is where I move fast. Rust and C are where I go when speed, memory safety or direct control matters. The best part is that you don't have to choose: a critical piece can move into Rust while the rest of the application stays comfortably in Python.

I learned how powerful that combination is on a project where Python alone simply couldn't deliver.

The problem: FTPS on Python 2#

An integration had to exchange files with partner systems over FTPS (FTP secured with TLS). The catch was the environment: the servers that would run it were tied to Python 2 and upgrading the surrounding platform wasn't an option.

Python 2's standard library couldn't give us reliable FTPS against the servers we had to talk to. The core problem is TLS session resumption: servers such as FileZilla Server and vsftpd require the data connection to resume the TLS session of the control connection (RFC 4217 allows servers to insist on this). Python's ftplib has never supported it. The gap has been tracked in CPython for years as issue #63699. With Python 2 at end of life, no fix was ever going to reach it. Add an ageing TLS stack underneath and the standard library was a dead end. Waiting wasn't an option. Weakening security to make it work was out of the question.

The solution: a Rust core with a Python wrapper#

Instead of patching around the problem, I built an internal FTPS library in Rust:

rustls for TLS: a modern, memory-safe TLS implementation that replaces OpenSSL completely. The library no longer depended on whatever OpenSSL version an old server happened to have.
suppaftp for the FTP protocol itself, with full support for both passive and active mode.
A thin Python wrapper so the rest of the integration could use the library like any other Python module.

I built this part myself. The team's strength was Python and the integration still had to ship on time, so rather than ask everyone to learn Rust, TLS internals and the Python C API under deadline pressure, I took on the heavy lifting and delivered the library as a building block they could rely on. To the rest of the team it was just another Python import with a handful of well-documented functions, backed by tests and native packages. They carried on building the integration while the hardest part was handled underneath.

Building on open source meant getting involved in it too. Testing against FileZilla Server surfaced two problems in suppaftp itself:

Session resumption. Connections failed with "TLS session of data connection not resumed". I traced it to the native-tls backend, which doesn't support session resumption. Then I showed that the rustls backend works because rustls resumes sessions by default (suppaftp #93).
Empty uploads over TLS. Uploads created files on the server but with no data. After ruling out the server and network by comparing other clients, I took it to the rustls maintainers (rustls #2297). The cause was that suppaftp closed TLS data connections without sending TLS close_notify, so the server couldn't tell the transfer had finished. I worked through the fix with the suppaftp maintainer and verified it with files up to 5 GB before it was released (suppaftp #95).

The interesting constraint was the bridge. The obvious tool for Rust and Python today is PyO3, but PyO3 no longer supports Python 2. So I built the bindings on Python's C API directly and wrapped them in Python, then shipped the whole thing as a native package for the platforms we needed.

The result was a fast, secure FTPS library that behaved like a normal Python module, running on an interpreter that the standard library had long stopped improving.

Choosing rustls also did away with the dependency on OpenSSL, which turned out to be one of the biggest wins:

No system OpenSSL to install, upgrade or patch. Legacy servers often carry old OpenSSL builds that lack modern TLS versions and ciphers. With rustls, TLS support ships inside the library itself.
The same TLS behaviour everywhere. Windows and Linux machines used identical TLS code, so a connection that worked in testing worked the same way on every customer server.
Self-contained packages. Without a native OpenSSL dependency to match, the library shipped as a single package per platform with nothing else to install.
A smaller attack surface. rustls implements only modern, safe protocol versions (TLS 1.2 and 1.3) and leaves out the legacy features behind many historical OpenSSL vulnerabilities.

What that project taught me#

The language boundary is a design decision. Keeping the interface between Python and Rust small (connect, upload, download, list, close) kept the bindings simple and the testing focused.
Rust removes whole classes of bugs. Network protocols and TLS are exactly where memory-safety bugs hurt most. Writing that layer in Rust, on a TLS stack that doesn't rely on OpenSSL, meant not worrying about them.
Don't lower the bar to fit old platforms. Legacy constraints are real, but they're a reason to bring modern tooling in, not a reason to accept weaker security.
Carry the complexity so the team doesn't have to. Building the hard part myself and handing it over as a simple, tested Python module let the team keep its pace and trust what it was building on.
Contribute back. A well-diagnosed bug report with a reproduction and a tested fix helps everyone who uses the library next, including your future self.

The modern route: PyO3 and maturin#

If you're on Python 3, you don't need to go through the C API by hand. PyO3 provides safe, ergonomic bindings between the two languages and maturin builds the result into a normal Python wheel. Here's the whole loop, end to end, with a small example from the world of product serialization.

The example: a GS1 check digit#

Every barcode on a product (GTIN-8, GTIN-13, SSCC and friends) ends with a check digit. The algorithm is simple: from the rightmost digit, multiply alternately by 3 and 1, add everything up and the check digit is whatever brings the sum to the next multiple of ten.

In Python:

def check_digit(body: str) -> int:
    total = sum(int(d) * (3 if i % 2 == 0 else 1) for i, d in enumerate(reversed(body)))
    return (10 - total % 10) % 10


assert check_digit("400638133393") == 1  # GTIN-13 4006381333931

Fine for one barcode. In a serialization system validating millions of codes, it becomes a hot loop worth optimising.

Set up the project#

pipx install maturin
maturin new --bindings pyo3 gs1fast
cd gs1fast

That creates a Cargo.toml, a pyproject.toml and src/lib.rs, already wired together.

Write the Rust#

use pyo3::exceptions::PyValueError;
use pyo3::prelude::*;

fn digit_sum(body: &str) -> Result<u32, char> {
    let mut sum = 0;
    for (i, ch) in body.chars().rev().enumerate() {
        let d = ch.to_digit(10).ok_or(ch)?;
        sum += if i % 2 == 0 { d * 3 } else { d };
    }
    Ok(sum)
}

/// Check digit for one GS1 key body (without its check digit).
#[pyfunction]
fn check_digit(body: &str) -> PyResult<u32> {
    let sum = digit_sum(body).map_err(|c| PyValueError::new_err(format!("not a digit: {c:?}")))?;
    Ok((10 - sum % 10) % 10)
}

/// Validate many complete codes in one call; returns one bool per code.
#[pyfunction]
fn validate_many(codes: Vec<String>) -> Vec<bool> {
    codes
        .iter()
        .map(|code| {
            let mut chars = code.chars();
            match chars.next_back().and_then(|c| c.to_digit(10)) {
                Some(check) => matches!(digit_sum(chars.as_str()), Ok(sum) if (10 - sum % 10) % 10 == check),
                None => false,
            }
        })
        .collect()
}

#[pymodule]
fn gs1fast(m: &Bound<'_, PyModule>) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(check_digit, m)?)?;
    m.add_function(wrap_pyfunction!(validate_many, m)?)?;
    Ok(())
}

Two details worth noticing. Invalid input becomes a proper Python ValueError, not a crash. And a &str or Vec<String> argument is converted from Python automatically.

Build and use it#

python -m venv .venv && source .venv/bin/activate
maturin develop --release

import gs1fast

gs1fast.check_digit("400638133393")  # 1
gs1fast.validate_many(["4006381333931", "4006381333932"])  # [True, False]

For distribution, maturin build --release produces a wheel you can pip install anywhere with the same platform, with no Rust toolchain needed on the target machine.

The rule that decides whether it's faster#

Crossing the boundary between Python and Rust costs something on every call: converting arguments, creating result objects. For a function this small, that overhead can be similar to the work itself.

So cross the boundary once per batch, not once per item. validate_many receives the whole list, does all the work in Rust and returns once. Calling check_digit a million times from a Python loop may barely beat pure Python; passing a million codes to validate_many is where the gain is. Always measure with timeit on realistic data before and after, because the numbers depend heavily on your data and machine.

When it's worth it#

Reach for Rust when a profiler points at a CPU-bound hot spot. Reach for it when the capability you need doesn't exist in your Python version, as with FTPS on Python 2. And reach for it when memory safety around binary formats and network protocols matters. For I/O-bound code (waiting on databases, APIs or disks) Python is almost never the bottleneck and staying in one language is the better trade.

Whether you go through PyO3 or the C API, the pattern is the same: keep Python for what it does best, move the critical piece to Rust and make the boundary between them small and well tested.

Context managers beyond with open(): owning resource lifecycles in production Python

2020-11-30T19:12:00+00:00

Every Python developer knows with open(...) as f. Fewer treat context managers as what they really are: the language's mechanism for owning a lifecycle. Anything with a beginning and an end (a transaction, a request, a connection pool, a batch of concurrent tasks, a temporary override) has a boundary. A context manager makes that boundary explicit, testable and impossible to forget.

In long-running services and integrations, those boundaries are where reliability is won or lost. This post skips the basics and focuses on the patterns I rely on in production code.

1. Error-translation boundaries#

Low-level exceptions should not leak across architectural layers. A TimeoutError from a socket means nothing to the code orchestrating a business process; SystemUnavailable("erp") does. A context manager is the cleanest place to translate one into the other, because it wraps exactly the region where the low-level calls happen.

from contextlib import contextmanager


class IntegrationError(Exception):
    """Base class for failures talking to an external system."""


class SystemUnavailable(IntegrationError):
    pass


@contextmanager
def boundary(system):
    try:
        yield
    except (TimeoutError, ConnectionError) as exc:
        raise SystemUnavailable(f"{system} is unavailable: {exc}") from exc
    except OSError as exc:
        raise IntegrationError(f"{system} failed: {exc}") from exc


with boundary("erp"):
    push_orders(batch)

Callers now handle one small, meaningful exception hierarchy, while from exc keeps the original traceback for debugging. Order matters here: TimeoutError and ConnectionError are subclasses of OSError, so the specific handler must come first.

2. Transactions and the unit of work#

A context manager is the natural shape of a transaction: commit if the block succeeds, roll back if it fails. But be careful with what libraries give you for free.

The sqlite3 trap

with sqlite3.connect(path) as conn: commits or rolls back the transaction, but it does not close the connection. In a long-running service that loop quietly leaks connections.

Owning the whole lifecycle explicitly removes the ambiguity:

import sqlite3
from contextlib import closing, contextmanager


@contextmanager
def unit_of_work(path):
    with closing(sqlite3.connect(path)) as conn:
        try:
            yield conn
        except BaseException:
            conn.rollback()
            raise
        else:
            conn.commit()

closing() guarantees the connection is closed; the try/except/else guarantees exactly one of commit or rollback. Catching BaseException (and always re-raising) means a KeyboardInterrupt or task cancellation also rolls back instead of leaving a half-written transaction.

3. Request-scoped context with contextvars#

Correlating log lines across a whole run (or a whole request) is invaluable, but threading a run_id parameter through every function is noise. contextvars provides context-local state that works correctly across threads and asyncio tasks. A context manager scopes it precisely:

import logging
from contextlib import contextmanager
from contextvars import ContextVar

run_id = ContextVar("run_id", default="-")


@contextmanager
def run_context(value):
    token = run_id.set(value)
    try:
        yield
    finally:
        run_id.reset(token)


class RunIdFilter(logging.Filter):
    def filter(self, record):
        record.run_id = run_id.get()
        return True


handler = logging.StreamHandler()
handler.addFilter(RunIdFilter())
handler.setFormatter(logging.Formatter("%(asctime)s [%(run_id)s] %(name)s %(message)s"))

Every log line emitted inside with run_context("a1b2c3"): now carries the run ID automatically, including lines from libraries you don't control. Using reset(token) rather than setting the old value back restores the previous state correctly, even when contexts are nested.

4. Transferring ownership with ExitStack#

Constructors that acquire several resources have a classic bug: if the third acquisition fails, the first two leak. ExitStack solves it. Its pop_all() method transfers ownership to the object once construction succeeds:

from contextlib import ExitStack


class MultiFileReader:
    def __init__(self, paths):
        with ExitStack() as stack:
            self.files = [stack.enter_context(open(path)) for path in paths]
            # Construction succeeded: take ownership, so nothing closes on exiting this block
            self._close = stack.pop_all().close

    def close(self):
        self._close()

    def __enter__(self):
        return self

    def __exit__(self, *exc_info):
        self.close()

If any open() fails, the stack closes every file opened so far and the exception propagates. If they all succeed, responsibility moves to the object. ExitStack.callback() extends the same guarantee to clean-up that isn't a context manager at all, such as stack.callback(shutil.rmtree, workdir).

5. Structured concurrency and deadlines#

In async code, the lifecycle to own is often a group of tasks. Python 3.11 added two context managers that change how concurrent integrations should be written:

import asyncio


async def sync_batches(batches, send, deadline=30):
    async with asyncio.timeout(deadline):
        async with asyncio.TaskGroup() as group:
            for batch in batches:
                group.create_task(send(batch))


try:
    asyncio.run(sync_batches(batches, send))
except* TimeoutError:
    log.error("sync exceeded its deadline")
except* IntegrationError as group:
    for error in group.exceptions:
        log.error("batch failed: %s", error)

TaskGroup guarantees that when the block exits, every task has finished: if one fails, its siblings are cancelled and all failures are raised together as an ExceptionGroup. asyncio.timeout() puts a single deadline over the whole operation. except* then handles each kind of failure separately. No orphaned tasks, no forgotten deadlines.

6. One object, two interfaces: ContextDecorator#

Some lifecycles apply to a block in one place and a whole function in another. Inheriting from ContextDecorator gives both from a single implementation:

import time
from contextlib import ContextDecorator


class timed(ContextDecorator):
    def __init__(self, label, log):
        self.label, self.log = label, log

    def __enter__(self):
        self.start = time.perf_counter()
        return self

    def __exit__(self, *exc_info):
        self.log.info("%s took %.3fs", self.label, time.perf_counter() - self.start)
        return False


@timed("nightly sync", log)
def nightly_sync(): ...


with timed("mapping", log):
    mapped = [transform(row) for row in rows]

7. Test the unhappy path#

The whole point of a context manager is what happens when things go wrong, so that is what the tests must prove:

import pytest

from app.db import unit_of_work


def test_failed_block_rolls_back(tmp_path):
    path = tmp_path / "test.db"
    with unit_of_work(path) as conn:
        conn.execute("CREATE TABLE orders (number TEXT)")

    with pytest.raises(RuntimeError):
        with unit_of_work(path) as conn:
            conn.execute("INSERT INTO orders VALUES ('PO-1')")
            raise RuntimeError("boom")

    with unit_of_work(path) as conn:
        assert conn.execute("SELECT COUNT(*) FROM orders").fetchone()[0] == 0

Design rules I follow#

Every resource has exactly one owner. The owner is visible in the code as a with block or an object with close().
Boundaries translate errors, so each layer speaks its own language.
Clean-up lives in finally, else or __exit__, never only on the happy path.
Never swallow exceptions by accident. Return True from __exit__ only when suppression is the explicit purpose.
Re-raise BaseException after rolling back, so cancellation and interrupts still propagate.
Prefer structured concurrency. Tasks should never outlive the block that started them.
Test the failure paths, because that is the only path the context manager truly exists for.

Treat context managers as a design tool rather than a convenience. Whole classes of production incidents (leaked connections, half-committed data, orphaned tasks, untraceable logs) simply stop happening.

The Hacker Mindset

2018-08-27T00:00:00+00:00

A hacker, in the original sense of the word, is someone who enjoys the intellectual challenge of building things and making them work. A hacker is curious about the corners other people ignore, believes information should be shared and likes helping others solve problems creatively. It has nothing to do with the kind of person who shows up in the news for breaking into systems.

Hackers are not crackers

People who break into systems for profit, fame or politics are crackers. Hackers build things; crackers break them. The rest of this post is about the first group.

Being a hacker is less about any one skill and more about a mindset. You start small, work hard on your craft, develop your own sense of design and slowly get to the point where you can build original things in your own style.

The best description of that mindset I've ever read is Eric S. Raymond's essay How To Become A Hacker. He describes five attitudes that hackers share. Read the original; it's worth it. Below is each attitude in a sentence and what it has come to mean for me after years of building software.

1. The world is full of fascinating problems waiting to be solved#

Raymond compares hackers to athletes: the motivation comes from a basic thrill in solving problems, sharpening your skills and exercising your intelligence.

For me, that thrill is the whole reason I still love this work. The trickiest integration, the report that's too big to render, the system that only fails on Tuesdays: those are the problems that make a day interesting. If you don't feel that pull naturally, you can grow it: pick a problem slightly beyond you and stay with it until it cracks.

“I find your lack of faith disturbing.” Darth Vader

The other half is faith in your own ability to learn. You rarely know everything you need at the start. Solve one small piece, learn from it and you'll know enough to solve the next piece. Repeat until done. That loop is how every large thing I've built actually got built.

2. No problem should ever have to be solved twice#

Creative brains are a limited resource. Every hour someone spends rediscovering a solution that already exists is an hour not spent on a new problem. That's why hackers treat sharing almost as a duty: solve it, then write it down and give it away.

This is the reason this blog exists. When I finally get something awkward working, a post here means the next person (often future me) doesn't have to start from zero.

It doesn't mean existing solutions are sacred. Studying the first attempt at a problem usually teaches you enough to do it better, so rebuilding it is fine. What isn't fine are artificial barriers that force people to reinvent wheels. And earning a living from your work is perfectly compatible with all of this; just don't lose your loyalty to the craft and to the people who taught you.

3. Boredom and drudgery are evil#

When a creative person is stuck doing mindless, repetitive work, the waste hurts everyone. So hackers automate the boring parts, for themselves and for everyone else too.

This one shaped my career. Whether I'm architecting a product, writing the code or running its DevOps, so much of the job is a war on drudgery: pipelines instead of manual releases, infrastructure as code instead of click-ops, generated test data instead of hand-built fixtures. If a task is done more than twice, it deserves a script.

The only exception is voluntary repetition: practising scales on a guitar or doing something repetitive to learn a skill or clear your head. That's a choice, not a burden.

4. Freedom is good#

Hackers are naturally suspicious of authority that exists only to control. Censorship, secrecy and forced "cooperation" all get in the way of solving problems.

That isn't the same as rejecting every rule. Teams need structure and systems need security. It means preferring openness where it's possible, questioning restrictions that serve no purpose and being willing to act on what you believe. Aaron Swartz remains the clearest example I know of someone who lived this.

“Do. Or do not. There is no try.” Yoda

5. Attitude is no substitute for competence#

You can adopt every attitude above and still not be a hacker. Becoming one takes intelligence, practice, dedication and hard work. I'd add one more: discipline.

So distrust posturing and respect competence in any form, especially in skills that demand focus and craft. The good news: once you truly respect competence, building it in yourself stops feeling like work and starts to feel like play.

The hacker mindset isn't something you get once. It's something you practise every time you stay curious a bit longer, share what you learned, automate the boring thing or choose the harder path of actually getting good.

The five attitudes are from Eric S. Raymond's How To Become A Hacker. The commentary is mine.