art.

Infinity has no decimal places

2026-10-12

I was reviewing a helper in my codebase. It counts decimal places of an amount. Here it is.

def _decimal_places(value: float) -> int:
    try:
        decimal_value = Decimal(str(value)).normalize()
    except (InvalidOperation, ValueError):
        return 0

    exponent = decimal_value.as_tuple().exponent
    return max(0, min(6, -exponent))

To understand how it works, I studied the helper in a Python REPL. 5.23 came back as 2. 5 and 5.0 both came back as 0.

str(value) hands the float to Decimal. Then .normalize() strips trailing zeros, and .as_tuple() breaks what is left into sign, digits, and exponent. The helper negates the exponent and clamps the result between 0 and 6. The normalize is why 5 and 5.0 agree.

Then I passed float('inf').

Decimal('Infinity') parses fine. The exponent came back as 'F'. A string. One line later the helper ran -exponent on it, and Python raised TypeError: bad operand type for unary -: 'str'.

So I went back and looked at the guard. The try/except wraps the construction only. A garbage string dies in there with InvalidOperation, and the helper returns 0. The guard caught the parse and missed the crash.

NaN is stranger. Its exponent is 'n', and its digits tuple is empty. It reaches the return line too.

My fix checks the type at the crash site.

    exponent = decimal_value.as_tuple().exponent
    if not isinstance(exponent, int):
        # NaN / Infinity have no decimal places
        return 0
    return max(0, min(6, -exponent))

The helper returns 0, the same answer bad input already gets.

Two ways to split one MQTT topic across consumers

2026-09-12

MQTT fans a topic out. Every client subscribed to it gets a copy of every message. That is the point of the protocol, and most of the time it is what I want too.

It is not what I wanted here. I had a Kafka consumer group in mind: several consumers on one topic, the messages split between them, each one handled once. I went looking for how to do that in MQTT.

The short answer is no on v3.1.1, which is what I was on. There is no consumer group in the protocol. So the question turned into what to do instead, and I found two answers and built both, in mqtt-client-load-balancing, one on a branch each.

Partition the topic yourself

The first way needs nothing from the broker. It uses the topic hierarchy as the partition key.

I picked three partitions. My publisher appends a partition number to the topic on every message, chosen at random. Each consumer then subscribes to a single partition. Every message still goes to exactly one subscriber, because no two of them subscribe to the same topic.

The publisher picks the partition, so nothing about the split lives in the broker. demoTopic/partition2 reaches consumer2 because it is the only client subscribed to it.

This is roughly how Kafka partitions a topic. The difference is that in Kafka the partition count is a property of the topic, held in one place. Here it is a convention, and every publisher has to know it.

That is what makes the approach expensive. The partition count is baked into my publishers, so the consumer count is too. I need one consumer per partition, and the two counts move together. Scaling out is not a consumer-side change. To go from three partitions to six, every device that publishes to the topic has to start using the new range. On a fleet of devices in the field, that is a rollout.

Let the broker rotate

MQTT v5 has the same thing as a protocol feature. From HiveMQ:

In a standard MQTT subscription, each subscribing client is privy to a copy of each message broadcasted to that topic. With shared subscriptions, clients sharing a subscription in the same group receive messages in rotation, a process sometimes referred to as client load balancing. The message load of a single topic is distributed across all subscribers.

It is a prefix on the subscription:

$share/<group>/topic

$share marks it as shared, <group> names the group, and the rest is the topic I want. Consumers that give the same group and the same topic form one group. The broker delivers each message to one member of it.

Which member is the broker's business. The spec does not name an algorithm. Mosquitto, which the demo runs on, goes round-robin. So the delivery is a rotation. A consumer that is still busy gets its turn anyway.

A group does not change what anyone outside it sees. The demo keeps a couple of subscribers on plain demoTopic to show it: they go on receiving every message, while the three in the group split their own copy between them.

The three in the group hold $share/shared-subscriber-group/demoTopic and split the stream. The plain sub holds demoTopic with no prefix and still gets all of it. The publisher is unchanged either way. It publishes to plain demoTopic, because $share is a subscribe-side prefix.

Scaling out is now what I wanted. I start another consumer, give it the same group, and the broker starts including it in the rotation. Nothing on the publishing side changes, because it never knew how many consumers there were.

The cost is a version floor. Shared subscriptions are v5. The prefix lives on the subscribe side, so the consumers and the broker have to speak it. Publishers are not in the way. That sounds like a small bill until someone asks who is holding the versions.

What it comes down to

Both approaches split the same stream. The difference is where the split is decided.

Partitioning puts the decision in every publisher. The topology is then spread across the fleet, and changing it means changing all of it. A shared subscription puts the decision in the broker, and the publishers never learn about it. That is why the consumer side can be resized on its own.

So MQTT has no consumer group on v3.1.1, and something close to one on v5. That is the answer to the question I started with.

We never took it. v5 was out by then, but our devices had already shipped on v3.1.1. The broker was not moving either. I no longer remember why. The devices on their own would not have stopped us, since a publisher never sees the prefix. The broker did.

So the shared subscription stayed a branch in this repository. Partitioning was the only one of the two I could actually have used, at the price above: every publisher in on the scheme, and a rollout to the fleet to change it.

Both branches run under Docker Compose, and the consumer logs show the split: github.com/ssupawat/mqtt-client-load-balancing

References

Fixing an SSE stream that dies at two minutes

2026-09-10

I got a chance to serve an AG-UI endpoint. I started where the protocol points. AG-UI's LangGraph integration ships a helper that registers the endpoint. It calls @app.post(path) and returns a StreamingResponse. From there I wired it into my backend, and it ran.

Then I started losing streams. About two minutes into a run, the stream would stop. After that the client sat with half an answer.

My first guess was an idle timeout. Something in front was hanging up on a stream that had gone quiet. The standard fix for that is a keepalive, because it stops the connection from looking idle. So I added one. Even so, I still lost the stream at two minutes.

So whatever sits in front counts the age of the connection. It hangs up at two minutes, whether or not bytes are moving. Next I asked the infra team to raise the limit, and they turned me down. In the end I never did learn which layer sets that number.

That settled the shape of the problem. From then on, every run longer than two minutes gets cut. And I have to survive it from my side.

Normally the browser would reconnect for me. EventSource does it for free. First it retries on its own. Then it sends back a Last-Event-ID header. That header names the last event the client saw. Then the server carries on from there. But I could not use any of it. That machinery is GET only, and AG-UI answers on POST. That puts me on fetch and a ReadableStream. Those will stream a response. Even so, recovering one is my job.

The second wall came right behind the first. My endpoint runs on several pods. Because of that, the retry goes through the load balancer. It lands wherever it lands. If the half-finished run lives in the process that started it, only that process can answer. Then I am back to pinning traffic with sticky sessions.

So I needed two things. First, the work has to keep going after the client is gone. Second, any pod has to be able to read a record of it.

I built reconnectable-sse to try it out. My production endpoint is Python. But this was a concept, and the language was free. So I used it as an excuse to get familiar with Go. In the end the shape is the part that matters.

It comes down to two lines:

if !streamExists(r.Context(), key) {
    go runEventGenerator(streamID, string(body))
}

runSSEResponse(r.Context(), streamID, lastEventID, w, flusher)

I give runSSEResponse the request context. A disconnect stops it. That is what I want, since there is nobody left to write to. Meanwhile runEventGenerator gets context.Background(), and it keeps running. Hand it the request context instead, and the disconnect kills the work itself. Then nothing is left to come back to. The run also has to finish and write its history to the database. Whether anyone is still watching is beside the point.

The shaded band is api-1's generator. It starts on the first request and runs to completion on context.Background(). It keeps going through the drop and the reconnect, even though api-2 is the one talking to the client now.

Resume then came down to an offset. Redis returns an entry id on every XADD. That id goes out as the SSE id. Then the client sends it back on the retry. From there it goes into XREAD as the starting point. That leaves no cursor table and no bookkeeping. In the end the id the client is holding is the offset.

I ran it. It does what I wanted. After a drop mid-stream, a client comes back through a different pod and picks up where it stopped.

Still, I left edges in it. The generator starts under a check-then-act. So two concurrent first requests with the same stream id can both start one.

Design, endpoints, and how to run it: github.com/ssupawat/reconnectable-sse