Exactly-Once Semantics

8 min read · Updated 2026-04-25

Exactly-once semantics is one of the most misunderstood concepts in distributed systems. Strictly speaking, in an asynchronous network with possible failures, “exactly-once delivery” is impossible. In practice, exactly-once processing is achievable — but only with deliberate design.

This lesson separates the impossible from the practical and walks through the patterns that actually work.

Why Exactly-Once Delivery Is Impossible

Consider a producer sending a message:

  1. Producer sends.
  2. Broker receives (?).
  3. Broker acknowledges (?).
  4. Network drops the ack.
  5. Producer doesn’t know if the message arrived.

If the producer doesn’t retry: possible message loss → at-most-once. If the producer does retry: possible duplicate → at-least-once.

There’s no third option. The Two Generals Problem proves you can’t have certainty about the other side’s state in an asynchronous network.

The same problem appears between broker and consumer. So end-to-end, you can have “deliver at most once” or “deliver at least once” — not both at once at the protocol level.

What “Exactly-Once” Actually Means in Practice

Modern systems claiming exactly-once are usually doing one of:

Idempotent processing
At-least-once delivery + consumer that handles duplicates safely. The most common practical answer.
Transactional brokers
Kafka transactions, AWS SQS FIFO. Producer-broker-consumer all coordinate via a transaction. Exactly-once between specific systems, with caveats.
Deduplication keys
Producer attaches a unique ID; broker or consumer dedups within a window. Stripe-style idempotency keys are the canonical example.

Pattern 1: Idempotent Consumers

The most pragmatic answer: design every consumer to safely handle duplicates.

Naturally idempotent
Same operation = same result
SET operations, upserts, "mark as read" flags. Reprocessing the same message produces the same end state. The cleanest case.
Made idempotent
Wrap with deduplication
Operations like "increment counter" or "send email" aren't naturally idempotent. Wrap with a dedup mechanism — track processed message IDs.

Implementation: deduplication store

def handle_message(msg):
    if has_processed(msg.id):
        return  # already processed, skip

    process(msg)
    mark_processed(msg.id)

The dedup store needs:

Pattern 2: Idempotency Keys at the API Layer

Stripe popularized the pattern: every state-changing API request can carry an Idempotency-Key. The server stores (key → response) and returns the saved response on retry.

POST /payments
Idempotency-Key: client-uuid-12345
{"amount": 100, "currency": "USD"}

Server logic:

def create_payment(request, idempotency_key):
    if existing := get_response(idempotency_key):
        return existing  # client retried; return stored response

    response = actually_create_payment(request)
    save_response(idempotency_key, response)
    return response

This pattern is now the standard for any state-changing SaaS API endpoint. Add it everywhere POST or PUT could trigger duplicate operations.

Pattern 3: Transactional Outbox

The classic problem: write to DB and publish a message — but how do you make those two operations atomic?

Naive: dual-write
DB write + broker publish, separately
Two operations. Either can fail. Failure between them = inconsistency. DB written, message not sent. Or message sent, DB rolled back.
Transactional outbox
Single DB transaction
Write the state change AND insert a row into an "outbox" table — in one DB transaction. A separate process (relay) reads the outbox and publishes.
BEGIN;
  UPDATE orders SET status = 'paid' WHERE id = 42;
  INSERT INTO outbox (event_type, payload) VALUES ('OrderPaid', '{...}');
COMMIT;

-- Relay process: read unsent outbox rows, publish to broker, mark sent.

The relay process can be a polling worker, a CDC stream (Debezium), or Kafka Connect. Publishing happens at-least-once (broker may receive a message twice if relay restarts mid-publish), but consumer-side idempotency catches duplicates.

Pattern 4: Kafka Transactions

Kafka has built-in transactional support for atomic write-to-multiple-topics:

producer.beginTransaction()
producer.send(topicA, msg1)
producer.send(topicB, msg2)
producer.send("__consumer_offsets", offsetCommit)  // commit consumer position
producer.commitTransaction()

The producer either commits all messages atomically or none. Combined with isolation.level=read_committed on consumers, you get exactly-once within Kafka.

Pattern 5: AWS SQS FIFO + Exactly-Once

AWS SQS FIFO queues offer exactly-once delivery — but with constraints:

For workloads that fit, it’s a clean solution: cloud-native, no operational overhead, exactly-once guaranteed.

Pattern 6: CDC + Exactly-Once Sinks

For streaming pipelines:

[ Postgres ] → [ Debezium / CDC ] → [ Kafka ] → [ Sink Connector ] → [ Snowflake / S3 / etc. ]

Modern sink connectors implement exactly-once by:

  1. Tracking processed Kafka offsets transactionally with the destination.
  2. Using natural keys / deduplication on the destination.
  3. Atomic writes (S3 versioning, transactional INSERT).

What to Use When

API endpoints
Stripe-style Idempotency-Key. Mandatory for any non-idempotent state-changing endpoint.
Cross-service async messaging
Outbox pattern. The standard pattern for "write to DB and publish event."
Streaming pipelines (Kafka)
Kafka transactions for in-Kafka work. Connector-level exactly-once for sinks.
Background jobs
Idempotent consumers + deduplication store. Track processed job IDs, skip duplicates.
AWS SaaS
SQS FIFO when throughput fits. Otherwise SQS Standard + idempotent consumers.
Webhook delivery
Sender retries on non-2xx. Receiver dedups on a (sender, event_id) tuple.

Recap