Java Microservices: Build an Idempotent Outbox Event Pipeline

Java Microservices Need an Explicit Delivery Contract

A database commit and a Kafka publish are two independent durability boundaries; treating them as one operation creates the classic failure window: the order is committed, the process dies before publish, and no downstream service hears about it. In java microservices, define the relay contract as at-least-once publication and idempotent consumption. Write these invariants into an ADR: every business event has a stable eventId, consumers may receive it more than once and out of order, and a producer never deletes an event until broker acknowledgement has been recorded.

For a java training exercise or a production design review, draw the four failure points around commit → lease → send → mark published and inject a process kill at each one. A kill after send but before mark published must produce a duplicate on the next relay attempt, not a lost event. This is a useful distinction for developers coming from a java course: duplicates are not evidence that the outbox failed; an event that never becomes visible is.

-- PostgreSQL migration: IDs are generated by the application, not by an IDENTITY column
create table outbox_event (
  id uuid primary key,
  aggregate_type varchar(80) not null,
  aggregate_id uuid not null,
  event_type varchar(120) not null,
  payload jsonb not null,
  occurred_at timestamptz not null,
  status varchar(16) not null default 'NEW',
  lease_owner varchar(80),
  lease_until timestamptz,
  published_at timestamptz
);

create index outbox_new_by_time on outbox_event (occurred_at)
  where status = 'NEW';
create index outbox_expired_lease on outbox_event (lease_until)
  where status = 'LEASED';

Use an application-generated UUID so the event can contain its final identifier before Hibernate flushes. A common mistake is to persist an entity with an IDENTITY key and assume an event can be assembled later without consequence; Hibernate may issue an early insert to obtain that key, which changes batching behavior and makes the write path harder to reason about. The partial indexes above match the two relay predicates rather than indexing every historical published row.

Spring Boot Training: Write the Outbox With the Aggregate

In a spring boot training project, make event creation part of the application service that changes state; do not publish directly from a JPA entity listener. Entity callbacks run during flush, can unexpectedly trigger lazy loading, and do not give a clean place to choose the event schema. The service below persists both rows in one database transaction, so a rollback leaves neither an order nor an event.

@Service
@RequiredArgsConstructor
class PlaceOrderService {
  private final OrderRepository orders;
  private final OutboxEventRepository outbox;
  private final ObjectMapper objectMapper;

  @Transactional
  public UUID place(PlaceOrder command) throws JsonProcessingException {
    Order order = Order.place(command.customerId(), command.lines());
    orders.save(order);

    UUID eventId = UUID.randomUUID();
    var payload = objectMapper.createObjectNode()
        .put("eventId", eventId.toString())
        .put("orderId", order.getId().toString())
        .put("customerId", command.customerId().toString())
        .put("schemaVersion", 1);

    outbox.save(OutboxEvent.newEvent(
        eventId, "Order", order.getId(), "order.placed", payload));
    return order.getId();
  }
}

Keep the payload an immutable event fact, not a serialized JPA entity. For example, publish line-item prices captured at ordering time rather than a later lookup of Order; a relay that rereads current tables can emit data that never existed when the transaction committed. Add a schema version from day one, and test backward compatibility with fixtures such as src/test/resources/events/order-placed-v1.json before a consumer deployment.

Spring Data JPA Relay: Lease Rows Without Duplicate Ownership

A naive findTop100ByStatusOrderByOccurredAt followed by entity updates is unsafe with multiple pods: each pod can read the same NEW rows before either commits. For this hot path, use Spring Data JPA only as the surrounding repository abstraction and execute an atomic native leasing statement. PostgreSQL's FOR UPDATE SKIP LOCKED lets competing relays skip rows already claimed instead of blocking behind the first worker.

@Repository
@RequiredArgsConstructor
class OutboxLeaseRepository {
  private final JdbcTemplate jdbc;

  @Transactional
  List<OutboxEventRow> lease(String owner, Duration ttl, int batchSize) {
    return jdbc.query("""
      with candidates as (
        select id from outbox_event
        where status = 'NEW'
           or (status = 'LEASED' and lease_until < clock_timestamp())
        order by occurred_at
        for update skip locked
        limit ?
      )
      update outbox_event e
      set status = 'LEASED', lease_owner = ?,
          lease_until = clock_timestamp() + (? * interval '1 millisecond')
      from candidates c
      where e.id = c.id
      returning e.id, e.event_type, e.payload::text
      """, rowMapper, batchSize, owner, ttl.toMillis());
  }
}

Publish outside the leasing transaction, wait for the Kafka producer future to complete, then run update outbox_event set status='PUBLISHED', published_at=clock_timestamp() where id=? and lease_owner=?. The owner predicate prevents an old worker from marking a row published after its lease expired and another worker reclaimed it. A lease expiry during a slow broker call can still create two sends; that is intentional at-least-once behavior, which is why the consumer design matters more than trying to eliminate every duplicate.

Hibernate ORM and Consumer-Side Idempotency

On the consumer, never implement deduplication as if (!repository.existsById(eventId)). Two consumer instances can both observe absence. Let the database serialize the decision with a unique primary key, and perform the domain mutation only when the insert succeeds. This avoids an extra read and is robust whether the consumer uses hibernate orm or direct JDBC.

create table processed_message (
  consumer_name varchar(100) not null,
  event_id uuid not null,
  processed_at timestamptz not null default clock_timestamp(),
  primary key (consumer_name, event_id)
);

@Transactional
public void onOrderPlaced(OrderPlaced event) {
  int inserted = jdbc.update("""
      insert into processed_message(consumer_name, event_id)
      values (?, ?)
      on conflict do nothing
      """, "inventory", event.eventId());

  if (inserted == 0) return; // duplicate delivery: acknowledge Kafka record
  inventory.reserve(event.orderId(), event.lines());
}

The deduplication insert and inventory.reserve must share the consumer database transaction. If the marker commits first and the reservation later fails, redelivery will skip work permanently. Conversely, an external HTTP call cannot be atomically included in this database transaction; pass eventId as an idempotency key to the downstream API or create another outbox for that side effect. This boundary is frequently missed in a java fullstack course when the UI-facing API is treated as if it were a transactional resource.

Measure the Java Backend Development Event Path

Measure end-to-end event age, not merely relay throughput. Add a Micrometer timer around successful broker acknowledgement and a gauge for the oldest unpublished occurred_at; expose them through Spring Boot Actuator and alert on the age gauge. Do not tag metrics with eventId, orderId, or a raw exception message: each creates unbounded Prometheus time-series cardinality. Safe tags are event_type, result, and a bounded relay name.

Timer.Sample sample = Timer.start(meterRegistry);
producer.send(record).whenComplete((result, error) -> {
  sample.stop(Timer.builder("outbox.publish")
      .tag("event_type", event.eventType())
      .tag("result", error == null ? "ack" : "error")
      .register(meterRegistry));
});

-- Run against a production-like data distribution, not an empty table
explain (analyze, buffers)
select id from outbox_event
where status = 'NEW'
order by occurred_at
limit 100 for update skip locked;

Establish a before-and-after baseline by recording p50/p95 event age, rows leased per second, database buffer reads from EXPLAIN (ANALYZE, BUFFERS), and Kafka acknowledgement latency at a fixed input rate. If p95 age rises while CPU is low, capture a Java Flight Recorder sample with jcmd <pid> JFR.start name=outbox settings=profile duration=60s filename=outbox.jfr; inspect JDBC waits, scheduler delays, and producer callback threads in JDK Mission Control. For java backend development, this separates an undersized batch or missing index from broker backpressure instead of guessing from application logs.

Frequently Asked Questions

How do java microservices avoid losing Kafka events after a database commit?

Insert an outbox row in the same database transaction as the aggregate update, then relay that row asynchronously. Lease rows atomically with a database lock, publish, and mark them published only after the producer acknowledgement. This converts the commit-to-publish crash window into a retryable row in the database.

Should spring data jpa use a polling repository for the outbox relay?

Use Spring Data JPA for ordinary persistence, but use a native query or JdbcTemplate for relay leasing. A read-then-update repository method permits concurrent workers to select the same event. Use PostgreSQL FOR UPDATE SKIP LOCKED, or the equivalent locking primitive in your database, in one statement that claims rows and assigns a lease owner.

How does hibernate orm implement idempotent Kafka consumers?

Create a processed_message table with a primary key of consumer name plus event ID. Insert with INSERT ... ON CONFLICT DO NOTHING before the domain mutation, and put both operations in the same local database transaction. Do not use existsById followed by save; that check-then-act sequence races across consumer instances.

What should a java course measure when tuning an outbox publisher?

Measure end-to-end event age from occurred_at to broker acknowledgement, p95 acknowledgement latency, relay batch size, and database buffer reads. Validate index changes with EXPLAIN ANALYZE on realistic row counts, then use JFR to determine whether time is spent in JDBC acquisition, SQL execution, or Kafka producer waits.