26 min read

What I learned at Devoxx Morocco 2026: Day 1 🇲🇦

Key ideas from day one of Devoxx Morocco 2026, talk by talk, with interactive examples.

conferencedevoxxapiaievent-driven

The key ideas from the talks I attended on day one of Devoxx Morocco 2026, one section per talk. Code examples are in TypeScript and framework-agnostic, and the important ideas come with small demos you can click around in.

The main room before the first talk
The main room
Close-up of a colorful painting at the venue
Spotted at the venue
Talk 1
An Opinionated Guide to Bulletproof APIs
Rustam Mehmandarov

The core idea: most of what makes an API robust has nothing to do with your endpoints. It's the layers a request passes through before and after your handler runs: gatekeepers, security, observability, contracts and versioning, and error handling.

Follow a request through the gates
Pick a request and watch where it gets stopped
  1. 1Sanitize
  2. 2Auth
  3. 3Roles
  4. 4Sign
  5. 5Validate
  6. 6Handler
Error mapper catches anything thrown above and returns application/problem+json
Request
POST /v2/talks HTTP/1.1
Authorization: Bearer eyJhbGciOi… (organizer)
X-Signature: 9f2c41…

{ "title": "Bulletproof APIs" }
Response201
HTTP/1.1 201 Created
Content-Type: application/json
X-Request-ID: req_ok_7f3a91

{
  "id": 42,
  "title": "Bulletproof APIs"
}

Every failure, at any gate, lands in the same error mapper and leaves as an RFC 9457 Problem Details response.

Gatekeepers: write it once, not per endpoint

Cross-cutting logic doesn't belong inside handlers. Input checks, auditing and custom validation go in middleware that you write once and attach where it's needed.

Reject garbage before routing

A request can be inspected before the framework figures out which route it matches. If it contains things that should never be there (HTML tags, <script>, javascript: URLs), throw it away before spending any CPU on routing, parsing or business logic. The earlier you reject, the less each bad request costs, which also helps against DoS.

const SUSPICIOUS = /<\s*\/?\s*(script|iframe|img|svg)\b|javascript:/i;

export const sanitize: Middleware = async (req, next) => {
	const body = await req.clone().text();
	const query = decodeURIComponent(new URL(req.url).search);

	if (SUSPICIOUS.test(body) || SUSPICIOUS.test(query)) {
		throw new ProblemError(400, "unsafe-input", "Request contains disallowed markup");
	}
	return next();
};

This is a cheap first filter, not a replacement for escaping output or parameterized queries.

Opt-in auditing

Auditing works best as a marker you put on the endpoints that need it. Everything else stays un-audited.

const audited = (handler: Handler): Handler => async (req) => {
	const entry = { requestId: req.headers.get("x-request-id"), method: req.method, path: new URL(req.url).pathname };
	auditLog.write({ ...entry, phase: "attempt" });

	const res = await handler(req);
	auditLog.write({ ...entry, phase: "result", status: res.status });
	return res;
};

router.post("/talks", audited(createTalk)); // audited
router.get("/talks", listTalks); // not audited

The same pattern works for custom validators, like a "no blocked words" rule defined once and attached to any field. When the rule changes, it changes in one place.

Security: three jobs, not one

Security is three separate questions:

  1. Authentication: who are you? (usually a )
  2. Authorization: you're logged in, but are you allowed to do this?
  3. Integrity: is this payload exactly what was sent, or did someone change it on the way?

The third one is easy to forget, and it's critical for things like payments, where a man-in-the-middle changing amount: 100 to amount: 10000 is the whole attack. The fix is to sign the payload with a shared secret (-SHA256) and verify it on the other side. Try breaking it:

Try to tamper with a signed payload
Real HMAC-SHA256, computed in your browser
Client signs
{
  "from": "acct_1029",
  "to": "acct_8821",
  "amount": 100,
  "currency": "MAD"
}
X-Signature
computing…
Server receives · editable
HMAC of what arrived
computing…
Even adding a space breaks the signature, which is why you verify the raw bytes you received, not JSON you parsed and re-serialized.
import { createHmac, timingSafeEqual } from "node:crypto";

export function verifySignature(rawBody: string, timestamp: string, signature: string, secret: string) {
	// Sign the timestamp too, and reject old ones, so a valid request can't be replayed later
	if (Math.abs(Date.now() - Number(timestamp)) > 5 * 60_000) return false;

	const expected = createHmac("sha256", secret).update(`${timestamp}.${rawBody}`).digest("hex");
	const a = Buffer.from(expected);
	const b = Buffer.from(signature);
	return a.length === b.length && timingSafeEqual(a, b);
}

Roles belong on the route

Rule of thumb: if you're writing user.hasRole(...) inside business logic, something is probably in the wrong place. Declare roles next to the route so the whole access policy is visible in one place:

// public
router.get("/talks", listTalks);

// organizers only
router.post("/talks", requireRole("organizer"), createTalk);
router.delete("/talks/:id", requireRole("organizer"), deleteTalk);

In larger setups, an is the only thing exposed to the internet. It does the first round of checks (identity, origin, keys), and the services behind it verify the tokens again. That's defense in depth.

The lens: observability is not logging

Logging is not observability. Logs tell you that something was slow. Observability tells you which calls, for whom, and where in the chain.

Two tools make the difference:

  • A . Generate an X-Request-ID at the edge (or accept the client's), attach it to every log line, and forward it on every downstream call, so one request can be followed across services.
  • Span attributes. With , tag each with whatever you'll want to filter by later: API version, resource ID, request ID, user.
import { AsyncLocalStorage } from "node:async_hooks";
import { trace } from "@opentelemetry/api";

const context = new AsyncLocalStorage<{ requestId: string }>();

export const requestId: Middleware = (req, next) => {
	const id = req.headers.get("x-request-id") ?? crypto.randomUUID();

	trace.getActiveSpan()?.setAttributes({
		"request.id": id,
		"api.version": resolveVersion(req),
		"user.id": currentUser(req)?.id ?? "anonymous",
	});

	return context.run({ requestId: id }, next);
};

// Every outgoing call carries the same ID
export const callService = (url: string, init: RequestInit = {}) =>
	fetch(url, { ...init, headers: { ...init.headers, "x-request-id": context.getStore()!.requestId } });

If no OpenTelemetry collector is running, none of this does anything: no errors, no meaningful overhead. So it's safe to add from day one.

The same eight requests, seen two ways:

Logs vs. traces
The same 8 requests, seen two ways
10:42:01.114 INFO gateway GET /v2/talks → 200 (733ms)
10:42:01.123 INFO talks-service fetched talks from db
10:42:01.135 INFO speakers-service GET /speakers/7 → 200
10:42:01.151 INFO gateway GET /v1/talks → 200 (92ms)
10:42:01.160 INFO talks-service fetched talks from db
10:42:01.188 INFO gateway GET /v1/talks → 200 (78ms)
10:42:01.197 INFO talks-service fetched talks from db
10:42:01.225 INFO gateway GET /v2/talks → 200 (644ms)
10:42:01.234 INFO talks-service fetched talks from db
10:42:01.246 INFO speakers-service GET /speakers/7 → 200
10:42:01.262 INFO gateway GET /v1/talks → 200 (101ms)
10:42:01.271 INFO talks-service fetched talks from db
10:42:01.299 INFO gateway GET /v2/talks → 200 (869ms)
10:42:01.308 INFO talks-service fetched talks from db

Some requests are slow. Which ones? Which speakers call belongs to which request? Was it a specific version, or a specific user? The logs can't say.

Spans with attributes like api.version and user.id let you ask questions. Here the answer is that v2 calls the speakers service once per talk.

Liveness vs. readiness

Cloud platforms (, with scale-to-zero, …) rely on two different health questions, and confusing them causes outages:

  • Liveness: "Are you healthy, or should I kill you and start a new one?"
  • Readiness: "You're running, but can you take requests yet?"

A simple liveness check: if response times go above 3 seconds, report DOWN and let the platform restart the instance.

Liveness vs. readiness
Drag through 90 seconds of one instance
p95 latency
275 ms
3s threshold
Liveness
UP
Readiness
UP
0st = 30s90s
UP DOWN no container (restart)

Both probes up. Requests flow normally.

Liveness answers "should you restart me?" and readiness answers "can I take requests yet?". Mixing them up gives you either restart loops or traffic sent to an app that is still booting.
router.get("/health/live", () => {
	const p95 = metrics.responseTimeP95();
	return p95 < 3_000 ? json({ status: "UP" }) : json({ status: "DOWN", p95 }, 503);
});

router.get("/health/ready", async () => {
	const ready = (await db.ping()) && cache.isWarm();
	return json({ status: ready ? "UP" : "DOWN" }, ready ? 200 : 503);
});

Contracts and versioning

OpenAPI should document your errors too

's one redeeming quality was the WSDL: a strict contract for exactly how to talk to a service. is the looser, human-readable successor. It doesn't force you to describe everything, but you should, including the error responses, not just the happy path.

Rules can also be enforced across a whole organization in the spec: every operation accepts an X-Request-ID header, and every operation has a documented generic 500. A small script over the spec can inject these automatically, or a linter like can enforce them in CI.

paths:
  /talks:
    post:
      parameters:
        - $ref: "#/components/parameters/XRequestId"
      responses:
        "201": { $ref: "#/components/responses/TalkCreated" }
        "400": { $ref: "#/components/responses/Problem" }
        "401": { $ref: "#/components/responses/Problem" }
        "403": { $ref: "#/components/responses/Problem" }
        "422": { $ref: "#/components/responses/Problem" }
        "500": { $ref: "#/components/responses/Problem" }

Versioning: pick a strategy (or two)

There are three common ways to version an API. In practice URL and header versioning are often combined: new clients use /v2/..., older ones send a header, and the version is resolved once, before routing.

Three ways to ask for version 2
Same resource, different contract
curl https://api.example.com/v2/talks/42
Good
Obvious. Easy to read, cache, share and try in a browser.
Watch out
The version leaks into every URL and every link you hand out.
v1 response
{
"id": 42,
"title": "Bulletproof APIs",
− "speakerId": 7,
− "speakerName": "Jane Doe"
}
v2 response
{
"id": 42,
"title": "Bulletproof APIs",
+ "speaker": {
+ "id": 7,
+ "name": "Jane Doe"
+ }
}
v2 changed the domain model (speaker became an object), and every strategy has to route the client to the right shape.
export function resolveVersion(req: Request): number {
	const path = new URL(req.url).pathname.match(/^\/v(\d+)\//)?.[1];
	const header = req.headers.get("x-api-version");
	const media = req.headers.get("accept")?.match(/vnd\.example\.v(\d+)\+json/)?.[1];
	return Number(path ?? header ?? media ?? 1);
}

This is why you version: in v2 the domain model changed (speakerId became a nested speaker object). That's a breaking change for every client, and versioning is how you ship it without breaking them.

Errors: RFC 9457 and the 200 that lies

(Problem Details) is a standard format for API errors. It replaces the older RFC 7807 and is much simpler than the name suggests:

{
	"type": "https://api.example.com/problems/validation",
	"title": "Request isn't valid",
	"status": 422,
	"detail": "'title' contains a blocked word.",
	"instance": "/v2/talks",
	"requestId": "req_7f3a91"
}

It's served as application/problem+json, and you can add your own fields (like requestId above). There are libraries for most stacks, but it's simple enough to hand-write. What matters is using the standard instead of inventing a new error shape for every API.

And never return an HTTP 200 that is actually an error. It's not just ugly. It makes your monitoring blind:

Same incident, two dashboards
One bad deploy, 12 minutes of errors
Errors returned as 200 OK
0%peak error rate
No alert. Everything looks fine.
0%25%50%
alert at 5%
10:0010:1510:3010:45
Errors returned as 4xx/5xx + Problem Details
34%peak error rate
Alert fired at 10:20
0%25%50%
alert at 5%
10:0010:1510:3010:45
Show as table
TimeAs 200 OKReal status codes
10:000%0.2%
10:010%0.5%
10:020%0.3%
10:030%0.6%
10:040%0.4%
10:050%0.2%
10:060%0.5%
10:070%0.3%
10:080%0.6%
10:090%0.4%
10:100%0.2%
10:110%0.5%
10:120%0.3%
10:130%0.6%
10:140%0.4%
10:150%0.2%
10:160%0.5%
10:170%0.3%
10:180%0.6%
10:190%0.4%
10:200%14.5%
10:210%19%
10:220%26.5%
10:230%34%
10:240%32%
10:250%33%
10:260%34%
10:270%32%
10:280%33%
10:290%27.5%
10:300%19%
10:310%13.5%
10:320%0.3%
10:330%0.6%
10:340%0.4%
10:350%0.2%
10:360%0.5%
10:370%0.3%
10:380%0.6%
10:390%0.4%
10:400%0.2%
10:410%0.5%
10:420%0.3%
10:430%0.6%
10:440%0.4%
10:450%0.2%
10:460%0.5%
10:470%0.3%
A bad deploy breaks a third of requests for 12 minutes. On the left the API returns 200 with the error in the body, so monitoring sees nothing and someone has to read the payloads. On the right the API uses real status codes, and the alert fires within a minute.

Throw, map, and catch everything

The cleanest way to get proper status codes is to not set them in handlers at all. Throw meaningful errors, map them to status codes in one place, and keep a catch-all at the end that turns anything unexpected into a generic 500:

export class ProblemError extends Error {
	constructor(public status: number, public type: string, public title: string, public detail?: string) {
		super(title);
	}
}

export const problemHandler: Middleware = async (req, next) => {
	const instance = new URL(req.url).pathname;
	const requestId = req.headers.get("x-request-id");

	try {
		return await next();
	} catch (err) {
		if (err instanceof ProblemError) {
			return problem({ type: `https://api.example.com/problems/${err.type}`, title: err.title, status: err.status, detail: err.detail, instance, requestId });
		}
		// The stack trace stays in OUR logs, never in the response
		logger.error({ err, requestId }, "unhandled error");
		return problem({ type: "about:blank", title: "Internal Server Error", status: 500, detail: "Something went wrong on our side. We've logged it.", instance, requestId });
	}
};

const problem = (body: { status: number } & Record<string, unknown>) =>
	new Response(JSON.stringify(body), {
		status: body.status,
		headers: { "content-type": "application/problem+json" },
	});

Stack traces should never reach users:

  1. They scare users.
  2. They're useless as bug reports.
  3. They're a security problem. They expose your libraries, versions and internal structure to anyone who can trigger an error.

With a request ID in every Problem response, a user can report one short string and you can find the full trace in your logs.

Edge cases worth deciding on purpose

  • Unknown fields. If a client sends a field you don't know, ignore it (be tolerant in what you accept). If a required field is missing, reject it. Libraries don't all agree on this. For example, strips unknown keys by default and .strict() turns them into an error.
  • Resumable uploads. Plain uploads break when the connection drops. The open protocol handles resuming, and it has clients for most languages.
Key takeaways
  • Use RFC 9457 for every error response, with a requestId field: one error shape across all APIs, and real error rates in monitoring.
  • Never wrap errors in a 200. Endpoints that do are invisible to alerting.
  • Add X-Request-ID end to end and tag spans with api.version and user.id. It turns "it's slow sometimes" into a query.
  • Keep liveness and readiness separate.
  • Document error responses in OpenAPI specs, and lint for it in CI.
  • Move repeated checks out of handlers into shared middleware: input filtering, role checks, auditing, validation.
  • Sign payloads (HMAC) wherever tampering would hurt, like payments and webhooks.
Talk 2
Agentic Sites: Building Hyper Personalized Websites
Carlos Sanchez

The core idea: instead of one static page for everyone, an assembles each page for the visitor in front of it, using what they browsed, read and searched for. Personalized pages get more clicks and more purchases, and with LLMs the content no longer has to be hand-written for every segment.

The catch is that this only works if it's fast, safe and measurable. Most of the talk was about how to get there.

One page, generated per visitor
Pick a visitor and watch their page get assembled
Signals
  • “coffee machine for camping”
  • Travel espresso maker2m 10s
  • Hand burr grinder48s
  • Brewing coffee on the trail
LLM steps
  1. Classify intent
    …
  2. Pick blocks
    …
  3. Recommend
    …
Time to page0.00 s / 2 s budget
shop.example/for-you
The model never writes HTML. It returns a small plan that picks from existing blocks and real catalog products, and the site renders it. Timings are illustrative.

Signals in, persona out

Personalization starts with tracking ordinary browsing signals: which products someone viewed, for how long, which articles they read, what they searched for. A small rolling profile kept on the client (a cookie or local storage) is enough.

type Signal =
	| { type: "view"; productId: string; dwellMs: number }
	| { type: "read"; articleId: string }
	| { type: "search"; query: string };

export function track(signal: Signal) {
	const profile = readProfile();
	profile.signals = [...profile.signals, signal].slice(-20); // a short rolling window is enough
	writeProfile(profile); // cookie or localStorage
	schedulePregeneration(profile);
}

The first LLM step classifies the visitor into one of a set of defined by the marketing team, like someone buying a travel espresso maker or someone who wants café-grade gear at home. A fixed list keeps the output predictable, and marketing stays in control of who the site talks to.

The model picks blocks, it doesn't write the page

The page is made of blocks: hero, product cards, tips, comparison, guide. Content authors publish those in the as usual. The LLM's job is only to return a small plan: which persona, which blocks in which order, the hero copy, and which products to feature.

import { z } from "zod";

const PagePlan = z.object({
	persona: z.enum(["outdoor-traveler", "student-small-space", "home-barista", "default"]),
	blocks: z.array(z.enum(["hero", "tips", "comparison", "product-cards", "guide"])).max(4),
	hero: z.object({ title: z.string().max(60), subtitle: z.string().max(120) }),
	products: z.array(z.string()).max(3),
});

export async function generatePage(profile: Profile) {
	const plan = PagePlan.parse(
		await llm.json({ prompt: pagePrompt(profile, catalog), maxOutputTokens: 300 })
	);
	// Only real catalog items get rendered, whatever the model said
	const products = plan.products.filter((id) => catalog.has(id));
	return plan.blocks.map((name) => blocks[name].render({ ...plan, products }));
}

This split does a lot of work:

  • The layout stays on-brand, because the site only renders blocks it already knows.
  • Products are checked against the catalog, so the model can't recommend something that doesn't exist.
  • Images are not generated. Brands want control over their imagery, so blocks use existing assets and the model only picks which ones.
  • The output is tiny, which is what makes it fast enough.

Behind it sits a over the catalog and content (), so the prompt only carries the products and articles that are relevant to this visitor.

Two seconds or it doesn't count

A generated page has to show up in under two seconds, or people leave. That budget drives most of the design:

  • Model choice is a latency decision. In the talk's benchmarks, a small model on a generated the page in about a second, while larger models took several seconds. Frontier models were far too slow for this job.
  • Cap in and out. Short prompts and a short, structured answer. This use case doesn't need much text, so don't let the model ramble.
  • Pre-generate. As signals come in, generate the "For you" page in the background. When the visitor clicks, it's already there.
Fit the page in two seconds
Pick a model, then apply the tricks
1.80 suser waits · 0.20 s to spare
Generation
User waits after clicking
0 s2 s budget4 s6 s
  • Classify intent320 ms
  • Pick blocks410 ms
  • Recommend1070 ms
Pre-generation doesn’t make the model faster. It moves the work off the click: the page is built in the background as signals come in. Timings are illustrative.
const ready = new Map<string, Promise<Page>>();

// Called on every new signal: start building the page before anyone asks for it
export function schedulePregeneration(profile: Profile) {
	const key = profileKey(profile); // hash of the persona-relevant signals
	if (!ready.has(key)) ready.set(key, generatePage(profile));
}

export const forYou = (profile: Profile) => ready.get(profileKey(profile)) ?? generatePage(profile);

Evaluate every model, for every site

New models show up every week, so model selection has to be a repeatable test, not a one-off opinion. The setup in the talk used (open source): the same prompts and test cases run against every candidate model, measuring accuracy and latency side by side.

# promptfooconfig.yaml, one suite per site
prompts:
  - file://prompts/classify-intent.txt
providers: # every candidate model gets the same prompts and tests
  - id: <fast-small-model>
  - id: <mid-size-model>
  - id: <frontier-model>
tests:
  - vars:
      signals: 'searched "coffee machine for camping", viewed travel espresso maker'
    assert:
      - type: is-json
      - type: javascript
        value: JSON.parse(output).persona === "outdoor-traveler"
      - type: latency
        threshold: 2000

Each step of the pipeline (intent classification, block selection, recommendation) gets its own tests. And an evaluation done for one brand doesn't transfer to another: different products, different shoppers, different queries. Each site needs its own suite.

Same models, same prompts, two sites
Shoppers name the product they want
  • Fast inference
    Passes, but not the best
    90%1.0 s
  • Mid-size model
    Picked for this site
    95%1.8 s
  • Frontier model
    Too slow
    100%6.2 s
Every model runs the same 20 test cases for a site. The pick is the most accurate one that clears 85% accuracy and the 2 s budget, and it changes from one site to the next. Results are illustrative.

Generate offline, serve the safe subset

Letting a model generate live on every request is the riskiest option. A safer pattern is to pre-generate a large number of variants offline, have against the brand guidelines, and only serve from the approved set. The request path then picks from content that's already been checked.

// Offline job: many candidates per persona, keep only what passes review
for (const persona of personas) {
	const candidates = await Promise.all(Array.from({ length: 200 }, () => generateHero(persona)));
	const approved = [];
	for (const hero of candidates) {
		if ((await judge(hero, brandGuidelines)).score >= 0.8) approved.push(hero);
	}
	await variants.save(persona, approved);
}

// Online: no generation on the request path
export const heroFor = (persona: Persona) => pickOne(variants.get(persona));

Measuring is the real work

Once generation is in place, most of the effort moves to measurement. Is the classification right? Which generated content gets clicks? Is it actually improving sales? Every generated page should be logged with its timings and its plan, so analytics can show which personas, blocks and copy perform. That data then feeds the next round of prompts and evals.

Key takeaways
  • Have the LLM return a plan, not a page. Constrain it to known blocks, personas and catalog ids, and validate the output.
  • Treat two seconds as a hard budget. Pick models by latency first, and cap tokens in and out.
  • Pre-generate pages in the background as signals arrive, so the click is instant.
  • Don't generate images. Reuse brand assets and let the model choose between them.
  • Evaluate per site with a repeatable harness (promptfoo), on accuracy and latency, and rerun it when new models come out.
  • Prefer offline generation plus an LLM judge over live generation when brand safety matters.
  • Log every generated page with its plan and timings, and measure the business impact.
Talk 3
Change is inevitable: versioning event-driven systems
Laila Bougria

Teams spend hours debating service boundaries, , , . The thing that gets left out is change. In a , events outlive the code that produced them, so change has to be designed for from the start, not handled when it shows up.

That starts with how events are designed.

Granular, business-meaningful events

Take a credit card service. The tempting design is one generic event, CreditCardTransactionMade, with an operationType of debit or credit. It works until the requirements start arriving: hotels reserving funds, fees on cash withdrawals, payments in another currency. Each one adds a new operation type or field, and every has to learn what it means.

Five requirements later
Ship each new requirement and watch the consumer
Event types
1
Consumer if/else
1
Producer emits
{
  "type": "CreditCardTransactionMade",
  "cardId": "card_42",
  "operationType": "Debit",
  "amount": 120
}
Balance consumer
on("CreditCardTransactionMade", (e) => {
  if (e.operationType === "Debit") balance.spend(e.amount);
  else balance.refund(e.amount);
});
A type field means every consumer has to know which values exist, which fields go with which value, and what each one means for them. That hidden knowledge is functional coupling.

Every if (e.operationType === ...) in a consumer is knowledge about the producer's business rules that has leaked out. That's : consumers have to understand values, field combinations and implicit rules that have nothing to do with their own logic.

The alternative is one event type per business fact, named the way the business talks about it:

// One generic event: the meaning depends on a field
type CreditCardTransactionMade = {
	cardId: string;
	operationType: "Debit" | "Credit" | "ReserveFunds" | "Cost";
	amount: number;
	reservedUntil?: string; // only for ReserveFunds
	cost: number; // only meaningful for some debits
};

// Granular events: the meaning is in the name, each carries only its own fields
type PaymentMade = { cardId: string; amount: number; currency: string };
type RefundIssued = { cardId: string; amount: number; currency: string };
type FundsBlocked = { cardId: string; amount: number; currency: string; until: string };
type CostsIncurred = { cardId: string; amount: number; currency: string; reason: string };
  • A type field is a red flag. If a consumer needs an if/else to know what it received, there are several event types hiding in one.
  • The meaning never changes. It's clear from the name alone.
  • Each event evolves on its own, and consumers subscribe only to what they care about, with no conditionals.

The generic design also makes breaking changes more likely. Removing a field that used to be required turns every message into a : consumers on the old contract can't it, no matter how many retries, so it lands in the . "Just upgrade" doesn't help either, because consumers still have in-flight messages, delayed messages scheduled under the old contract, and failed messages waiting to be retried. A producer-driven break forces the change onto the consumer's schedule, or forces the producer to roll back.

Public vs. private events, and summary events

Granular events can be too granular for some consumers. Billing doesn't want every payment and refund. It wants the balance. If it subscribes to the granular events and adds them up itself, the functional coupling is back: it has to know that PaymentMade, RefundIssued and CostsIncurred count toward the balance and FundsBlocked doesn't, and it has to change whenever that rule changes.

The fix is to separate private events, used inside the , from public events, published to the rest of the system. A summary event aggregates the private ones inside the boundary and publishes one public fact:

// Inside the card service: private events in, one public event out
on("PaymentMade", (e) => period(e.cardId).add(e.amount));
on("RefundIssued", (e) => period(e.cardId).subtract(e.amount));
on("CostsIncurred", (e) => period(e.cardId).add(e.amount));

on("BillingPeriodEnded", async ({ cardId }) => {
	const p = period(cardId); // the card's own cycle, e.g. the 4th to the 3rd
	await publish("BalanceSummarized", {
		cardId,
		period: p.range,
		balance: p.total,
		currency: p.currency,
	});
});
How far does a change travel?
Costs move from monthly to yearly billing
Card service · inside the boundary
PaymentMadeRefundIssuedCostsIncurredFundsBlocked
Other teams · outside
  • Billing
    charges the balance, subscribes to
    PaymentMadeRefundIssuedCostsIncurred
  • Statements
    prints the monthly statement, subscribes to
    PaymentMadeRefundIssuedCostsIncurred
  • Notifications
    warns near the limit, subscribes to
    PaymentMadeRefundIssuedCostsIncurred
Flip the switch to ship the change.
The summary event hides two business rules: which events count toward the balance, and when a billing period starts and ends. Both can change without anyone outside noticing.
  • The change radius stays inside the boundary. The team that owns the private events also owns all their consumers.
  • Keep granular events private by default, and expose them only when another team truly needs them. A summary event is the more likely public one.
  • Share as little as possible. Once data is public, you lose control over who depends on it, and every shared field makes versioning harder.

Change is a process, not a moment

In a distributed system, a change doesn't happen at one instant. Old and new messages coexist for a while, so every change needs coordination. That means two policies, decided before the first production release: a versioning strategy and an upgrade strategy.

A schema is the contract, not the encoding

"It's JSON" is an encoding, not a contract. Without a schema, consumers guess: is cardId always there, what precision does amount have, is temperature in Celsius, Fahrenheit or Kelvin? Wrong guesses mean messages get processed the wrong way, silently. A schema states those things explicitly. Here it is in , a common schema format for events:

{
	"type": "record",
	"name": "PaymentMade",
	"namespace": "com.example.cards",
	"fields": [
		{ "name": "cardId", "type": "string", "doc": "Card the payment was made with" },
		{
			"name": "amount",
			"type": { "type": "bytes", "logicalType": "decimal", "precision": 12, "scale": 2 },
			"doc": "Amount in the payment currency"
		},
		{ "name": "currency", "type": "string", "doc": "ISO 4217 code, e.g. MAD" },
		{ "name": "merchantNote", "type": ["null", "string"], "default": null }
	]
}

Avro in two minutes

Avro is an Apache data format built for exactly this job. It has two parts:

  • The schema, written in JSON like the one above: field names, types, documentation and defaults.
  • The data, a compact binary encoding. It holds only the values, in schema order: no field names, no types, no punctuation.

That second point is both the trick and the catch. The bytes are small, but they're meaningless without the schema that wrote them. That's why Avro is usually paired with a schema registry: each message starts with a small header carrying the schema's ID, and the consumer fetches that schema to decode it.

The same event, as Avro
Real Avro bytes, encoded in your browser
Avro bytes (16)
0e 63 61 72 64 5f 34 3204 2e e006 45 55 5200
  • cardId0e 63 61 72 64 5f 34 32length 7, then "card_42"
  • amount04 2e e02 bytes: 12000 cents (decimal, scale 2)
  • currency06 45 55 52length 3, then "EUR"
  • merchantNote00union branch 0: null, nothing else
JSON70 B
Avro16 B
Avro + registry header21 B
Avro bytes carry no field names and no types, only values in schema order. That makes them small, and it means nobody can read them without the schema that wrote them.

When the consumer's schema is newer or older than the producer's, Avro resolves the two by field name. Fields missing from the message get the default from the reader's schema, fields the reader doesn't know are skipped, and a required field with no value and no default is an error. The compatibility modes below are built on exactly those rules.

A few more things worth knowing:

  • Optional means "union with null, default null": ["null", "string"] with "default": null, like merchantNote above.
  • Types: primitives (string, int, long, double, boolean, bytes), complex types (record, enum, array, map, unions), and logical types on top of them, like decimal, date, timestamp-millis or uuid.
  • Money is a decimal, not a double, so amounts never pick up floating-point rounding errors.
JSONAvro
On the wireText, field names in every messageBinary, numbered fieldsBinary, values only
SchemaOptional, via .proto filesJSON, needed to read anything
Fields matched byNameNumberName, against the writer's schema
Typical homeHTTP APIsgRPC and internal servicesKafka and event streams

And once there's a schema, it can be versioned under a compatibility mode:

  • Forward compatible (producers first). Consumers on the old schema can read messages written with the new one. You can add any field and delete optional fields. A good fit for short-lived messages on .
  • Backward compatible (consumers first). Consumers on the new schema can read messages written with the old one. You can delete any field and add optional fields. A good fit for long-lived messages, like topics or .
  • Full. Both at once: only optional fields can be added or deleted. That doesn't mean "make everything optional", which would defeat the point of having a schema.
  • Transitive. Any of the above, checked against all previous versions instead of just the last one. The strictest option, for very long-lived events or parts of the system that upgrade at different speeds.
Which changes does each mode allow?
Pick a mode, then try a change against the registry
Producers upgrade first. Short-lived messages: queues like RabbitMQ, SQS, Service Bus.
ChangeForwardBackwardFull
Schema registryrejected
$ registry publish card.payment-made --schema v3.avsc --compatibility forward
# delete a required field: periodId
ERR rejected: not forward compatible
    consumers still on v2 require it, and new messages don't have it
Any of the three can also be made transitive: checked against every earlier version, not just the previous one. That is the strictest option, for very long-lived events.

One trap: schema compatibility is not . If schemas are shared as a package of generated classes, removing a property looks like a major SemVer bump, even when it's perfectly forward compatible. Don't apply SemVer to those packages. Schema compatibility is only about whether a message can still be deserialized.

Headers are part of the contract too

A message is a payload plus metadata. Headers get used for routing by intermediaries and for processing by consumers, so changing them can break things just as badly. Filtering on headers is usually a sign that one event type means several things, which should be separate event types.

(a specification) structures that metadata. Four attributes are required, id, source, specversion and type, and an extension model covers attributes specific to your system:

{
	"specversion": "1.0",
	"id": "9b2f6c1e-8d0a-4c7e-a3b5-1f2e3d4c5b6a",
	"source": "/cards/transactions",
	"type": "com.example.cards.payment-made.v2",
	"time": "2026-11-04T10:42:01Z",
	"datacontenttype": "application/json",
	"dataschema": "https://registry.example.com/schemas/payment-made/2",
	"cardregion": "eu",
	"data": { "cardId": "card_42", "amount": 120, "currency": "EUR" }
}

(cardregion is an extension attribute.) CloudEvents has SDKs and protocol bindings for pretty much every messaging protocol.

When you have to break the contract

Sometimes compatibility can't be kept. Then the rollout is a procedure:

  1. Deprecate the old version and set a removal deadline.
  2. Notify the consumers.
  3. Dual publish the old and the new event until the deadline, so consumers can move at their own pace.
  4. Stop publishing the old version at the deadline. Be generous with the window (three to six months) and strict about the date. Extending it for one consumer costs everyone their autonomy.

On the consumer side, the order of operations matters. With a per event type, subscribe to the new topic first, then stop the old subscription by changing its to false, without deleting it. Deleting a subscription deletes its queue, along with the in-flight messages.

Moving consumers from v1 to v2
A breaking change, rolled out without losing a message

One event type per topic, one subscription queue per consumer.

v1
Publisher
emits
Topic
payment-v1
Queue
subscribed2
Handler
running
v2
Publisher
stopped
Topic
payment-v2
Queue
no subscription0
Handler
removed
Messages lost
0
Duplicates
0
Outcome
All good
Subscribing and changing a filter are two separate operations; no transaction covers both. Subscribe first and accept duplicates, then let an idempotent handler drop them by logical message id.
// 1. Subscribe to v2 first
await broker.subscribe({ topic: "payment-made-v2", subscription: "billing" });

// 2. Then stop new v1 messages, keeping the queue (on Service Bus: the SQL filter "1=0")
await broker.updateFilter({ topic: "payment-made-v1", subscription: "billing", filter: "1=0" });

// 3. Both versions overlap for a while: deduplicate on a logical id that's the same in v1 and v2
export async function onPaymentMade(event: PaymentMadeV1 | PaymentMadeV2) {
	if (await processed.has(event.paymentId)) return;
	await db.transaction(async (tx) => {
		await billing.charge(tx, toV2(event));
		await processed.add(tx, event.paymentId);
	});
}

Enforce the promise with a registry

A compatibility mode is only a promise until something enforces it. A is the single source of truth for schemas, shared by producers and consumers. It rejects any new version that breaks the chosen mode, and producers can validate payloads on the way out, so a poison message never reaches the broker.

export async function publish(type: string, data: unknown) {
	const schema = await registry.latest(type); // registering it already enforced the compatibility mode
	schema.validate(data); // a payload that doesn't match never leaves the service
	await broker.send({
		specversion: "1.0",
		id: crypto.randomUUID(),
		source: "/cards/transactions",
		type,
		dataschema: schema.url,
		data,
	});
}

The catch is that most registries are tied to a or protocol: 's to Kafka, ' to Event Hubs. That leaves , or users without a natural option.

is the CNCF answer, from the same group behind CloudEvents. It's a protocol- and vendor-agnostic set of specifications for registries of:

  • Schemas, with compatibility defined in the core specification.
  • Message definitions, meaning the envelope and headers, not just the payload.
  • Endpoints: which service emits which message, carrying which schema.

It also has a deprecation model. Consumers can subscribe to registry events like "a new version was added", "this schema is deprecated, use that one", or "this endpoint will be decommissioned". Producers no longer need to know who their consumers are to tell them about a change, which keeps producers and consumers decoupled even during an upgrade.

Key takeaways
  • Design granular, business-meaningful events. One event type per business fact, and treat any type field as a red flag for functional coupling.
  • Keep granular events private inside the service boundary, and publish summary events to the rest of the system.
  • Pick a versioning strategy before the first production release: forward, backward, full, or transitive, based on how long messages live.
  • Use a schema so the contract says what's required, the precision, and the units. "It's JSON" isn't a contract.
  • Treat headers as part of the contract, and structure them with CloudEvents.
  • Enforce compatibility with a registry, or it's just a promise.
  • Have a breaking-change runbook: deprecate, notify, dual publish, subscribe before unsubscribing, keep handlers , and stick to the deadline.
Next: Day 2What I learned at Devoxx Morocco 2026: Day 2 🇲🇦Key ideas from day two of Devoxx Morocco 2026, talk by talk, with interactive examples.