Backward and Forward Compatibility

tldr

• Distinguish between “breaking change” vs “backward compatible change”
• There are writers and readers
• backward/forward compatible changes strictly speaking are from reader’s perspective
• Even when talking about schema evolutions, compatibility is based on if the reader can read old/new data produced by the writer

Overview

I’ve heard about Avro before but I never used it. Also I was confused about “backward compatibility” and “forward compatibility”. Sometimes I use it in my day to day work but I have the feeling that I’m not using it correct.

So I went to vibe with Gemini and Claude to figure things out. Ironically Gemini did a poor job despite I pay it 20 bucks a month, where Claude explained it nice to me even on the free version.

Again this is me learning things with LLMs and trying to write it down, there might be mistakes but for now it’s the best to my knowledge that I believe what I’m writing is correct.

My confusions

There are 2 confusions before I started to dig a bit deeper.

1. The usual things I say at work, does it make sense?
2. What is forward compatibility?

The usual things I say at work

My new backend change is backward compatible, we can move it without coordinating the frontend

After talking with Claude I think the above is understandable but technically incorrect. What I actually mean is:

My new backend change is not a breaking change, we can move it without coordinating the frontend.

If I really want to use the term backward/forward compatible to describe it I would say:

My new backend change is not a breaking change because our clients are forward compatible.

The above is an example of publishing server changes, but similar conversations also happen when people decide if they should bump the major of their semver.

You can’t remove that function because it would break backward compatibility, if you want to do that you have to bump the major.

Where the above should be technically

You can’t remove that function because it would be a breaking change, to release a breaking change you need to bump the major.

What is forward compatibility?

Can an old reader handle new data? If yes then the reader is forward compatible with new data.

This really is it. But it gets a bit confusing when people like me misuse “backward compatible” with “non-breaking”. I’ll try to explain what backward/forward compatibility is in the following sections.

Note that I didn’t explain what is a reader and what is a writer formally. Let’s have a look together.

The definition

Let’s put aside the difference between breaking changes and backward compatible changes for a second and construct our mental model with the following.

There are two parties and a schema

1. Producer/Writer
2. Consumer/Reader
3. A schema

When we talk about backward compatibility, it is always referring to if the reader can handle old (looking backward) data.

When we talk about forward compatibility, it is always referring to if the reader can handle new (looking forward) data.

Schema changes (aka schema evolution) could be categorized as backward compatible or forward compatible by testing if an old reader can handle new data (forward compatibility) and if a new reader can handle old data (backward compatibility).

How can a reader be backward compatible

Assume that I have a v2 reader and it has the following code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

type Data struct {
    Name string `json:"name"`
    Age int `json:"age,omitempty"`
}

func handlePayload(payload json.RawMessage) {
    var data Data
    err := json.Unmarshal(payload, &data)
    if err != nil {
        panic(err)
    }

    fmt.Printf("%+v\n", data)
}

This reader can take v1 payloads like

1
2
3

{
    "name": "Bingus"
}

It can also take v2 payloads like

1
2
3
4

{
    "name": "Bingus",
    "age": 21
}

The fact that the v2 reader could take a payload with only name in it (the v1 payload) means the reader can handle old data, hence it is backward compatible.

How can a reader be forward compatible

Same example but let’s go to the v1 reader.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

type Data struct {
    Name string `json:"name"`
}

func handlePayload(payload json.RawMessage) {
    var data Data
    err := json.Unmarshal(payload, &data)
    if err != nil {
        panic(err)
    }

    fmt.Printf("%+v\n", data)
}

This reader can take v2 payloads like

1
2
3
4

{
    "name": "Bingus",
    "age": 21
}

Because age is simply ignored.

The fact that the v1 reader can take a v2 payload means the reader can handle new data, hence it is forward compatible

Common schema changes

People also say if a schema evolution (schema change) is backward or forward compatible or not.

When they say that they meant with the new schema change from v1 to v2, how does a v2 reader handle v1 data? how does a v1 reader handle v2 data?

Note that this is only a general discussion, different protocols have different tools to handle changes, like Avro has an alias thing to help renaming

Common schema changes are

Adding an optional field

backward compatible?

Because the field is optional, a v2 reader should be able to handle v1 data (where the field was not presented). This means the change is backward compatible.

forward compatible?

A v1 reader should be able to ignore the new field, hence it is also forward compatible.

verdict

Adding an optional field to the schema is full compatible

Adding a required field

backward compatible?

Adding a required field means a v2 reader needs the new field, so v2 reader won’t be able to work with v1 data, meaning the schema change is not backward compatible.

forward compatible?

The v1 reader doesn’t know about the new required field, so when it receives v2 data, it simply ignores the extra field. Hence, the schema change is forward compatible.

verdict

Adding a required field to the schema is forward compatible

Removing a required field

backward compatible?

By removing a required field, the v2 reader would be fine with old data because it no longer reads it. Hence it is backward compatible

forward compatible?

The v1 reader won’t be fine because new data would not send back a field required by v1, hence this is not forward compatible

verdict

Removing a required field from the schema is backward compatible

Renaming a field

This is a bit tricky because people would say: I can create a new field with a new name and keep my old field. That could be in the “add a new optional field” or “add a new required field” based on how you do it.

Let’s assume that we are (1) only renaming, and (2) the field is a required field

backward compatible?

A v2 reader won’t be able to handle old data because it is looking for a required field that does not exist. Hence not backward compatible.

forward compatible?

A v1 reader won’t be able to handle new data because it is looking for a required field that does not exist. Hence not forward compatible.

verdict

Not compatible at all.

The role the schema plays

Note that in the golang example above the behavior of “ignoring new fields” was handled in application logic. Some schemas provide ways to verify if your schema change is backward compatible or forward compatible by simply checking if the schema change added a new optional field or added a required field or removed a required field etc.

This is where a schema registry would come into play. You can configure your schema to only allow full compatible changes. That way if any changes would introduce incompatibility it would be rejected.

Difference between different schemas

As you can see in the golang example above, respecting an optional field is baked in the application code, the JSON schema itself doesn’t really enforce anything on the application side. The application could totally treat an optional field as a required field. This is something that is really fragile (relying on programmers to do the right thing). Hence different solutions came out.

I’m only touching on the surface of these since I don’t have much experience and today’s topic is backward/forward compatibility.

I might be writing a new post when I drill down to Avro.

JSON Schema

This is the one that relies on programmers to treat the fields the right way. Not much to be talking about here tbh.

Avro

Avro provides two schemas, the writer schema and the reader schema. When you add a new optional field you are also required to set up the default value of the field in the reader’s schema. This way when a reader with v2 tries to read data provided by a v1 writer, the new fields would have default values defined by the reader’s schema.

This is what makes the change backward compatible. (v2 reader reads v1 data)

Avro also provides ways to rename by using the alias keyword.

Protobuf and Thrift

Both protobuf and thrift use one schema, but they generate source code and use that opportunity to set default values of optional fields.

Back to my initial confusion about my daily talk

I think the best is to start distinguish between “non-breaking/breaking” change vs “backward/forward” compatible change.

Technically speaking “my server change is fine because clients are forward compatible” is correct, but it does sound weird doesn’t it?

Future things

• Dive deeper into Avro, Protobuf and Thrift

Quiz time (LLM Generated)

1. A new version of a mobile app (v3) can still open files saved by the old version (v1). Is this backward or forward compatible? Which party is being tested — the file, or the app?
2. Your coworker says: “We can’t remove this API field, it’ll break backward compatibility.” Rewrite this sentence using the more precise term.
3. True or false: “Forward compatible” means data from the future can be read.
4. A schema adds a new required field. Is this backward compatible, forward compatible, both, or neither? Justify using the reader/data relationship, not just the label.
5. In Avro, you add a new optional field to your schema and give it a default value of 0. Which schema — the reader’s or the writer’s — needs to declare that default for backward compatibility to work?
6. A protobuf message removes field number 7 and, six months later, a teammate adds a brand-new field and accidentally reuses number 7. What goes wrong, and for which compatibility direction (backward/forward) does this cause problems?
7. Fill in the blank: “My library’s v2.1.0 release is backward compatible” really means, strictly speaking, that ______ is forward compatible with ______.
8. GraphQL doesn’t really have a “forward compatibility” problem the way JSON or protobuf do. Why not — what about GraphQL’s design makes the unknown-field problem mostly moot?
9. Scenario: a server adds a new field to its response payload and does not bump its major version. An old client, written before this field existed, keeps working fine. Using strict terminology, write one sentence describing this fact from the client’s perspective, and one sentence describing it from the server’s perspective (loose/producer convention).
10. Rank these schema changes from “safest” to “most dangerous,” in terms of compatibility breakage, and say which direction (backward/forward/both) each risks: (a) renaming a field with no alias, (b) adding an optional field, (c) removing a required field, (d) adding a required field.

Answers (remove display:none in the following div yourself)