At Rainway, we develop real-time streaming software with high performance demands. We use schema-based serialization to make all the parts of our application communicate with each other quickly and type-safely. Dissatisfied with the available formats like Protocol Buffers and FlatBuffers, we created our own open-source solution: Bebop. (You can read more about Bebop here).

A few months ago, we added a way to define your own tagged union types in the schema language. They’ve turned out to be awesome! This post explains why they work so well for us, and what we like about our implementation. If you’re using Bebop in a project, we’re sure you’ll find them useful, too.

How did we get here?

Early on when using Bebop for our Rainway Gaming service, we were writing a bunch of records side-by-side, like so:

// A message telling the remote to simulate a keypress.
struct KeyboardInput { int32 keyCode; bool isDown; }

// A message telling the remote to simulate a mouse movement.
struct MouseMoveInput { int32 x; int32 y; bool isAbsolute; }

// A message telling the remote to simulate a mouse click.
struct MouseClickInput { MouseButton button; bool isDown; }

Bebop, like any serialization solution, generates some model classes or interfaces, and all the encode and decode functions for translating between binary buffers and those model types.

But, when a binary buffer arrives over the network, how can you know which type to decode it as?

To solve this, we first introduced the idea of opcodes. An opcode is a 4-byte identifier for a record in your Bebop schema. You define opcodes in the schema file, and then prepend them to your Bebop-encoded buffers before sending them over the wire. It looks like this:

[opcode("Keyb")]
struct KeyboardInput { int32 keyCode; bool isDown; }

[opcode("Mous")]
struct MouseMoveInput { int32 x; int32 y; bool isAbsolute; }

[opcode("Clic")]
struct MouseClickInput { MouseButton button; bool isDown; }

Then the receiver does something like the below to figure out which Bebop function to call:

void HandleInputMessage(void* buffer) {
  uint32 opcode = readUint32LE(buffer);
  switch (opcode) {
    case KeyboardInput::opcode:
      PressKey(KeyboardInput::decode(buffer + 4));
      return;
    case MouseMoveInput::opcode:
      MoveMouse(MouseMoveInput::decode(buffer + 4));
      return;
    case MouseClickInput::opcode:
      ClickMouse(MouseClickInput::decode(buffer + 4));
      return;
  }
}

This worked for a while, but the feeling that we were trying to wrap a feature “around” Bebop gnawed at us. The opcode-plus-buffer format was just an ad-hoc way to tag Bebop messages that didn’t really mesh with the rest of the wire format. Could we do better?

Tagged unions to the rescue!

We realized that what we were doing — sending a tag, and then a value whose format depends on the tag — was essentially a wire format for a tagged union, à la Haskell or Rust:

-- An "algebraic data type" in Haskell, a functional programming language.
-- (They're also known as "sum types" in that environment.)
data InputMessage = KeyboardInput Int Bool
                  | MouseMoveInput Int Int Bool
                  | MouseClickInput MouseButton Bool
// Rust implements the same idea in its "enum" definitions.
enum InputMessage {
  KeyboardInput { keyCode: i32, isDown: bool },
  MouseMoveInput { x: i32, y: i32, isAbsolute: bool },
  MouseClickInput { button: MouseButton, isDown: bool },
}

Enviously, we implemented tagged unions in the Bebop schema language. It looks like this:

// A tagged union in Bebop.
union InputMessage {
  1 -> struct KeyboardInput { int32 keyCode; bool isDown; }
  2 -> struct MouseMoveInput { int32 x; int32 y; bool isAbsolute; }
  3 -> struct MouseClickInput { MouseButton button; bool isDown; }
}

The numbers before the arrows are the (now single-byte) “tags” or “discriminators” (Bebop makes you explicitly write these byte values out to discourage you from changing them and breaking the format).

Essentially, this definition is saying: the wire format of an InputMessage is either a byte 1 followed by a KeyboardInput, or a byte 2 followed by a MouseMoveInput, et cetera.

These unions may be nested inside other structs, or inside each other. Here at Rainway, we’ve now gotten rid of opcodes again, and the format for our application messages is a union of every relevant message. It works great!

The benefits are twofold: Firstly, the schema itself now accurately describes the variety of messages and tags that occur. Secondly, the user now doesn’t need to write a tricky function like HandleInputMessage above, that peeks inside the buffer and makes a decision. They simply call InputMessage::decode on the buffer and handle the resulting nicely-typed value.

But what is that value?

Generating obvious code

The hard part about these tagged union types is deciding how to represent them in the generated code. Many popular programming languages don’t “natively” support tagged unions the way Haskell or Rust do. So, how do we represent something like InputMessage in C++ or C#?

Nevertheless, Bebop is not the only schema language with a feature like this. Protocol Buffers has a similar feature called oneof, but the generated C++ code, for example, is quite different. Let’s compare our approaches.

Protocol Buffers

Here is a small Protocol Buffers schema demonstrating oneof. It defines foo, a tagged union of a string and an integer.

syntax = "proto3";

message SampleMessage {
  oneof foo {
    string x = 1;
    int32 y = 2;
  }
}

Now let’s look at the protoc C++ output for this simple union type (I’ve snipped out a lot of irrelevant lines).

class SampleMessage : public ::google::protobuf::Message {
 public:
  SampleMessage();
  // ...

  enum FooCase {
    kX = 1,
    kY = 2,
    FOO_NOT_SET = 0,
  };

  // ...
  const ::std::string& x() const;
  void set_x(const ::std::string& value);

  // ...
  ::google::protobuf::int32 y() const;
  void set_y(::google::protobuf::int32 value);

  // ...
  void clear_foo();
  FooCase foo_case() const;

  // ...
 private:
  union FooUnion {
    FooUnion() {}
    ::google::protobuf::internal::ArenaStringPtr x_;
    ::google::protobuf::int32 y_;
  } foo_;
};

In this example, protoc generates a class with a foo_case() function telling which of the cases you may access, and x() and y() accessors that won’t cause an error if you access them at the wrong time. Behold:

inline ::google::protobuf::int32 SampleMessage::y() const {
  if (has_y()) {
    return foo_.y_;
  }
  return 0;  // <-- Fail silently?
}

Interface-wise, this is a “product pretending to be a sum”. x() and y() are both always available, but there’s an invisible expectation to only use the right one. It’s quite easy to use this class wrong and not even realize it, leading to mysterious bugs.

Bebop

In Bebop, unions are valid top-level definitions, but their branches must not be raw values. The corresponding schema therefore looks like this:

union Foo {
   1 -> struct X { string x; }
   2 -> struct Y { int32 y; }
}

Bebop speaks a more modern C++. It generates a class that’s easy to read, and almost impossible to use wrong. Instead of a C-style union with a precarious interface, it uses std::variant:

struct X {
  std::string x;
  // encode and decode functions...
}

struct Y {
  int32_t y;
  // encode and decode functions...
}

struct Foo {
  std::variant<X, Y> variant;
  // encode and decode functions...
}

After decoding a Foo from a buffer, you can always safely access its variant field. C++17, in turn, provides a function called std::visit that helps match on this value and handle all the cases. In a future blog post, we’ll talk about how we use this in practice.


Being in control of the generated code, and thereby being able to keep it lightweight and modern, is one of the reasons we built Bebop. We encourage you to give Bebop a try, even if a schema language sounds like more complexity than you need! Our motto is to make it about as simple to use as JSON, but faster and safer. If you have similar thoughts and love what you’ve read, come work with us! Check out our openings here.