In our 0.3.0-beta.1 SDK release we’re excited to announce the beta for the new ‘Bring Your Own Frame Buffer’ (BYOFB) feature in addition to bugfixes and improvements. In order to participate in this beta release, all SDK runtimes must be updated to 0.3.0-beta.1 to continue to communicate with each other - this beta offering is not backward compatible with our stable 0.2.8 release.

Bring Your Own Frame Buffer

We’ve introduced a new set of functions to the Rainway SDK that allow for direct submission of audio and video data within the context of a Rainway Stream. These include rainway_submit_frame(...), rainway_submit_audio(...) for submitting video and audio data, and an onStreamInput(...) callback to process remote input.

We expect that most folks will leverage this new functionality via our C++ SDK, so we’ve included some snippets below demonstrating how it works at a high level, using C++. For example, to submit video frames:

// stream to submit frame / audio to.
// You can obtain a Stream from the SDK's `onStreamStart` event.
Stream stream;

extern winrt::com_ptr<ID3D11Texture2D> renderFrame();
auto frame = renderFrame();
stream.submitFrame(frame);

and to submit audio:

Stream stream;

extern int16_t *audioBuffer;
extern usize_t audioBufferSamples

auto submission = RainwayAudioSubmit {
    44100, // samples per sec
    2, // channels
    RainwayAudioBuffer {
        RAINWAY_AUDIO_BUFFER_PCM, // buffer type
        audioBuffer, // buffer of samples
    },
    audioBufferSamples, // number of samples in buffer
};
stream.submitAudio(submission);

Handling stream input is done with the onStreamInput event:

config.onStreamInput = [&](const Runtime &runtime, const Stream &stream, const Peer &peer, const RainwayInput &input) {
    switch (input.tag) {
    case RAINWAY_INPUT_MOUSE_ABSOLUTE:
        mouseAbsoluteInput(peer, input.MOUSE_ABSOLUTE.x, input.MOUSE_ABSOLUTE.y);
        break;
    case RAINWAY_INPUT_KEYBOARD:
        keyboardInput(peer, input.KEYBOARD_INPUT.keycode);
        break;
    // ...
    }
};

It’s important to note that while we believe BYOFB will be extremely valuable for several use cases, our high-level APIs aren’t going away! You may continue to build applications with Rainway automatically handling capture and input exactly as you would today.

Improved Support for Multiple Peers and Streams

It is now possible for two or more peers to subscribe to the same stream. In addition, we’ve also added the ability to host several streams at the same time, provided you have a graphics card that supports it.

These improvements are available for both our new BYOFB streams and existing full-desktop or App Isolation streams.

Native SDK consumers can now subscribe to several new events that relate to the lifetime of a stream (whether using BYOFB or not).

Event Description
config.onStreamStart A local stream has started.
config.onStreamEnd A local stream has ended, normally as the result of all other peers leaving.
config.onStreamPeerJoin A new peer has joined an existing stream.
config.onStreamPeerLeave A peer has left a local stream.
config.onStreamError An error occured relating to a stream.

Bugfixes and Improvements

  • [Documentation] Documented new BYOFB APIs.
  • [Examples] Added a new C++ MP4 playback using BYOFB example demonstrating BYOFB capabilities.
  • [Web SDK] Fixed types, making configuration callbacks optional.
  • [Web SDK] Added dispose method on the runtime, to faciltate Rainway SDK cleanup.
  • [.NET SDK] Fixed an issue that could lead to an unexpected exception when a stream was terminated.
  • [Native SDKs] Added support for querying active stream IDs to peers.
  • [Bugfix] Made createDataChannel async in all SDK runtimes, since creation does not occur synchronously.