Event buffers: programming

This page covers using event buffers from code.

For the concepts and day-to-day use, see Event buffers: overview.

For full descriptions of the programming interfaces (e.g. all the optional parameters that aren't mentioned here), see the API reference for C++ (bm_xxx), Python (midas.client), JavaScript (midas.js) and the JSON-RPC methods.

Buffers are implemented as shared memory buffers on the host machine. Programs running on remote machines can access the buffers via mserver. The code interfaces handle this transparently.

Each buffer is organized as a FIFO (First-In-First-Out) memory. Consumers can specify which type of events they want to receive from a buffer, using the Event ID and Trigger Mask options present in each event header.

The Event ID of the request and the event must match exactly. Use the special value -1 to match all events.
The Trigger Mask is a bitmask that will match if any bit of the request matches the event's trigger mask. Use the special value -1 to accept all trigger masks.

Blocking vs non-blocking access

When registering to receive events from a buffer you can specify which events to read:

GET_ALL means that you want to read all events. If your program can't consume events fast enough, the buffer will get full and the producer will have to slow down. Use this for cases like data loggers that absolutely cannot miss any events.
GET_NONBLOCKING means that you are happy to miss events if you can't keep up. The producer can still write events.
GET_RECENT means that you are only interested in events if they are less than 1s old. It's like GET_NONBLOCKING but prioritises more recent events. Use this for cases like data visualisers that should show "live" data.

Connecting to an event buffer

If the buffer does not already exist, it will be created for you with the specified size (which is an optional argument in the Python interface).

C++Python

HNDLE buffer_handle;
INT status = bm_open_buffer("SYSTEM", DEFAULT_BUFFER_SIZE, &buffer_handle);

buffer_handle = client.open_event_buffer("SYSTEM")

Sending an event

Note that frontend frameworks have their own systems for sending events. These code examples are purely for users writing other programs that want to send events to buffers.

C++Python

// Create the event
char* payload = (char*)malloc(1e6);
EVENT_HEADER* pevent = (EVENT_HEADER*)payload;
pevent->event_id = 123;
pevent->trigger_mask = 0;
pevent->time_stamp = ss_time();
pevent->serial_number = 1;

char* pdata = (char*) (pevent + 1);

bk_init32(pdata);

WORD* bk_data;
bk_create(pevent, "MYBK", TID_WORD, (void **) &bk_data);

*bk_data++ = 1;
*bk_data++ = 2;
*bk_data++ = 3;
*bk_data++ = 4;
bk_close(pdata, bk_data);

pevent->data_size = bk_size(pdata);

// Send the event
INT status = bm_send_event(buffer_handle, pevent, 0, BM_WAIT);

# Create the event
event = midas.event.Event()
event.header.event_id = 123
event.header.serial_number = 1
event.create_bank("MYBK", midas.TID_WORD, [1,2,3,4])

# Send the event
client.send_event(buffer_handle, event)

Consuming events

There are two ways to read events from a buffer. You can either manually read them, or have a callback function get executed whenever an event is available.

Reading events manually

In this version we say that we will manually read events from the buffer later.

Registering the event request

C++Python

INT request_id = 0;
INT status = bm_request_event(buffer_handle, 123, TRIGGER_ALL, GET_ALL, &request_id, nullptr);

The fourth argument chooses how to read the buffer. With GET_ALL, your program must read the events fast enough, or the programs writing to the buffer will have to wait. To miss events rather than slow the experiment down, use GET_NONBLOCKING or GET_RECENT instead (see Blocking vs non-blocking access):

INT status = bm_request_event(buffer_handle, 123, TRIGGER_ALL, GET_RECENT, &request_id, nullptr);

request_id = client.register_event_request(buffer_handle, event_id = 123)

The request uses GET_ALL unless you say otherwise, so your program must read the events fast enough, or the programs writing to the buffer will have to wait. To miss events rather than slow the experiment down, give a different sampling_type (see Blocking vs non-blocking access):

request_id = client.register_event_request(buffer_handle, event_id = 123, sampling_type = midas.GET_RECENT)

Reading the event

C++PythonJS

char* destination = (char*)malloc(1e6); //(1)!
char banklist[16384] = {};

while (true) {
  INT size = sizeof(destination);
  INT status = bm_receive_event(buffer_handle, destination, &size, BM_NO_WAIT);

  if (status == CM_SUCCESS) {
    EVENT_HEADER* ev_header = (EVENT_HEADER*) destination;
    BANK_HEADER *bk_header = (BANK_HEADER*) (destination + 1);
    bk_list(bk_header, banklist);

    printf("Got event with event ID %d, serial number %d, banks %s\n", pheader->event_id, pheader->serial_number, banklist);
  }

  status = cm_yield(0);

  if (status == RPC_SHUTDOWN || status == SS_ABORT) {
    break;
  }
}

Ensure this is large enough for your maximum event size!

while True:
    # If there's an event ready, `event` will contain a `midas.event.Event`
    # object. If not, it will be None. If you want to block waiting for an
    # event to arrive, you could set async_flag to False.
    event = client.receive_event(buffer_handle, async_flag=True)

    if event is not None:
        print(f"Got event with event ID {event.header.event_id}, serial {event.header.serial_number}, #banks = {len(event.banks)}")

The web server handles connecting to the event buffer each time you call bm_receive_event. As this is an ephemeral connection, you must tell the web server which event you last received, or else you run the risk of being sent the same event multiple times (if no other events got written to the buffer since your previous request).

let last_event;

function get_event() {
  let params = Object();
  params.buffer_name = "SYSTEM";
  params.event_id = 123;
  params.get_recent = true;

  if (last_event) {
    params.last_event_header = [last_event.event_id, last_event.trigger_mask, last_event.serial_number, last_event.time_stamp];
  }

  mjsonrpc_call("bm_receive_event", params, "arraybuffer").then(function (rpc) {
    parse_event(rpc);
    setTimeout(get_event, 1000);
  }).catch(function (error) {
    setTimeout(get_event, 1000);
  });
}

function parse_event() {
  if (rpc.result) {
    let status = rpc.result.status;

    if (status === 209) {
      console.log("No event available");
    } else {
      console.log("Event request failed - " + status);
    }

    return;
  }

  let event = bkToObj(rpc);

  // Cache the event, so we can tell the server to not send us the same event again next time...
  last_event = event;

  console.log("Got event with event ID " + event.event_id + ", serial " + event.serial_number + ", #banks = " + event.bank.length);

  // Do all your other work
}

Executing a callback function

In this version a callback function will be called whenever a new event is available. Deep down, the callback function gets executed as part of cm_yield().

Note that this functionality is not currently available in Python.

C++

void process_event(HNDLE hbuf, HNDLE request_id, EVENT_HEADER *pheader, void *pevent) {
  char banklist[16384] = {};
  BANK_HEADER *bk_header = (BANK_HEADER*) (pheader + 1);
  bk_list(bk_header, banklist);

  printf("Got event with event ID %d, serial number %d, banks %s\n", pheader->event_id, pheader->serial_number, banklist);
}

// Later, in main function
INT request_id = 0;
INT status = bm_request_event(buffer_handle, 123, TRIGGER_ALL, GET_ALL, &request_id, process_event);

while (true) {
  INT status = cm_yield(1000);

  if (status == RPC_SHUTDOWN || status == SS_ABORT) {
    break;
  }
}

Cancelling an event request

C++Python

INT status = bm_remove_event_request(buffer_handle, request_id);

client.deregister_event_request(buffer_handle, request_id)