Communicating between programs

Midas provides several ways for clients to communicate with each other. Which method to use depends on the type of data you want to pass, the volume of data, and whether multiple clients need access to the same data.

Checking if another program is running

You may wish to check whether another program/client is running and connected to midas. You must know the name that the other client used when connecting (including any manipulation due to having multiple instances running, or running a frontend with the -i parameter).

All interfaces let you specify whether to search for an exact match (including numerical suffixes), or to search for any instance of the program. In all cases, setting the "unique" parameter to true requires an exact match. Note the default values are different in JS/Python!

C++PythonJavascript

INT status = cm_exist("other_client_name", true); //(1)!

if (status != SUCCESS) {
  printf("Other client isn't running\n");
}

The second argument is the "unique" flag, and must be specified.

if not client.client_exists("other_client_name"): #(1)!
    print("Other client isn't running")

There is an optional second argument called exact_match (aka the "unique" flag), which defaults to True.

let params = Object();
params.name = "other_client_name"; //(1)!

mjsonrpc_call("cm_exist", params).then(function(rpc) {
  if (rpc.result.status != 1) {
    console.log("Other client isn't running");
  }
}).catch(function(error) {
  mjsonrpc_error_alert(error);
});

There is an optional parameter unique, which defaults to false.

Starting other programs

If a program has a start command defined, it is possible to start that program using code.

Context/environment

As noted in Programs: configuring, the program is started in the context/environment of the calling code. In particular, if your code is running on a remote machine, the launched program will also execute on the remote machine rather than the main midas machine!

C++Python

midas::odb prog("/Programs/other_client_name");
std::string cmd = prog["Start command"];
int pid = 0;
INT status = ss_exec(cmd.c_str(), &pid);

if (status == SUCCESS) {
  printf("Successfully launched other program! It has pid %d.\n", pid);
} else {
  printf("Failed to launch other program: %d!\n", status);
}

client.start_client("other_client_name") #(1)!

Use the name that is in /Programs in the ODB.

Communication options

Feature	ODB polling	ODB hotlinks	Event buffers	JSON RPC	Binary RPC
Multiple clients can access the same data
Reader must send a response
Best for numbers/strings/JSON
Best for binary data
Best for high data rates
Allows "lazy" consumers (they can ignore some updates)

ODB polling

The simplest approach to pass data between programs is for one program to write a value to the ODB, and have another program read from the ODB. Each time the consumer reads from the ODB, it will see the most recent value that was written by the sending program.

This system is useful if you only have a few keys/directories to check, don't expect them to change very frequently, and don't need to be told immediately each time they are changed.

You can have multiple producers and consumers all looking at the same ODB keys.

Producer: See writing a value to the ODB.
Consumer: See reading a value from the ODB.

ODB hotlinks

Your consumer can get notified each time a program writes to the ODB. This is called "watching" an ODB key/directory, or setting up a "hotlink".

This system is useful if you want code to be executed automatically when an ODB value is changed. Take note of the warnings in the hotlinks documentation about strategies to use if you are watching a large directory/array.

You can have multiple producers and consumers all looking at the same ODB keys.

Producer

See writing a value to the ODB.

Consumer

See hotlinks in the ODB.

Note that some frontend frameworks already have a hotlink created for /Equipment/xxx/Settings, and use a different system for that callback function.

Event buffers

You can pass midas events between programs using event buffers.

This is useful if your data is getting produced by a frontend, will eventually end up in a midas data file, or is a binary blob that can benefit from being packaged as a midas event and sent through an event buffer.

You can have multiple producers and consumers all writing to / reading from the same event buffers.

Producer

See sending an event.

Note that frontend frameworks provide their own systems for creating events.

Consumer

See consuming events.

Note that your consumer can choose whether to read all events, or just the most recent ones (i.e. whether the consumer can block the producer if the consumer isn't keeping up).

Remote procedure calls (RPCs)

Remote Procedure Calls (RPCs) are direct connections between programs. The consumer opens a connection to the producer, sends a request, and waits for the response. The response can either be text (for jrpc) or a binary blob (for brpc).

In both cases, the consumer specifies a command and a set of arguments, both of which are strings. Often the arguments are encoded as JSON, but they do not have to be. The text response from jrpc is also often encoded as JSON, but does not have to be.

JSON RPC (jrpc)

This system allows a direct client-to-client connection, with the response being sent as a text string.

The producer registers a callback function, which generates the data and sends a response to the consumer. The callback accepts a command and a set of arguments, both of which are strings. It also accepts a maximum response length, as the calling code may not accept arbitrary-length responses. You should ensure that your response is less than the specified number of characters/bytes.

Our example producers don't do anything with the "arguments" we send (they only look at the "command"). However we show the consumers sending some arguments for completeness.

Javascript interface may not expose the text response

mjsonrpc (the javascript interface for calling JRPC functions) does not return any message if the callback function doesn't return a status code of "SUCCESS". This can be annoying if you want to show a specific error message to the user without them having to trawl through the midas message log.

The python interface provides a way around this, with the optional return_success_even_on_failure argument for register_jrpc_callback(). If you set this parameter to False (the default), then you get the "normal" behaviour, where the returned status code and result string are exactly what is returned from the callback function. If you set this parameter to True, then the status code will always be "SUCCESS", and the result string will be JSON-encoded text of the form `{"code": 604, "msg": "Some error message"}.

For C++ producers, you could implement something similar yourself.

C++ producerPython producerC++ consumerPython consumerJS consumer

Note that the TMFE frontend framework has a different way to implement JRPC callbacks.

INT rpc_callback(INT index, void *prpc_param[]) {
  // Convert the prpc_param array into arguments/types we expect.
  const char* cmd  = CSTRING(0);
  const char* args = CSTRING(1);
  char* return_buf = CSTRING(2);
  int   return_max_length = CINT(3);

  INT retval = SUCCESS;

  if (strcmp(cmd, "some_cmd") == 0) {
    snprintf(return_buf, return_max_length, "Some text");
  } else {
    retval = FE_ERR_DRIVER;
    snprintf(return_buf, return_max_length, "Unknown command %s", cmd);
  }

  return retval;
}

// Later, possibly in main:
cm_register_function(RPC_JRPC, rpc_callback);

while (true) {
  int status = cm_yield(1000);

  if (status == RPC_SHUTDOWN || status == SS_ABORT) {
    break;
  }
}

def rpc_handler(client, cmd, args, max_len):
    ret_int = midas.status_codes["SUCCESS"]
    ret_str = ""

    if cmd == "some_cmd":
        ret_str = "Some text"
    else:
        ret_int = midas.status_codes["FE_ERR_DRIVER"]
        ret_str = f"Unknown command {cmd}"

    return (ret_int, ret_str)

# Later, possibly in main:
with midas.client.MidasClient("some_producer") as client:
    client.register_jrpc_callback(rpc_handler)

    while True:
        client.communicate(1000)

std::string args = "{\"int_list\": [1,2,3,4]}";

HNDLE conn;
INT status = cm_connect_client("some_producer", &conn);

if (status != SUCCESS) {
  printf("Failed to connect to other client!\n");
  exit(1);
}

const int max_len = 100000;
char* resp = (char*)calloc(max_len);
status = jrpc_client_call(conn, "some_cmd", args.c_str(), resp, max_len)

if (status == SUCCESS) {
  printf("The response was: %s\n", resp);
} else {
  printf("Failed to call other client!\n");
}

cm_disconnect_client(conn, FALSE);

jargs = {"int_list": [1,2,3,4]}

conn = client.connect_to_other_client("some_producer")
resp = client.jrpc_client_call(conn, "some_cmd", json.dumps(jargs), 100000)

print(f"The response was: {resp}")

client.disconnect_from_other_client(conn)

let jargs = {"int_list": [1,2,3,4]};

let params = Object();
params.client_name = "some_producer";
params.cmd = "some_cmd";
params.args = JSON.stringify(jargs);
params.max_reply_length = 100000;

mjsonrpc_call("jrpc", params).then(function(rpc) {
  if (rpc.result.status == 1) {
    console.log("The response was: " + rpc.result.reply);
  } else {
    mjsonrpc_error_alert("Got unexpected status " + rpc.result.status);
  }
}).catch(function(error) {
  mjsonrpc_error_alert(error);
});

Binary RPC (brpc)

This system allows a direct client-to-client connection, with the response being sent as a binary blob.

The producer registers a callback function, which generates the data and sends a response to the consumer. The callback accepts a command and a set of arguments, both of which are strings. It also accepts a maximum response length, as the calling code may not accept arbitrary-length responses. You should ensure that your response is less than the specified number of bytes.

In the example code here we just send 4 32-bit signed integers, but you can of course encode any arbitrary data!

The python interface allows you to return a midas event, and it will be encoded to binary for you. It will cost a few bytes of overhead (due to the event/bank headers), but may save you a lot of time and code as you don't have to handle the binary conversion!

C++ producerPython producerC++ consumerPython consumerJS consumer

Note that the TMFE frontend framework has a different way to implement BRPC callbacks.

INT brpc_callback(INT index, void *prpc_param[]) {
  // Convert the prpc_param array into arguments/types we expect.
  const char* cmd  = CSTRING(0);
  const char* args = CSTRING(1);
  char* return_buf = CSTRING(2);
  size_t return_length = CINT(3);

  INT retval = SUCCESS;

  if (strcmp(cmd, "some_cmd") == 0) {
    (int32_t*)pdata = (int32_t*)return_buf;
    *pdata++ = 123456;
    *pdata++ = 101010;
    *pdata++ = 999999;
    *pdata++ = 456789;

    // Set the return length so consumer knows how much data there is!!!
    return_max_length = (sizeof(int32_t)) * 4;
  } else {
    retval = FE_ERR_DRIVER;
    return_max_length = 0;
    snprintf(return_buf, return_max_length, "Unknown command %s", cmd);
  }

  return retval;
}

// Later, possibly in main:
cm_register_function(RPC_BRPC, brpc_callback);

while (true) {
  int status = cm_yield(1000);

  if (status == RPC_SHUTDOWN || status == SS_ABORT) {
    break;
  }
}

import midas.client
import ctypes
import struct

def brpc_handler(client, cmd, args, max_len):
    ret_int = midas.status_codes["SUCCESS"]

    # In python you can return either binary data (as a ctypes char array)
    # or return a midas.event.Event object
    ret_data = ctypes.create_string_buffer(16)

    data = [123456, 101010, 999999, 456789]

    if cmd == "some_cmd":
        struct.pack_into("<IIII", ret_data, 0, *data)
    else:
        ret_int = midas.status_codes["FE_ERR_DRIVER"]

    return (ret_int, ret_data)

# Later, possibly in main:
with midas.client.MidasClient("some_producer") as client:
    client.register_brpc_callback(brpc_handler)

    while True:
        client.communicate(1000)

std::string args = "{\"int_list\": [1,2,3,4]}";

HNDLE conn;
INT status = cm_connect_client("some_producer", &conn);

if (status != SUCCESS) {
  printf("Failed to connect to other client!\n");
  exit(1);
}

const int max_len = 100000;
char* resp = (char*)calloc(max_len);
status = brpc_client_call(conn, "some_cmd", args.c_str(), resp, max_len)

if (status == SUCCESS) {
  int32_t* cast_resp = (int32_t*)resp;
  printf("The response was: %d %d %d %d\n", cast_resp[0], cast_resp[1], cast_resp[2], cast_resp[3]);
} else {
  printf("Failed to call other client!\n");
}

cm_disconnect_client(conn, FALSE);

import midas.client
import json
import struct
import ctypes

jargs = {"int_list": [1,2,3,4]}

with midas.client.MidasClient("some_consumer") as client:
    conn = client.connect_to_other_client("some_producer")
    resp_len = 100000

    # If the producer is returning a Midas event, you can pass `as_event=True`
    # and we'll automatically parse the payload (so `resp` will be a midas.event.Event
    # rather than raw binary data).
    resp = client.brpc_client_call(conn, "some_cmd", json.dumps(jargs), max_len=resp_len)

    # If you don't know the full structure/length of the response in advance, you can look
    # at `resp_len` to see the number of bytes returned.
    cast_resp = struct.unpack("<IIII", resp)

    print(f"The response was: {cast_resp[0]}, {cast_resp[1]}, {cast_resp[2]}, {cast_resp[3]}")

    client.disconnect_from_other_client(conn)

let jargs = {"int_list": [1,2,3,4]};

let params = Object();
params.client_name = "some_producer";
params.cmd = "some_cmd";
params.args = JSON.stringify(jargs);
params.max_reply_length = 100000;

mjsonrpc_call("brpc", params, "arraybuffer").then(function(rpc) {
  // If all your data is one type, you can cast the response to a typed ArrayBuffer
  let cast_resp = new Int32Array(rpc);

  console.log("The response was: " + cast_resp[0] + ", " + cast_resp[1] + ", " + cast_resp[2] + ", " + cast_resp[3]);
}).catch(function(error) {
  mjsonrpc_error_alert(error);
});