History: programming

As a reminder, midas provides multiple "drivers", allowing you to write data in multiple formats:

• the legacy midas driver writes .hst binary files
• the newer FILE driver writes .dat binary files
• the MYSQL, PGSQL, SQLITE and ODBC drivers write to an SQL database

Details on event names and tag names

The below table shows how event and tag names vary based on the data source, whether the mlogger driver uses per-variable or per-equipment logging, and whether the equipment defines names in the ODB settings. The table uses:

• EqName as a placeholder for the equipment name
• VarName as a placeholder for the variable name
• DirName as a placeholder for the subdirectory name if variables are in subdirectories (e.g. Variables/DirName/VarName instead of just Variables/VarName)
• DefName as a placeholder for the relevant entry in Settings/Names or Settings/Names xxx.

The table also shows whether ODB entries that are arrays get exposed as arrays in the history system, or get split up. File-based drivers only split arrays when the array entries have been given names. SQL drivers always split arrays into separate columns, calling the tags VarName[0], VarName[1] etc.

Restrictions on sub-directories

There are several restrictions to putting variables in sub-directories (as Variables/DirName/VarName instead of just Variables/VarName):

• You cannot use a Names array.
• You can only go one level deep (Variables/DirName/AnotherDir/VarName is not allowed).
• Subdirectories are not supported for virtual events.
• You may experience issues if using per-equipment logging and your subdirectory contains variables with different types (e.g. a mix of doubles and integers). It is recommended to use per-variable logging in this case. Remember that all the SQL drivers use per-variable logging automatically, but it is optional when writing to binary files.

Edge cases

• For virtual events, the name of the link itself is used (not the target of the link).
• When SQL databases are being used, some command-line/code interfaces may show two copies of event names and tag names. The "real" event/tag names are always available, but you may also see copies where the name has been manipulated to be all lower case, with spaces and other punctuation replaced by underscores (e.g. Trigger rate may be available as both Trigger rate and trigger_rate).
• You must ensure that all history definitions have a unique event name + tag name combination. The two main edge cases are:
  • If using per-equipment logging AND you use the Names xxx format, you cannot have the same name appear in any of that equipment's names arrays (e.g. Names Voltage and Names Current cannot both contain Channel 1 - but you could use Channel 1 (V) and Channel 1 (A) instead). This doesn't affect things when you use the common Names array, as mlogger automatically uses both the variable name and the defined name.
  • You cannot use the same name for a virtual event and an equipment.

Code interfaces

Getting the list of history events

These examples print the names of all the history events. Use an event name to get its list of tags, or to read its data.

#include "history.h"

HNDLE hDB;
cm_get_experiment_database(&hDB, nullptr);

MidasHistoryInterface* mh = nullptr;
int status = hs_get_history(hDB, 0, HS_GET_DEFAULT | HS_GET_READER | HS_GET_INACTIVE, 0, &mh); //(1)!
if (status != HS_SUCCESS)
   return status;

std::vector<std::string> events;
mh->hs_get_events(0, &events);

for (const auto& ev : events)
   printf("%s\n", ev.c_str());

events = client.hist_get_events() #(1)!

for ev in events:
    print(f"{ev}")

mjsonrpc_call('hs_get_events', {}).then(function(rpc) {
   for (const ev of rpc.result.events) {
      console.log(ev);
   }
}).catch(function(error) {
   mjsonrpc_error_alert(error);
});

Getting the list of history tags

These examples print the names of the tags (variables) in one history event, given in event_name. See Details on event names and tag names for how the names are made from the ODB.

std::vector<TAG> tags;
mh->hs_get_tags(event_name, 0, &tags);

for (const auto& tag : tags)
   printf("%s\n", tag.name); //(1)!

tags = client.hist_get_tags(event_name)

for tag in tags:
    print(f"{tag['name']}") #(1)!

mjsonrpc_call('hs_get_tags', {events: [event_name]}).then(function(rpc) {
   for (const tag of rpc.result.events[0].tags) {
      console.log(tag.name); //(1)!
   }
}).catch(function(error) {
   mjsonrpc_error_alert(error);
});

Getting the history data

These examples read the values of one variable, given by event_name and tag_name, from the last three hours, and print each value with its timestamp. If the variable is an array, the C++ and JavaScript examples read its first element, and the Python example reads all its elements.

time_t end = time(nullptr);
time_t start = end - 3 * 3600;
time_t interval_secs = 10;

const char* event_names[] = {event_name}; //(1)!
const char* tag_names[] = {tag_name};
int indices[] = {0};
int num_entries[1] = {0};
time_t* timestamps[1] = {nullptr};
double* values[1] = {nullptr};
int hs_status[1] = {0};

mh->hs_read(start, end, interval_secs, 1, event_names, tag_names, indices,
            num_entries, timestamps, values, hs_status); //(2)!

printf("%s/%s[%d] has %d entries in this time period\n", event_name, tag_name, indices[0], num_entries[0]);

for (int i = 0; i < num_entries[0]; i++)
   printf("%ld: %g\n", (long)timestamps[0][i], values[0][i]);

free(timestamps[0]); //(3)!
free(values[0]);

end = datetime.datetime.now()
start = end - datetime.timedelta(hours=3)
interval_secs = 10
data = client.hist_get_data(start, end, interval_secs, event_name, tag_name) #(1)!

for elem in data: #(2)!
    print(f"{elem['event_name']}/{elem['tag_name']}[{elem['index']}] has {elem['num_entries']} entries in this time period")

    for i in range(len(elem['timestamps'])):
        print(f"{elem['timestamps'][i]}: {elem['values'][i]}") #(3)!

const end = Date.now() / 1000;
const start = end - 3 * 3600;

mjsonrpc_call('hs_read', {
   start_time: start, end_time: end,
   events: [event_name], tags: [tag_name], index: [0] //(1)!
}).then(function(rpc) {
   const data = rpc.result.data[0];
   console.log(`${event_name}/${tag_name}[0] has ${data.count} entries in this time period`);

   for (let i = 0; i < data.count; i++) {
      console.log(`${new Date(data.time[i] * 1000)}: ${data.value[i]}`);
   }
}).catch(function(error) {
   mjsonrpc_error_alert(error);
});

Defining a new history panel

These examples create a history panel called My plot name in the Default group, showing one variable with the label Temperature. Each variable is given as <event>:<tag>, with [index] for an element of an array.

std::vector<std::string> vars = {"Lakeshore:L336[0]"};
std::vector<std::string> labels = {"Temperature"};
hs_define_panel2("Default", "My plot name", vars, labels); //(1)!

vars = ["Lakeshore:L336[0]"]
labels = ["Temperature"]
client.hist_create_plot("Default", "My plot name", vars, labels) #(1)!

const panel = '/History/Display/Default/My plot name';

mjsonrpc_db_create([{path: panel, type: TID_KEY}]).then(function(rpc) {
   if (rpc.result.status[0] !== 1) { //(1)!
      throw new Error('History plot already exists');
   }
   return mjsonrpc_db_paste([panel], [{
      'Variables': ['Lakeshore:L336[0]'],
      'Label': ['Temperature'],
      'Colour': ['#00AAFF'],
      'Formula': [''],
      'Timescale': '1h',
      'Show values': true,
      'Show fill': true,
      'Show run markers': true
   }]);
}).catch(function(error) {
   mjsonrpc_error_alert(error);
});

Command-line tools

See History: overview for details on command-line tools that can analyse history files. mhdump works even if you do not have access to the main midas machine (and only have .hst/.dat binary files).

File formats and database schema

See History file formats and database schema for the formats of the history files, the SQL database schema, and how to read them offline.