Alarms: configuring
This page covers how to define alarms and notifications.
For the concepts and day-to-day use, see Alarms: overview.
Configuring alarm classes
For all the keys, see /Alarms/Classes.
Alarm classes are defined in the ODB in /Alarms/Classes. A class says what happens when one of its alarms is triggered. It can write a message to the message log, write an entry in the elog, run a command and stop the run. It also sets the colors of the alarm banner on the Status page, and whether web pages play a sound.
Note about intervals
The System message interval and Execute interval settings are useful to reduce the number of messages written or the number of times that the command gets executed. But they have a surprising effect as they apply to all alarms of the same class!
For example, say that the interval is set to 60s, and a message about alarm X has just been written. If alarm Y (of the same class) gets triggered within 60s, no message will be written! Also, if multiple alarms of the same class are triggered at the same time, only the first one will be recorded.
You can set System message interval to 0 to remove any throttling of messages. Execute interval is different: the command is only executed if Execute interval is greater than 0, so set it to a small value (e.g. 1) for no throttling.
The default values are for a System message interval of 60s, and an Execute interval of 0. So with the default settings, Execute command is never executed.
Creating a new alarm class
To create a new alarm class, it is easiest to copy an existing one, either using the web interface ( / icons) or the odbedit command-line program:
odbedit -c 'copy "/Alarms/Classes/Warning" "/Alarms/Classes/My new class"'
The All class
If you create the special alarm class /Alarms/Classes/All, it will apply to all alarms that are raised, in addition to the class specified for that alarm. This is useful if you want to send messages about all alarms to Slack, for example, without having to set the "send to Slack" script as the Execute command in all your alarm classes.
Configuring alarms
For all the keys, including the ones that midas writes to record when an alarm was checked and triggered, see /Alarms/Alarms.
Alarms are defined in the ODB in /Alarms/Alarms, with one subdirectory per alarm. How to create new alarms is explained for each alarm type later.
Each alarm has a Type, an Alarm Class (which must already exist) and an Alarm Message that is shown in the banner and written to the message log. Evaluated alarms also have a Condition. To avoid triggering on transient problems, set Trigger count required so that the condition must be true for several checks in a row.
Alarm types
1 - Internal alarms
Internal alarms are raised directly by code. The code specifies the alarm name, the alarm class, and the message. If the alarm does not already exist, an entry is created automatically in /Alarms/Alarms. An error message will be printed if the alarm class does not exist in /Alarms/Classes.
2 - Program not running
Alarms can be raised if an important midas program is not running. This can be applied to both built-in programs, and user-written programs that use midas.
The evaluation of whether a program is running or not happens at least every 10s. However the raising of an alarm (or auto-restart) can be delayed by setting /Programs/xxx/Check interval, which is in ms.
Creating a new program alarm
To create a new "Program not running" alarm, set the ODB key /Programs/xxx/Alarm class to a valid alarm class. The corresponding entry in /Alarms/Alarms will be created for you automatically the first time that the alarm is triggered. The program alarm will not be visible on the alarms webpage until the alarm has triggered.
3 - Evaluated alarms
Evaluated alarms compare an ODB value to a threshold, and raise an alarm if the condition is met.
Comparison syntax
- Valid comparators are:
=,==,!=,<,<=,>,>=,&. - You can check if any part of an ODB array matches a condition using wildcards
[*]or array slices[3-5,9-11]. - Booleans can be compared to
Y,yor1. - String comparisons are not supported (only numerical/boolean).
- You can use special functions
access()andaccess_running()to determine how many seconds have elapsed since an ODB key was written to (useful for high-level checks that a program is both running and doing what it should).access_running()is only evaluated if a run is in progress.
Example comparisons
Simple check on a single ODB entry
/Runinfo/Run number > 100
Check if any element of the array is > 100
/Equipment/HV/Variables/Input[*] > 100
Check if parts of an array match. For "dash" notation, both ends are inclusive (i.e. [4-6] matches 4,5,6).
/Equipment/HV/Variables/Input[2-3] > 100
/Equipment/HV/Variables/Input[1,4,5-8,10] > 100
Check if a single bit is set
/Equipment/Environment/Variables/MyBitmask & 8
Check a boolean against Y, y or 1
/Some/Boolean != y
/Some/Boolean == 1
Check if it's been more than 10s since an ODB key was last written to
access(/Some/path) > 10
Creating a new evaluated alarm
To create a new Evaluated alarm, it is easiest to copy an existing one, either using the web interface ( / icons) or the odbedit command-line program:
odbedit -c 'copy "/Alarms/Alarms/Demo ODB" "/Alarms/Alarms/My new alarm"'
If you have many alarms to create, you could use code to create alarms instead. Evaluated alarms have a Type of 3.
4 - Periodic alarms
A periodic alarm is activated periodically according to the time in /Alarms/Alarms/xxx/Check interval.
Creating a new periodic alarm
You can create a periodic alarm the same way as for an evaluated alarm, but set the Type to 4. The Condition does not need to be set.
Example scripts that can be called
Alarm classes can be configured to call any arbitrary script using /Alarms/Classes/xxx/Execute command. The placeholder %s can be used, and will be replaced with the alarm message.
Some example uses of scripts are shown below.
Send an email
The simplest way to send a message is using the mail command, but note that messages like these may get caught in spam filters.
You could create a script send_email.sh that contains the following:
#/bin/bash
echo $1 | mail -s "Alarm was raised" your.name@domain.edu
odbedit -c 'msg midas INFO my_alarm "Alarm was sent to your.name@domain.edu"'
Ensure the script is executable (chmod +x send_email.sh), and set /Alarms/Classes/xxx/Execute command to /path/to/my_script.sh.
The second line of this script just adds a message to the midas message log. Your script could of course be more complicated, and read email addresses from a file or the ODB etc.
Send a Slack message
A more modern way for notification is to use messaging apps such as Slack. To send alarms to Slack, do the following:
- Go to https://api.slack.com/apps
- Create an App "midas alarms", select "From scratch"
- Name it "midas alarm" or similar, select your workspace
- Click on "Add features and functionality"
- Select "Incoming Webhooks"
- Activate Incoming Webhooks
- Click on "Add New Webhook to Workspace"
- Select channel where alarms get posted, allow access
- Copy the sample curl request and replace "Hello World" with "$1", such as:
(leave the keys xxx, yyy, zzz as shown on the web page).
curl -X POST -H 'Content-type: application/json' --data "{"text":"$1"}' https://hooks.slack.com/services/[xxx]/[yyy]/[zzz]
- Create a shell that contains the curl command from step 1.
- Ensure the shell script is executable (e.g.,
chmod +x script.sh) - Call the shell script from the alarm system by putting
path/to/script.sh '%s'under/Alarms/Classes/xxx/Execute commandin the ODB.
Note that you can also forward midas messages to Slack, as well as sending alarm notifications.
Send a Telegram notification
A telegram bot needs to be created to post midas alarms into telegram channels.
- Go to https://web.telegram.org and login by scanning the QR-code with the telegram app on your phone
- Go to https://t.me/botfather
- Click on "open in web"
- A chat window will open.
- Send
/newbotinto the chat and assign name and username to your bot - At the end of the bot creation a access token will be displayed. Save that access token.
- Post
/mybotsinto the chat - Select the bot that was just created
- Click on "Bot settings"
- Click on "Allow Groups"
- Enable groups for the bot
- Click on "back to settings"
- Click on Channel Admin Rights
- Click on "Manage channel" (a checkmark should appear next to the text)
- Close this chat.
- Add the created bot to a telegram group chat via searching for the bot name in "add members" (same as adding new People to the group)
- Now you need the chat_id of the same group chat. When you open the group chat in a webbrowser the URL should look like this
https://web.telegram.org/a/#-1700000000(where 1700000000 will be a different number for you) - The group chat_id is then
-1001700000000(add-100in front of the number in your URL)
- With the access token from step 1 and chat_id from step 2, you can post to the telegram group like this: (replace
<chat_id>and<accessToken>with your numbers)curl -s -X POST "https://api.telegram.org/bot<accessToken>/sendMessage" -d chat_id=<chat_id> -d text="🚨 Midas alarm triggered with message = $1" - Create a shell script with this command
- Ensure the shell script is executable (e.g.,
chmod +x script.sh) - Call the shell script from the alarm system by putting
path/to/script.sh '%s'under/Alarms/Classes/xxx/Execute commandin the ODB.
Note that you can also forward midas messages to Telegram, as well as sending alarm notifications.
Send a Mattermost notification
First a mattermost bot needs to be created. To do this you need to be an Admin of the Team that the bot should post to.
- In the drop down menu on the top left corner of the mattermost page click on "Integrations" (note: this field will be missing if you are not an Admin of the selected mattermost team)
- Select "Incoming Webhooks"
- Click on "Add Incoming Webhook"
- Give the bot a title and username (e.g. midasalarms) and select the channel to post to, click "save"
- A URL for the incoming webhook should be displayed (https://mattermost.gitlab.rlp.net/hooks/
). Copy that URL.
- Copy the webhook URL into the curl command in the following script:
#!/bin/bash alarm_message=$1 generate_post_data() { cat <<EOF { "text": "@all Midas alarm triggered with message = $alarm_message", "icon_emoji":":rotating_light:", "username":"Midas" } EOF } curl -i -X POST -H 'Content-Type: application/jsoni' --data "$(generate_post_data)" https://mattermost.gitlab.rlp.net/hooks/<token> - Ensure the shell script is executable (e.g.,
chmod +x script.sh) - Call the shell script from the alarm system by putting
path/to/script.sh '%s'under/Alarms/Classes/xxx/Execute commandin the ODB.
Note that you can also forward midas messages to Mattermost, as well as sending alarm notifications.
Call a phone using bird / MessageBird
We will use a text-to-voice to read out midas alarms in a phone call to shifters. The service is called bird (formerly messagebird). Note that bird hosts two platforms: "engagement" and "connectivity". Each has its own API, documentation, and web interface. We will use the engagement platform. Be sure to look at the correct documentation.
- Make an account: https://app.bird.com/
- Add some funds to your main wallet: https://app.bird.com/settings/billing/overview/wallets/
- Buy a number. This number will be the where the call originates from, so you should pick something local to your area.
- In the left-hand navigation side panel: "Developer" → "Numbers" → "Numbers & Senders"
- Top-right: "Browse Inventory"
- Make sure your number is capable of voice calling (it seems like most of them are)
- Follow the on-screen prompts to finish your purchase and activate the number. You will have to create a UseCase. Either "Mixed" or "Security Alert" should do.
- Ensure compliance with local regulations
- Go back to the number summary: "Developer" → "Numbers" → "Numbers & Senders". You should now have your number listed. Click on it.
- At the top there is a menu, select "Voice"
- You should see some links to set up brand identification and compliance (what will you be using the number for, the company associated with the number, etc). Follow the on-screen instructions. Once compliance has been requested it may take a few days to be approved.
- While you wait, you also need to create a Voice Channel. In this same menu, click on "+ Create channel". You can see your channels from the side bar navigation: "Developer" → "Voice" → "Voice Setup".
You need a few pieces of info:
- API Access Key
- In the side bar navigation: "Developer" → "API Access Keys" (at the bottom)
- Make a new key. Keep it somewhere safe.
- Workspace ID
- In the side bar navigation: "Settings" (at the bottom) → "Workspaces" → Click on your workspace
- Channel ID
- In the side bar navigation: "Developer" → "Voice" → "Voice Setup" → Click on your voice channel → Settings (top right gear)
- Call ID
- Only available once the call is placed. See the script below.
- Outgoing number
- In the side bar navigation: "Developer" → "Numbers" → "Numbers & Senders"
Below is an example python script to place a call.
- Tweak it using the information you gathered.
- Call the script from the alarm system by putting
python path/to/script.py '%s'under/Alarms/Classes/xxx/Execute commandin the ODB.
You can follow the documentation here: https://docs.bird.com/api/voice-api/voice-calls-api.
import json, requests, time, sys
# settings
workspaceId = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
channelId = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
APIKEY = 'YourAPIKey'
outgoing = '16042221047' # outgoing number with country code
targetnumber = 'YourPurcahsedPhoneNumber' # who to call
message = "Midas alarm - " + sys.argv[1] # message
# place a call
# https://docs.bird.com/api/voice-api/voice-calls-api/initiate-an-outbound-call
def place_call(number):
# place the call
response = requests.post(
f"https://api.bird.com/workspaces/{workspaceId}/channels/{channelId}/calls",
headers={"Authorization":f"AccessKey {APIKEY}","Content-Type":"application/json"},
data=json.dumps({
"from": outgoing,
"to": number,
"ringTimeout": 60,
"maxDuration": 120,
"record": False,
})
)
# call id is needed for the next steps
data = response.json()
callId = data['id']
return callId
# get the call status
# https://docs.bird.com/api/voice-api/voice-calls-api/get-a-call-insights
def get_call_status(callId):
response = requests.get(
f"https://api.bird.com/workspaces/{workspaceId}/channels/{channelId}/calls/{callId}/insights",
headers={"Authorization":f"AccessKey {APIKEY}","Accept":"*/*"},
)
data = response.json()
return data['status']
# text-to-voice: play message in the call
# https://docs.bird.com/api/voice-api/voice-calls-api/say-text-to-speech-tts
def play_message(callId, message):
response = requests.post(
f"https://api.bird.com/workspaces/{workspaceId}/channels/{channelId}/calls/{callId}/say",
headers={"Authorization":f"AccessKey {APIKEY}","Content-Type":"application/json"},
data=json.dumps({
"text": message,
"locale": "en-US",
"voice": "text",
"loop": 10, # repeat the message this many times
"timeout": 30
})
)
return response.json()
# RUN =========================
# place call
callId = place_call(targetnumber)
# wait until call is answered, or until 30s has passed
t0 = time.time()
while time.time() - t0 < 30:
status = get_call_status(callId)
print(f'Call status: "{status}"')
# call was answered
if status == 'ongoing':
break
# check again later
time.sleep(1)
# play the message
time.sleep(0.5)
play_message(callId, message+',') # comma after message introduces a pause between loops
Note that you can also forward midas messages to bird, as well as sending alarm notifications.