Exploring the inter-process messaging bus D-Bus using Elisp
Published on Apr 16, 2023.
As a user, you typically expect your software application to seamlessly interact with other processes and bits of the local system. For example, you might want it to immediately react to new USB devices being connected or mute itself on an incoming (video) call. One pretty neat way of accomplishing this is via the inter-process messaging bus D-Bus1.
D-Bus has been around since almost two decades and is likely already installed on your system if you are running a Linux-based OS. Many of the applications and services running on your computer will be providing various methods and signals via/ D-Bus that can be used to interact with them. A method can be called via a D-Bus message and will be answered by the application with a response. Signals, on the other hand, can be subscribed to, and will be triggered on given events such as the before-mentioned connection of a new USB device. A nice feature of D-Bus is that it provides introspection: you can explore any registered application on the bus and discover its methods and signals from within D-Bus.
In this post, I will give an example on how one can use dbus.el, the Elisp
D-Bus bindings that Emacs ships with, to interact with other local processes.
The application I want control from within Emacs is the distributed, private messenger
GNU Jami.
My goal is to create a chat-bot that can trigger actions on commands received
via text-messages and handle received files. Specifically, I want to use Org
mode’s org-capture mechanism to store notes, references and action items into
my todo lists. This post, however, will only explore Jami’s D-Bus interface to
create an account, add a contact and send and react to messages. If you are not
interested in the low-level mechanics or D-Bus itself, then feel free to skip
head to the jami-bot or read about how to use the bot to capture notes in Org
mode! Otherwise, let’s dive into D-Bus – after making sure we have our tools ready!
Prerequisites
Besides Emacs2 and a system running D-Bus, we will need to have the Jami daemon,
jamid, installed on the local system. On Debian, this can be done by simply
running sudo apt install jami which will also install the GUI application. The
latter is not strictly necessary but can be more comfortable to use during some
of the steps. The version installed through apt, however, is likely older than
what is provided on the official download pages – consider updating should you
run into any connectivity issues later.
Another useful – but optional – tool is the graphical D-Bus debugger, D-Feet. If you feel that exploring long lists of D-Bus nodes via Elisp function calls is a little daunting (or tedious), this tool will allow you to click through to the information you want very quickly.
Getting started with the D-Bus API in Emacs
Emacs comes with its own D-Bus API which is documented quite well and includes
several examples. This makes it easy to interact with any services on D-Bus
through a simple Elisp function call. But first, the dbus package needs to be
loaded:
(require 'dbus)
Now, we can already find out what services are available via D-Bus. Generally,
there are two separate busses to consider: the system bus and the session
bus. Anything started by the user would typically reside on the latter while the
former has services such as network, power management and the like. As Jami is a
service started by the user, it is found on the session bus. dbus-list-names
can list anything registered there:
(dbus-list-names :session )
I needed to shorten the list considerably despite my system running a rather
lightweight desktop environment only. So you might see a considerably longer
list. You find things like my terminal program (org.xfce.Terminal5) or the
currently running Firefox profile (org.mozilla.firefox.ZGVmYXVsdC1lc3I_). What
I am looking for is cx.ring.Ring which is Jami – the messenger was called
“GNU Ring” until not so long ago and the API has not been updated yet.
Alternatively, you can also check whether or not a service is registered by pinging it:
(dbus-ping :session "cx.ring.Ring")
On my system, this actually has the side effect of starting the Jami daemon
(jamid) even if it was not running before. The daemon is what is running in
the background and handling all interactions with the various network
connections Jami establishes to allow messages, calls and video chats. So it is
essential for using Jami and even the GUI builds upon it. If this command
returns nil, check your Jami installation and that jamid is running.
Let us take a closer look into the Jami D-Bus interface. It is structured into a number of interfaces that correspond to different aspects of the messenger:
(dbus-introspect-get-all-nodes :session "cx.ring.Ring" "/cx/ring/Ring")
The different names indicate what each node does, e.g. handling calls, video or
plugins. This feature of being able to list parts of the API and even – as we
will see later – find the arguments expected by methods is called
introspection and a nice feature of D-Bus. It is similar to how Emacs allows
you to explore its functions and variables from within the running software. So
if you, for example, wonder how to use the function
dbus-introspect-get-all-nodes, you can from within Emacs press C-h f to open
the help for functions, write dbus-introspect-get-all-nodes (or use
tab-complete) and press enter to see the signature, a short explanation and the
source code to the function. The former is (dbus-introspect-get-all-nodes BUS
SERVICE PATH). The session was already explained above. The service consists of
a name separated by dots while the path denotes an object on the D-Bus separated
by slashes like a path on a filesystem. For the examples below, the service is
always cx.ring.Ring.
The anatomy of Jami’s D-Bus interface
Despite the ability for introspection in D-Bus, it will likely be necessary to look into the API as defined in the Jami source code. You can find these in the jami-daemon/bin/dbus/cx.ring.Ring.ConfigurationManager.xml file in the Jami source repository. This document includes helpful documentation for many of the nodes, i.e. methods and signals. There is only so much one can guess from the node’s name alone. I have unfortunately not found a way to retrieve these additional docstrings from within D-Bus – so having the above file open as a reference is useful for the following exercise.
Of the above mentioned interfaces, only the ConfigurationManager is needed in
the beginning. While it would be cool to automatically answer calls and e..g
record and transcribe them, I have not yet had the time to dig into the
CallManager sufficiently. But the methods and signals of the
ConfigurationManager will already allow us to create an account, add a contact
and receive and send messages.
ConfigurationManager
The interface of the ConfigurationManager is rather extensive. It consists of
methods and signals. The former are callable and often yield a response
while the latter are something that one subscribes to in order to get notified
of e.g. state changes, received messages and the like. To get a complete list of
the interface, one can use the following introspection function:
(dbus-introspect-get-interface :session "cx.ring.Ring" "/cx/ring/Ring/ConfigurationManager" "cx.ring.Ring.ConfigurationManager")
The returned list will be very long but we will only need a fraction of these commands in the following.
Jami account management via D-Bus
The ConfigurationManager exposes all methods required to setup and manage Jami
accounts. If you want, you could also start up the GUI application and set up an
account there instead – whether you use the GUI or D-Bus, accounts will be
visible from either client, even simultaneously3.
First, let us list the already existing accounts (if any) by calling the method
getAccountList on the ConfigurationManager interface:
(dbus-call-method :session "cx.ring.Ring" "/cx/ring/Ring/ConfigurationManager" "cx.ring.Ring.ConfigurationManager" "getAccountList")
The function dbus-call-method we will use quite extensively in the following
as it is one of the main ways to interact with an application over D-Bus. The
above method getAccountList does not take any arguments; otherwise, they would
be provided as additional arguments to dbus-call-mehod. As I do not yet have
any Jami accounts set up, the returned value is empty.
As with most messengers, an account in Jami has a ton of configuration parameters that can be modified. So let us have a look at the template that is being used to fill these settings before we actually create a new account. There are two types of account: “SIP” or “RING”. We want to use the latter:
(dbus-call-method :session "cx.ring.Ring" "/cx/ring/Ring/ConfigurationManager" "cx.ring.Ring.ConfigurationManager" "getAccountTemplate" "RING")
The method for creating a new account is called addAccount (of course). We can use introspection to find out what arguments it takes:
(dbus-introspect-get-method :session "cx.ring.Ring" "/cx/ring/Ring/ConfigurationManager" "cx.ring.Ring.ConfigurationManager" "addAccount" )
From this, we see that it takes and array details of type string string as
input and returns the ID of the created account. What exactly these details
are would be a bit mysterious though without the clues we found in the account
template above. After looking through the documentation for the mapping between
D-Bus types and Elisp we can construct a proper call to the method and
configure key settings of the account in one go:
(dbus-call-method :session "cx.ring.Ring" "/cx/ring/Ring/ConfigurationManager" "cx.ring.Ring.ConfigurationManager" "addAccount" '(:array (:dict-entry :string "Account.type" :string "RING") (:dict-entry :string "Account.alias" "jami-bot") (:dict-entry :string "Account.displayName" "jami-bot")))
The alias/display name are not the same as the user name: the latter is registered and unique on the Jami name server while the former can be changed at any time and are only shown to one’s contacts. The ID returned by the method call is a shortened ID that is used by the Jami daemon to identify the account internally. On the Jami network, the account is referred to a by a longer hash (the address, always present) or by a name which can be registered and maps to the address. I hope the examples below will make this a little clearer.
Let’s first see that the account is now present by checking the list of accounts:
(dbus-call-method :session "cx.ring.Ring" "/cx/ring/Ring/ConfigurationManager" "cx.ring.Ring.ConfigurationManager" "getAccountList")
Great! The account settings can be retrieved using a call to the method
getAccountDetails and the account ID as argument. In the row
Account.username you should see the full address that identifies the user on
the network:
(dbus-call-method :session "cx.ring.Ring" "/cx/ring/Ring/ConfigurationManager" "cx.ring.Ring.ConfigurationManager" "getAccountDetails" "d362747388cf970f")
To change any of these settings, simply call setAccountDetails. According to
the documentation, this should work with an incomplete list of details and only
affect those that are explicitly set. However, it my experiments, almost all
details missing in the argument were simply wiped – including, but not limited
to, Account.enabled which controls whether or not the account will be visible
on the network. This might be a bug in Jami, an artifact of the D-Bus API in
Emacs or caused by the way I call the method. In any case, better set all the
details at once as with the lengthy call below:
(dbus-call-method :session "cx.ring.Ring" "/cx/ring/Ring/ConfigurationManager" "cx.ring.Ring.ConfigurationManager" "setAccountDetails" "d362747388cf970f" '(:array (:dict-entry :string "Account.accountDiscovery" :string "false") (:dict-entry :string "Account.accountPublish" :string "false") (:dict-entry :string "Account.activeCallLimit" :string "-1") (:dict-entry :string "Account.alias" :string "jami-bot") (:dict-entry :string "Account.allModeratorEnabled" :string "true") (:dict-entry :string "Account.allowCertFromContact" :string "true") (:dict-entry :string "Account.allowCertFromHistory" :string "true") (:dict-entry :string "Account.allowCertFromTrusted" :string "true") (:dict-entry :string "Account.archiveHasPassword" :string "false") (:dict-entry :string "Account.audioPortMax" :string "32766") (:dict-entry :string "Account.audioPortMin" :string "16384") (:dict-entry :string "Account.autoAnswer" :string "false") (:dict-entry :string "Account.defaultModerators" :string "") (:dict-entry :string "Account.deviceName" :string "reform") (:dict-entry :string "Account.dhtProxyListUrl" :string "https://config.jami.net/proxyList") (:dict-entry :string "Account.displayName" :string "jami-bot") (:dict-entry :string "Account.dtmfType" :string "overrtp") (:dict-entry :string "Account.enable" :string "true") (:dict-entry :string "Account.hostname" :string "bootstrap.jami.net") (:dict-entry :string "Account.localInterface" :string "default") (:dict-entry :string "Account.localModeratorsEnabled" :string "true") (:dict-entry :string "Account.mailbox" :string "") (:dict-entry :string "Account.managerUri" :string "") (:dict-entry :string "Account.managerUsername" :string "") (:dict-entry :string "Account.peerDiscovery" :string "false") (:dict-entry :string "Account.presenceSubscribeSupported" :string "true") (:dict-entry :string "Account.proxyEnabled" :string "false") (:dict-entry :string "Account.proxyServer" :string "dhtproxy.jami.net:[80-95]") (:dict-entry :string "Account.publishedAddress" :string "") (:dict-entry :string "Account.publishedSameAsLocal" :string "true") (:dict-entry :string "Account.rendezVous" :string "false") (:dict-entry :string "Account.ringtoneEnabled" :string "false") (:dict-entry :string "Account.ringtonePath" :string "default.opus") (:dict-entry :string "Account.sendReadReceipt" :string "true") (:dict-entry :string "Account.type" :string "RING") (:dict-entry :string "Account.upnpEnabled" :string "true") (:dict-entry :string "Account.useragent" :string "") (:dict-entry :string "Account.videoEnabled" :string "false") (:dict-entry :string "Account.videoPortMax" :string "65534") (:dict-entry :string "Account.videoPortMin" :string "49152") (:dict-entry :string "DHT.PublicInCalls" :string "false") (:dict-entry :string "DHT.port" :string "0") (:dict-entry :string "RingNS.uri" :string "") (:dict-entry :string "TLS.certificateFile" :string "/home/hanno/.local/share/jami/d362747388cf970f/ring_device.crt") (:dict-entry :string "TLS.certificateListFile" :string "") (:dict-entry :string "TLS.password" :string "") (:dict-entry :string "TLS.privateKeyFile" :string "/home/hanno/.local/share/jami/d362747388cf970f/ring_device.key") (:dict-entry :string "TURN.enable" :string "true") (:dict-entry :string "TURN.password" :string "ring") (:dict-entry :string "TURN.realm" :string "ring") (:dict-entry :string "TURN.server" :string "turn.jami.net") (:dict-entry :string "TURN.username" :string "ring")))
Establishing contact
Now let us put the newly created account to use and start a chat with another
Jami user. For that, we will need to know the full address such as, for example,
badac18e13ec1a6e1266600e457859afebfb9c46. As this is error prone to type in
and tedious to spell out, the GUI application allows you to scan a QR code from
your friend’s phone instead. Alternatively, you can register a name on the
network that is easier to remember but must be unique. Let us assume that the
person we would like to contact has done so and search for the user name on the
network.
Searching for profiles and signal handling
The name lookup is one of the features of Jami that is handled in two steps:
first, we trigger the lookup by a call to the corresponding method,
lookupName. Then, we have to wait until Jami signals that the name is indeed
registered and has been found. This is done via the signal registeredNameFound
which we have to subscribe to by registering a function.
What does this function have to look like? That depends on the signal in question. Via introspection, we can find out what the expected function arguments are:
(dbus-introspect-get-signal :session "cx.ring.Ring" "/cx/ring/Ring/ConfigurationManager" "cx.ring.Ring.ConfigurationManager" "registeredNameFound")
For our purposes, a simple function like this will do for now:
(defun my-registeredNameFound-handler (account status address name) (message "jami received profile for account: %s, status: %s, address: %s, name: %s" account status address name))
It will simply dump its arguments into the *Messages* buffer. Let us register the function for the signal registeredNameFound:
(dbus-register-signal :session "cx.ring.Ring" "/cx/ring/Ring/ConfigurationManager" "cx.ring.Ring.ConfigurationManager" "registeredNameFound" #'my-registeredNameFound-handler)
The function dbus-register-signal returns an object that can be used via dbus-unregister-object to remove the registration again.
Now we can trigger the name lookup for my-best-friend4:
(dbus-call-method :session "cx.ring.Ring" "/cx/ring/Ring/ConfigurationManager" "cx.ring.Ring.ConfigurationManager" "lookupName" "" "ns.jami.net" "my-best-friend")
The lookup is done on Jami’s default name server ns.jami.net, but you could put your own here, if so desired.
Shortly after, you should see something like this in your *Messages* buffer:
jami received profile for account: , status: 0, address: b8e0350a62caf0173d18e0d1256a8cf2ea88e75b, name: my-best-friend
The status flag is explained in the docstring to the corresponding signal in the cx.ring.Ring.ConfigurationManager.xml file:
| SUCCESS | 0 | everything went fine. Name/address pair was found. |
| INVALIDNAME | 1 | provided name is not valid. |
| NOTFOUND | 2 | everything went fine. Name/address pair was not found. |
| ERROR | 3 | An error happened |
Adding a contact
If you have either found a contact by looking up the name or know a contact’s ID string, you can now add that contact to our account:
(dbus-call-method :session "cx.ring.Ring" "/cx/ring/Ring/ConfigurationManager" "cx.ring.Ring.ConfigurationManager" "addContact" "d362747388cf970f" "b8e0350a62caf0173d18e0d1256a8cf2ea88e75b")
This will send a contact request to the person in question and – if accepted –
initiate a conversation. Alternatively, you could also start a new conversation
(startConversation) and add members to it (addConversationMember), if you
would like to use a group chat instead.
Conversations and sending messages
For any account, you can list the available conversations:
(dbus-call-method :session "cx.ring.Ring" "/cx/ring/Ring/ConfigurationManager" "cx.ring.Ring.ConfigurationManager" "getConversations" "d362747388cf970f")
To see who is a member of the conversation, use getConversationMembers and provide account and conversation as arguments:
(dbus-call-method :session "cx.ring.Ring" "/cx/ring/Ring/ConfigurationManager" "cx.ring.Ring.ConfigurationManager" "getConversationMembers" "d362747388cf970f" "70bcc0ce73be4091b5e159dc22d6cc5bb4e1a252")
Knowing its ID, we can send a message to a conversation via sendMessage:
(dbus-call-method :session "cx.ring.Ring" "/cx/ring/Ring/ConfigurationManager" "cx.ring.Ring.ConfigurationManager" "sendMessage" "d362747388cf970f" "70bcc0ce73be4091b5e159dc22d6cc5bb4e1a252" "this is the first line of the message\nThis is the second line" "" :int32 0)
The last two arguments are the commitId and the flag. The former is used to
refer to an existing message (a commit since git is used behind the scenes
to manage conversation data). In the above example, we are sending a single, new
message, so the commitId is empty and the flag set to 0. If a commitId would
have been given, then this would have been sent as a reply to that message
instead. If the flag is set to ’1’ then this would have been an edit to an
existing message.
You can find out the id of a previous message by searching for it. I have,
however, not looked much into that feature. Alternatively, one can wait for a
signal that a new message arrived (messageReceived) and retrieve both message
and its id from a registered handler. As a more trivial example, we can dump the meta information and contents of the message to the minibuffer and *Messages* buffer:
(defun my-messageReceived-handler (account conversation message) (message "jami received msg: account: %s, conversation: %s, msg: %s" account conversation message)) (dbus-register-signal :session "cx.ring.Ring" "/cx/ring/Ring/ConfigurationManager" "cx.ring.Ring.ConfigurationManager" "messageReceived" #'my-messageReceived-handler)
Some example messages:
jami received msg: account: d362747388cf970f, conversation: 70bcc0ce73be4091b5e159dc22d6cc5bb4e1a252, msg: ((author b8e0350a62caf0173d18e0d1256a8cf2ea88e75b) (body Test!) (id f23a3db43597ad1bdd7e8027cd3e6ca2f4ad7820) (linearizedParent d899f9926e8bedc907b85e6f60bf0f03672c5881) (parents d899f9926e8bedc907b85e6f60bf0f03672c5881) (timestamp 1678032406) (type text/plain))
jami received msg: account: d362747388cf970f, conversation: 70bcc0ce73be4091b5e159dc22d6cc5bb4e1a252, msg: ((author b8e0350a62caf0173d18e0d1256a8cf2ea88e75b) (displayName IMG20230304115628.jpg) (fileId 15af639ad4cab741e15717cb867cb613f0e8c4ff3740082762842406.jpg) (id 15af639ad4cab741e15717cb867cb613f0e8c4ff) (linearizedParent eab0a2c8e2f1ca5f7de8991375dafb8962178cae) (parents eab0a2c8e2f1ca5f7de8991375dafb8962178cae) (sha3sum 457c749572575f6827b2ae0860d065911adc959a0d57d4ccc47ab26cc93e7c07a01122e28344ee7745aca8ea85216ba6c6aba8cf42aaaebd68ff33fa6ad66929) (tid 3740082762842406) (timestamp 1678033015) (totalSize 2971303) (type application/data-transfer+json))
In the first example, we have received a simple text (’Test!’). The message contains, amongst others, fields for the author, message id, and timestamp. In the second example, the message is a file transfer with additional fields such as the id and fileId. Knowing these, we can download the file to the local machine using downloadFile:
(dbus-introspect-get-method :session "cx.ring.Ring" "/cx/ring/Ring/ConfigurationManager" "cx.ring.Ring.ConfigurationManager" "downloadFile")
Following the example above, this could be:
(dbus-call-method :session "cx.ring.Ring" "/cx/ring/Ring/ConfigurationManager" "cx.ring.Ring.ConfigurationManager" "downloadFile" "d362747388cf970f" "70bcc0ce73be4091b5e159dc22d6cc5bb4e1a252" "15af639ad4cab741e15717cb867cb613f0e8c4ff" "15af639ad4cab741e15717cb867cb613f0e8c4ff_3740082762842406.jpg" "/home/hanno/tmp/jami.jpg")
Timestamps are given in seconds and can be converted into calendrical information using decode-time:
(decode-time 1678032925)
(25 15 17 5 3 2023 0 nil 3600)
The values are (SECONDS MINUTES HOUR DAY MONTH YEAR DOW DST UTCOFF), respectively.
Defining a helper routine for accessing Jami methods via D-Bus
To save some keystrokes, we can define a helper routine with all the constant parts:
(defun jami-dbus-call-cfgmgr-method (method &rest args) "Call jami D-Bus METHOD on the cfgmgr interface with arguments ARGS." (apply #'dbus-call-method `(:session "cx.ring.Ring" "/cx/ring/Ring/ConfigurationManager" "cx.ring.Ring.ConfigurationManager" ,method ,@(when args args))))
That means that sending a message would become:
(jami-dbus-call-cfgmgr-method "sendMessage" "d362747388cf970f" "70bcc0ce73be4091b5e159dc22d6cc5bb4e1a252" "this is the first line of the message\nThis is the second line" "" :int32 0)
Or, with yet another helper function, we can get this even shorter:
(defun jami-send-message (account conversation text &optional reply) "Add TEXT to CONVERSATION via ACCOUNT. REPLY optionally specifies a message id." (jami-dbus-call-cfgmgr-method "sendMessage" account conversation text `(,@(if reply reply "")) :int32 0))
Which would allow to send a message with a call to:
(jami-send-message "d362747388cf970f" "70bcc0ce73be4091b5e159dc22d6cc5bb4e1a252" "this is the first line of the message\nThis is the second line")
Final thoughts
D-Bus is a powerful tool to interact with services running on the system. Its introspection mechanism provides a lot of the necessary information to use this API – however, additional documentation is almost a must, especially when dealing with more complex arguments to methods. Accessing services on D-Bus from Emacs is easy enough, though converting different data types took me some trial and error.
Jami is unusual as it exposes most if not all important functions via D-Bus. In fact, in Jami’s case, D-Bus is the best-supported API to write clients with. Despite the sparse documentation outside the API definition, the API is simple enough and Jami can be scripted with only a few lines of code to do text messaging from within Emacs!
While this post does not really drive that last point home, the next one in this series will demonstrate a Jami chat bot written in Elisp!
Tags: emacs, programming, jami, lisp, dbus
Footnotes:
Not sure what the correct capitalization of D-Bus is, there seem to be a variation of conventions around. I will stick with D-Bus over DBUS or dbus
Emacs needs to be compiled with D-Bus support – which is the default.
If you ever encounter
issues where GUI and daemon do not seem to be in sync, try to close the GUI and
kill the daemon (killall jamid) before restarting Jami.
With appologies to the person that might actually register this name at some point.