Remotely executed functions in C with a single extra source and header file.

The purpose of this experiment is to minimize the libraries and dependencies needed to execute remote procedure calls (RPC) from one C program to another.

To setup a client call or a server function, a program only needs to compile and link dstc.c, include dstc.h, and add a single macro. A program can be both a DSTC client and server.

DSTC uses reliable_multicast to ensure that data is delivered robustly. See https://github.com/PDXostc/reliable_multicast for details

The dstc.c and dstc.h files currently weigh in at ~1000 lines of code (according to cloc). The sample multi-user chat server is 36 lines of code.

The underlying reliable multicast can transmit 25 million signals / second between a publisher and subscriber running on two Dell servers with 10Gb Ethernet between them, using a single thread.

You just need gcc and reliable multicast to build and deploy your services. Any Posix-compliant OS is a suitable target environment.

All scalars, arrays, unions and structs can be transmitted, as long as they do not contain pointers.

If a server function is registered in multiple processes / nodes across a network, all of them will be invoked in parallel with a (single) client call to the given function.
The provided chat system is implemented in ~50 lines of C code.

A client call to a server can include a pointer to a client-side function that can be invoked by the server code.

This allows a service to use event-driven programming to replace synchronous RPC calls with return values that don't risk blocking threads and resources across the network as load increases.

Reliable multicast will retransmit any dropped packets via a sideband TCP channel, combining TCP-level robustness with the scalability of UDP.

Since the purpose is to provide bare-bones RPC mechanisms with a minimum of dependencies, there are several limitations, listed below

All functions that are to be remotely executed must have a return type of void. See callbacks above for an event-driven solution.

UDP/IP packets have a maximum of 64K. Meaning that your function call arguments, taking overhead data into consideration, should stay under 63K.

Arguments are currently copied across the network in their native format using memcpy() without respect to endianness or padding. This means that arguments will only be transferred correctly between a client and server using the same endianness, which is little-endian on x86.

See gcc __attribute__ ((packed)) and __attribute__ ((endianness(big))) for how this can easily be achieved in a mixed-architecture deployment.

DSTC uses reliable_multicast (RMC)as its transport layer. Download, build and install RMC from:

Reliable Multicast v1.3

Update Makefile in this DSTC directory to point to the include and library directories of the installed RMC code.

Build DSTC using

make
sudo make DESTDIR=/usr/local install

Build examples using

make examples
sudo make DESTDIR=/usr/local install_examples

To fix the following error

error while loading shared libraries: libdstc.so: cannot open shared object file: No such file or directory

Run this command

sudo /sbin/ldconfig -v

The following environment variables are recognized and used by DSTC:

• DSTC_NODE_ID [int]
  Sets the DSTC node ID. Each ID has to be unique across all DSTC instances running in a network.
  Default is 0, which will assign a random number as the node ID.

• DSTC_MAX_NODES [int]
  Maximum number of DSTC nodes that we will see on the network. Each node will require 128KB of ram. If more than the given number of nodes are active on a network, traffic will be lost.
  Default is 32.

• DSTC_MCAST_GROUP_ADDR [string]
  Specifies the UDP Multicast group address to use in AAA.BBB.CCC.DDD format. All DSTC nodes that share the same multicast group and port will see each other on the network. In heavy traffic scenarios DSTC services should be organized across multiple groups in order to minimize traffic to be processed by each node.
  Default is 239.40.41.42.

• DSTC_MCAST_GROUP_PORT [int]
  UDP Port to be used for multicast traffic in the given group address.
  Default is 4723.

• DSTC_MCAST_IFACE_ADDR [string]
  Specify the IP address of the network interface to exclusively use for multicast traffic.
  Default is 0.0.0.0, indicating that all available interfaces are used.

• DSTC_MCAST_TTL [int]
  Specify the IP time to live for multicast packets, indicating how many router hops they should traverse before being dropped.
  Default is 1.

• DSTC_CONTROL_LISTEN_IFACE [string]
  Specify the IP address of the network interface to exclusively use for listening to incoming control channel traffic from other DSTC nodes.
  Default is 0.0.0.0, indicating that all available interfaces are to be listened on.

• DSTC_CONTROL_LISTEN_PORT [int]
  TCP Port to be used by the control channel listener.
  Default is 0, indicating that an ephereal TCP port is to be assigned by the operating system.

• DSTC_LOG_LEVEL [int]
  Specifies the log level on stdout. The following values are available:
  0 - No logging
  1 - Fatal errors
  2 - Errors
  3 - Warnings
  4 - Information
  5 - Comments
  6 - Debug
  Default is 2 - Errors.

The client program invokes a C function on the server that prints the name and age provided as arguments by the client.

term_1$ ./examples/print_name_and_age/print_name_and_age_server

term_2$ ./examples/print_name_and_age/print_name_and_age_client

Exit the server with ctrl-c.

The chat example allows multiple users to exchange messages between each other. This demonstrates:

• How a single client call can trigger multiple server-side function calls

• How the DSTC multicast socket can be integrated into a (e)poll() vector.

• How a program can simultaneously act as a client and a server.

term_1$ ./examples/chat/chat

term_2$ ./examples/chat/chat

term_3$ ./examples/chat/chat

Enter user name in all terminals, followed by chat.

Exit with ctrl-c.

In this example we will show how you can export a simple function, print_name_and_age(), to be callable from a remote client.

The server program to be executed by the remote client is written as you would any C function:

void print_name_and_age(char* name, int age)
{
    printf("Name: %s\n", name);
    printf("Age: %d\n", age);
}

The function cannot return any value and must be of void return type.

In order to export the code, you add a macro at the beginning of the same file, or any source file included in the library build:

DSTC_SERVER(print_name_and_age, char, [32], int,)

The arguments to the macro are as follows:

• print_name_and_age
  This is the name of the function to export. A wrapper function will be created that will receive the call from the remote client, decode the incoming data, and invoke the server function locally.

• char, 32
  This indicates that the first parameter (name) should be encoded, transmitted, and decoded as a 32 byte char array. In this case the generated server-side decoder function will extract 32 bytes of data and provide a pointer to that data as the name argument to the local function call of print_name_and_age().

• int,
  This indicates that the second argument (age) should be encoded, transmitted, and decoded as an integer. The empty field after the extra comma (,) specifies that this argument is a scalar and not an array. The generated server-side decoder function will extract sizeof(int) (4) bytes of data from the buffer received from the remote client, convert it to an integer, and provide that integer as the age argument to the local print_name_and_age() function call.

In order for a client to execute a remote function, it needs a local function to call to encode and transmit the data to the remote server that will execute the function. This local, client-side function is generated by a macro:

DSTC_CLIENT(print_name_and_age, char, [32], int,)

The macro parameters, (print_name_and_age, char, [32], int,) must be identical to those provided to DSTC_SERVER on the server side.

The macro will expand to the following client-side function

void dstc_print_name_and_age(char[32], int);

This function can be called by a client who wants to remotely execute the server-side dstc_print_name_and_age().

Compile and link dstc.c with your code.

Both DSTC_CLIENT and DSTC_SERVER can accept basic C data type arguments (except pointers) like structs and fix-size arrays.

The DECL_DYNAMIC_ARG macro can be used in DSTC_CLIENT and DSTC_SERVER to specify that the given argument has dynamic length.

Below is an example from examples/dynamic_data/dynamic_data_client.c where the dynamic_message() function accepts a dynamic length argument and an array of four integers.

DSTC_CLIENT(dynamic_message, DECL_DYNAMIC_ARG, int, [4])

The client-side call to dynamic_message is as follows:

char *first_arg = "This string can be variable length";
int second_arg[4] = { 1,2,3,4 };

// Use the DYNAMIC_ARG() macro to specify that we want to provide a dynamic
// length string (that includes the terminating null char):

dstc_dynamic_message(DYNAMIC_ARG(first_arg, strlen(first_arg) + 1), second_arg);

The first argument to DYNAMIC_ARG is expected to be void*. The second argument is expected to be uint32_t.

The server-side declaration of dynamic arguments is identical to the client side. From examples/dynamic_data/dynamic_data_client.c:

DSTC_SERVER(dynamic_message, DECL_DYNAMIC_ARG, int, [4])

An example of the actual function to be called is given below:

void dynamic_message(dstc_dynamic_data_t dynarg, int second_arg[4])
{
    printf("Data:          %s\n", (char*) dynarg.data);
    printf("Length:        %d\n", dynarg.length);
    printf("Second Arg[0]: %d\n", second_arg[0]);
    printf("Second Arg[1]: %d\n", second_arg[1]);
    printf("Second Arg[2]: %d\n", second_arg[2]);
    printf("Second Arg[3]: %d\n", second_arg[3]);
}

The dstc_dynamic_data_t struct is defined in dstc.h as:

typedef struct {
  uint32_t length;
  void* data;
} dstc_dynamic_data_t;

When dynamic_message() is called, it can check dynarg.length for the number of bytes available in the memory pointed to by dynarg.data.

The memory referred to by the dstc_dynamic_data_t struct is owned by the DSTC system and should not be modified or freed. Once the called function returns, the memory pointed to by the data element will be deleted.

A DSTC_CLIENT-declared call can accept a function pointer argument to be forwarded by the call to the remote server. The receiving server function, declared via DSTC_SERVER will receive a corresponding function pointer to invoke in order to make a callback to the client.

This allows the server to deliver execution results to the client in lieu of return values. The callback can only be invoked once and will only be received by the sending client.

If multiple servers execute a call and invoke their callbacks, only one of those callbacks will be forwarded to the client-side code. The rest of the callbacks are silently dropped. It is undefined which of the server callbacks will be executed.

Below is an example from examples/callback/callback_client.c where a call is made to the remote double_value() in order to double the provided value and send back the result through a callback.

DSTC_CLIENT(double_value, int,, DECL_CALLBACK_ARG);

The double_value() function accepts the value to double and a callback function pointer.

The call to the function on the client side looks like below.

dstc_double_value(42, CLIENT_CALLBACK_ARG(double_value_callback,int,));

The 42 argument is the value to double.

The CLIENT_CALLBACK_ARG(double_value_callback,int,) specifies that a pointer to double_value_callback() function should be sent to the remote server, and that this callback function takes a single integer (the doubled value) as its sole argument.

The callback function implementation is a regular C function that prints out the doubled value it receives from the remote server's callback invocation:

void double_value_callback(int value)
{
    printf("Callback received: %d\n", value);
}

The server-side declaration of a callback argument to a function is the same as the client side. Below is code from examples/callback/callback_server.c:

DSTC_SERVER(double_value, int,, DECL_CALLBACK_ARG)

The implementation is as follows:

void double_value(int value, dstc_callback_t callback_ref)
{
    DSTC_CALLBACK(callback_ref, int,);

    printf("double_value(%d) called with a callback\n", value);
    dstc_callback_ref(value + value);
}

The dstc_callback_t callback_ref argument declares a DSTC-specific variable that hosts all information necessary to make a remote callback to the calling process.

The DSTC_CALLBACK(callback_ref, int,); function sets up the necessary code to define a local callback function dstc_callback_ref() in this case, that will forward the callback to the client process.

Finally, the callback itself is invoked through a regular C call to dstc_callback_ref().

Please note that "callback_ref" specifies the name of both the argument and the generated local callback function.

RPC encoding is done by the code generated by the DSTC_CLIENT macro. The encoding (for now) is done by simply copying out the bytes from the argument to a data buffer to be transmitted.

The code generated by DSTC_SERVER will decode the incoming data.

To see what the generated code looks like, build the examples using "make nomacro". The nomacro files will contain the expanded macros at the end of the file.

Note: The "nomacro" feature requires the "clang-format" package. This can be installed on Ubuntu systems with:

sudo apt install -y clang-format