SDxmlrpc

Introduction

SDxmlrpc is an XML-RPC server for StructuredData. Most of the functions and commands SDpyshell offers, can also be used with SDxmlrpc.

What is XML-RPC ?

This is a way to perform remote procedure calls. When you call a remote procedure, your system transfers the name of the procedure and it’s parameters to program on your local host or a remote server. The server then executes the procedure and sends the results back to your system. In order to make these calls easy to use the implementation of remote procedure calls makes them look very much like ordinary (local) procedure calls.

In XML-RPC the parameters and results of a procedure are encoded in strings by using XML. XML-RPC also uses the well known HTTP (Hypertext Transfer Protocol) for communication between client and server.

There are XML-RPC libraries for many programming languages, including Python, Perl, Ruby, C, C++ and Java.

What is the difference between SDxmlrpc and SDpyshell ?

SDpyshell can be used as an interactive shell and as an interpreter for scripts. It is intended to be used by humans. SDxmlrpc on the other hand is intended to be used by programs. These programs may run on the same host or on a different host than the SDxmlrpc server and can use it’s functions much like a library.

Basic function

SDxmlrpc implements most of the functions of SDpyshell for XML-RPC. Since there are only small differences to the functions of SDpyshell, this documentation often refers to the documentation there and mainly emphasizes the differences to SDpyshell.

The StructuredDataContainer key

Almost all commands and functions of the SDxmlrpc server work with a StructuredDataContainer that is provided as parameter. In SDpyshell, if this parameter is omitted, the commands use a global variable that is a StructuredDataContainer. In SDxmlrpc, the StructuredDataContainer must always be specified as parameter, it must not be omitted. As a further difference to SDpyshell, the StructuredDataContainer is only referenced by a key which is a string. This is needed since XML-RPC is independent of a programming language, so we cannot transfer python references.

In order to isolate applications that use the SDxmlrpc server at the same time, the keys to the StructuredDataContainer are generated by a random process. A key can only be generated by calling function newsdc. It creates a new StructuredDataContainer and returns it’s key which is a string.

In order to preserve memory on the SDxmlrpc server, a StructuredDataContainer gets deleted if it was not read or written to for a certain time.

Return values

The SDxmlrpc server only gives access to functions of the functional and the text layer (see also Command layers). Functions of the functional layer return data structures where functions of the text layer always return a string which may be a multi-line string.

Functions

See Commands and Functions for a detailed description. Note that only commands whose names start with “fun.” and “txt.” are accessible via XML RPC.

Examples

Here are some very simple examples on how to use XML-RPC with various programming languages. All examples require a running StructuredData XML-RPC server. In order to start the XML-RPC server go to the directory “samples” then issue this command:

SDxmlrpc --precmdfile Xprecmd.txt --port 8000

The file Xprecmd.txt loads the file idcp_db.cache.SDCyml under the name “idcp_db”. It has this content:

# create a named StructuredDataContainer:
fun.namedsdc("idcp_db")
# read idcp_db StructuredData file:
fun.read("idcp_db.cache.SDCyml","","idcp_db")

The XML-RPC clients try to connect on port 8000 to the hostname of your host. Note that this is not identical to “localhost”. All examples determine the name of your system before trying to connect. In all the examples, a query is performed which is equivalent to the following SDpyshell command:

find("id-data.*.names.devicename")

The output of the programs is always this:

id-data.U125/1.names.devicename : U125IV
id-data.U125/2.names.devicename : U125ID2R
id-data.U139.names.devicename   : U139ID6R
id-data.U2.names.devicename     : U2IV
id-data.U3.names.devicename     : U3IV
id-data.U4.names.devicename     : U4IV
id-data.U41.names.devicename    : U41IT6R
id-data.U48.names.devicename    : U48IV
id-data.U49/1.names.devicename  : U49ID4R
id-data.U49/2.names.devicename  : U49ID8R
id-data.UE112.names.devicename  : UE112ID7R
id-data.UE46.names.devicename   : UE46IT5R
id-data.UE49.names.devicename   : UE49IT4R
id-data.UE52.names.devicename   : UE52ID5R
id-data.UE56/1.names.devicename : UE56ID6R
id-data.UE56/2.names.devicename : UE56ID3R
id-data.UE56R.names.devicename  : UE56IV
id-data.Ubonsai.names.devicename: U1IV

Python

This is the example query in python (file “xmlrpc-python.py” in directory “samples”):

#!/usr/bin/env python

# How to start the server:
# cd samples
# SDxmlrpc --precmdfile Xprecmd.txt --port 8000

import socket
import xmlrpclib

host= socket.gethostname()

s = xmlrpclib.ServerProxy('http://%s:8000' % host)

result= s.txt.find("id-data.*.names.devicename","","","idcp_db")

print result

Perl

This is the example in perl (file “xmlrpc-perl.py” in directory “samples”). You need to have XML::RPC installed, it can be found here http://search.cpan.org/~daan/XML-RPC-0.9/lib/XML/RPC.pm:

#!/usr/bin/perl

# This program needs XML::RPC:
# http://search.cpan.org/~daan/XML-RPC-0.9/lib/XML/RPC.pm

# How to start the server:
# cd samples
# SDxmlrpc --precmdfile Xprecmd.txt --port 8000

use strict;

use Sys::Hostname;
use XML::RPC;

my $host= hostname;

my $xmlrpc = XML::RPC->new("http://$host:8000/RPC2");

my $result= $xmlrpc->call("txt.find","id-data.*.names.devicename","","","idcp_db");

print "$result\n";

C

This is the example in C (file “xmlrpc-c.c” in directory “samples”). You need to have XML-RPC installed, which can be downloaded here http://xmlrpc-c.sourceforge.net. As mentioned in the comment you can create the binary from the c-source with this command:

gcc -Wall xmlrpc-c.c -lxmlrpc_client -o xmlrpc-c

Here is the program:

/*

This program needs XML-RPC:
http://xmlrpc-c.sourceforge.net

How to start the server:
cd samples
SDxmlrpc --precmdfile Xprecmd.txt --port 8000

compile with this command:
gcc -Wall xmlrpc-c.c -lxmlrpc_client -o xmlrpc-c
*/

#include <stdlib.h>
#include <string.h>
#include <stdio.h>

#include <unistd.h>  /* only needed for my_hostname */
#include <netdb.h>   /* only needed for my_hostname */

#include <xmlrpc-c/base.h>
#include <xmlrpc-c/client.h>

#define NAME "Xmlrpc-c Test Client"
#define VERSION "1.0"

#define PORT 8000

static char *_get_hostname(void)
  {
    struct hostent* h;
    char hostname[1024];
    hostname[1023] = '\0';
    gethostname(hostname, 1023);
    h = gethostbyname(hostname);
    return strdup(h->h_name);
  }

static void dieIfFaultOccurred (xmlrpc_env * const envP)
  {
    if (envP->fault_occurred)
      {
        fprintf(stderr, "ERROR: %s (%d)\n",
            envP->fault_string, envP->fault_code);
        exit(1);
      }
  }

void print_unpack_string(xmlrpc_env *envP, xmlrpc_value *resultP)
  /* prints the string
   * DECREASES the reference counter of resultP
   */
  {
    const char *ptr;
    xmlrpc_read_string(envP, resultP, &ptr);
    dieIfFaultOccurred(envP);
    puts(ptr);
    xmlrpc_DECREF(resultP);
  }

int main(int const argc, const char ** const argv)
  {
    char serverUrl[256];
    xmlrpc_env env;
    xmlrpc_value * resultP;

    sprintf(serverUrl, "http://%s:%d/RPC2", _get_hostname(), PORT);
    xmlrpc_env_init(&env);
    xmlrpc_client_init2(&env, XMLRPC_CLIENT_NO_FLAGS, NAME, VERSION, NULL, 0);
    dieIfFaultOccurred(&env);

    resultP= xmlrpc_client_call(&env, serverUrl, "txt.find", "(ssss)",
                                "id-data.*.names.devicename","","","idcp_db");
    dieIfFaultOccurred(&env);
    print_unpack_string(&env, resultP);

    xmlrpc_env_clean(&env);
    xmlrpc_client_cleanup();
    return 0;
}

C++

This is the example in C++ (file “xmlrpc-cpp.cpp” in directory “samples”). You need to have XML-RPC installed, which can be downloaded here http://xmlrpc-c.sourceforge.net. As mentioned in the comment you can create the binary from the c-source with this command:

g++ -Wall --std=c++0x xmlrpc-cpp.cpp -lxmlrpc_client++ -o xmlrpc-cpp

Here is the program:

/*

This program needs XML-RPC:
http://xmlrpc-c.sourceforge.net

How to start the server:
cd samples
SDxmlrpc --precmdfile Xprecmd.txt --port 8000

compile with this command:
g++ -Wall --std=c++0x xmlrpc-cpp.cpp -lxmlrpc_client++ -o xmlrpc-cpp
*/

#include <string.h>

#include <unistd.h>  /* only needed for my_hostname */
#include <netdb.h>   /* only needed for my_hostname */

#include <iostream>
#include <string>

#include <xmlrpc-c/base.hpp>
#include <xmlrpc-c/client_simple.hpp>

#define PORT 8000

static char *_get_hostname(void)
  {
    struct hostent* h;
    char hostname[1024];
    hostname[1023] = '\0';
    gethostname(hostname, 1023);
    h = gethostbyname(hostname);
    return strdup(h->h_name);
  }

static std::string url(std::string host, int port)
  {
    std::string st= std::string("http://");

    st.append(host);
    st.append(":");
    st.append(std::to_string(port));
    st.append("/RPC2");
    return st;
  }

int main(int const argc, const char ** const argv)
  {

    std::string const serverUrl= url(std::string(_get_hostname()), PORT);
    xmlrpc_c::clientSimple myClient;
    xmlrpc_c::value *result_p;

    result_p= new xmlrpc_c::value();
    myClient.call(serverUrl, "txt.find", "ssss", result_p,
                  "id-data.*.names.devicename", "", "", "idcp_db");

    std::cout << std::string(xmlrpc_c::value_string(*result_p)) << "\n";
    delete result_p;

    return 0;
}

Invoking SDxmlrpc

Here is a short overview on the SDxmlrpc command line options:

--version

show program’s version number and exit

-h, --help

show this help message and exit

--summary

print a summary of the function of the program

--info

Show ip, port and process ID on stderr.

--localhost

start server on ‘localhost’ instead of DNSDOMAINNAME. In this case the server can only be contacted from applications running on the same host.

--host

start server on HOST instead of DNSDOMAINNAME. This may be needed for hosts with more than one network interface.

--port=PORT

start xmlserver on port PORT

-p COMMANDS, --precmd=COMMANDS

specify COMMANDS to perform before any other action

--precmdfile=FILE

specify a FILE to execute before any other action

-M, --module

specify a MODULE to import at make its functions accessible by XMLRPC

-I, --searchpath

specify a DIRECTORY to prepend it to the module search path.

--no-locking

do not lock file accesses

--pidfile=PIDFILE

specify the PIDFILE where PID’s of sub processes will be stored

--kill

just kill old servers, do not start new ones.

--restart

restart the already running server

Precommands

Precommands are commands that are executed at the start of the SDxmlrpc server before any other command. These commands can be given as a command line parameter (–precmd) or they can be read from a file (–precmdfile). A typical application is to put the command to read a StructuredData file in a file and provide it’s name with –precmdfile.

Extensions

These are user supplied python modules that can be loaded by the SDxmlrpc. The module name (the filename without “.py”) is provided with the command line option “-M”. In this case the python module is loaded and it’s functions are accessible with the module name as a prefix.

You can use command line option “-I” in order to extend the search path for extensions which are basically python modules. Keep in mind that extensions are also searched in all paths specified by the “PYTHONPATH” environment variable.

Here is an example:

We have a file “myXext.py” with this content:

import StructuredData.SDshelllibTxt as txt
import StructuredData.SDshelllibFun as fun

def ids():
    p= fun.paths("id-data.*",sdc="idcp_db")
    return fun.poppath(p, no=-1)

def show_ids(formatspec="yaml"):
          return txt.format(ids(), formatspec)

We also have a file “Xprecmd.txt” with this content:

# create a named StructuredDataContainer:
fun.namedsdc("idcp_db")
# read idcp_db StructuredData file:
fun.read("idcp_db.cache.SDCyml","","idcp_db")

Now we start SDxmlrpc with “-M” to load the extension and with “–precmdfile” to load the sample StructuredData file from the “samples” directory:

SDxmlrpc -M myext --precmdfile Xprecmd.txt --localhost --port 8000

To test the server we use the interactive python shell. We start python by entering “python” on the command line:

$ python
Python 2.7.3 (default, Jul 24 2012, 10:05:39)
[GCC 4.7.0 20120507 (Red Hat 4.7.0-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import xmlrpclib
>>> import pprint
>>> s = xmlrpclib.ServerProxy('http://localhost:8000')
>>> pprint.pprint(s.myXext.ids())
['U125/1',
 'U125/2',
 'U139',
 'U2',
 'U3',
 'U4',
 'U41',
 'U48',
 'U49/1',
 'U49/2',
 'UE112',
 'UE46',
 'UE49',
 'UE52',
 'UE56/1',
 'UE56/2',
 'UE56R',
 'Ubonsai']

>>> print s.myXext.show_ids("yaml")
- U125/1
- U125/2
- U139
- U2
- U3
- U4
- U41
- U48
- U49/1
- U49/2
- UE112
- UE46
- UE49
- UE52
- UE56/1
- UE56/2
- UE56R
- Ubonsai

Process management

When SDxmlrpc is started it is useful to know it’s process id (PID) to be able to restart the server by killing the old one and starting a new one. This is done with the options –pidfile and –kill. –pidfile is used to specify the name of a PID file, this file contains a line with the process id (PID) of the server and the command that was used to start the server. When –pidfile is provided, the process named in this file (and it’s children) are killed first. When SDxmlrpc is started, it’s PID and command line are put to the PID file. If you dont’t want to restart an SDxmlrpc server but just want to kill the old one, use –pidfile together with –kill.

Restarting the server

The server can forced do a complete restart. In this case it reloads the precommand file and recreates all internal variables. This may be useful when the StructuredData file on disk was changed and the server needs to reload the file.

The server can either be restared by sending it signal SIGUSR1 like in:

kill SIGUSR1 PID

where PID is the process ID, an integer, of the SDxmlrpc server, or by invoking:

SDxmlrpc --pidfile PIDFILE --restart

when PIDFILE is the process ID file that was created when the server was started.