SDxmlrpc ======== Introduction ------------ SDxmlrpc is an `XML-RPC `_ server for StructuredData. Most of the functions and commands SDpyshell offers, can also be used with SDxmlrpc. What is XML-RPC ? +++++++++++++++++ This is a way to perform `remote procedure calls `_. When you call a remote procedure, your system transfers the name of the procedure and it's parameters to program on your local host or a remote server. The server then executes the procedure and sends the results back to your system. In order to make these calls easy to use the implementation of remote procedure calls makes them look very much like ordinary (local) procedure calls. In XML-RPC the parameters and results of a procedure are encoded in strings by using `XML `_. XML-RPC also uses the well known `HTTP (Hypertext Transfer Protocol) `_ for communication between client and server. There are XML-RPC libraries for many programming languages, including Python, Perl, Ruby, C, C++ and Java. What is the difference between SDxmlrpc and SDpyshell ? +++++++++++++++++++++++++++++++++++++++++++++++++++++++ SDpyshell can be used as an interactive shell and as an interpreter for scripts. It is intended to be used by humans. SDxmlrpc on the other hand is intended to be used by *programs*. These programs may run on the same host or on a different host than the SDxmlrpc server and can use it's functions much like a library. Basic function -------------- SDxmlrpc implements most of the functions of SDpyshell for XML-RPC. Since there are only small differences to the functions of SDpyshell, this documentation often refers to the documentation there and mainly emphasizes the differences to SDpyshell. The StructuredDataContainer key +++++++++++++++++++++++++++++++ Almost all commands and functions of the SDxmlrpc server work with a :ref:`StructuredDataContainer ` that is provided as parameter. In SDpyshell, if this parameter is omitted, the commands use a global variable that is a StructuredDataContainer. In SDxmlrpc, the StructuredDataContainer must *always* be specified as parameter, it must not be omitted. As a further difference to SDpyshell, the StructuredDataContainer is only *referenced* by a *key* which is a string. This is needed since XML-RPC is independent of a programming language, so we cannot transfer python references. In order to isolate applications that use the SDxmlrpc server at the same time, the *keys* to the StructuredDataContainer are generated by a random process. A *key* can only be generated by calling function newsdc. It creates a new StructuredDataContainer and returns it's *key* which is a string. In order to preserve memory on the SDxmlrpc server, a StructuredDataContainer gets deleted if it was not read or written to for a certain time. Return values +++++++++++++ The SDxmlrpc server only gives access to functions of the functional and the text layer (see also :ref:`Command layers `). Functions of the functional layer return data structures where functions of the text layer always return a string which may be a multi-line string. Functions --------- See :ref:`Commands and Functions ` for a detailed description. Note that only commands whose names start with "fun." and "txt." are accessible via XML RPC. Examples -------- Here are some very simple examples on how to use XML-RPC with various programming languages. All examples require a running StructuredData XML-RPC server. In order to start the XML-RPC server go to the directory "samples" then issue this command:: SDxmlrpc --precmdfile Xprecmd.txt --port 8000 The file Xprecmd.txt loads the file idcp_db.cache.SDCyml under the name "idcp_db". It has this content:: # create a named StructuredDataContainer: fun.namedsdc("idcp_db") # read idcp_db StructuredData file: fun.read("idcp_db.cache.SDCyml","","idcp_db") The XML-RPC clients try to connect on port 8000 to the hostname of your host. Note that this is *not identical* to "localhost". All examples determine the name of your system before trying to connect. In all the examples, a query is performed which is equivalent to the following SDpyshell command:: find("id-data.*.names.devicename") The output of the programs is always this:: id-data.U125/1.names.devicename : U125IV id-data.U125/2.names.devicename : U125ID2R id-data.U139.names.devicename : U139ID6R id-data.U2.names.devicename : U2IV id-data.U3.names.devicename : U3IV id-data.U4.names.devicename : U4IV id-data.U41.names.devicename : U41IT6R id-data.U48.names.devicename : U48IV id-data.U49/1.names.devicename : U49ID4R id-data.U49/2.names.devicename : U49ID8R id-data.UE112.names.devicename : UE112ID7R id-data.UE46.names.devicename : UE46IT5R id-data.UE49.names.devicename : UE49IT4R id-data.UE52.names.devicename : UE52ID5R id-data.UE56/1.names.devicename : UE56ID6R id-data.UE56/2.names.devicename : UE56ID3R id-data.UE56R.names.devicename : UE56IV id-data.Ubonsai.names.devicename: U1IV Python ++++++ This is the example query in python (file "xmlrpc-python.py" in directory "samples"):: #!/usr/bin/env python # How to start the server: # cd samples # SDxmlrpc --precmdfile Xprecmd.txt --port 8000 import socket import xmlrpclib host= socket.gethostname() s = xmlrpclib.ServerProxy('http://%s:8000' % host) result= s.txt.find("id-data.*.names.devicename","","","idcp_db") print result Perl ++++ This is the example in perl (file "xmlrpc-perl.py" in directory "samples"). You need to have XML::RPC installed, it can be found here http://search.cpan.org/~daan/XML-RPC-0.9/lib/XML/RPC.pm:: #!/usr/bin/perl # This program needs XML::RPC: # http://search.cpan.org/~daan/XML-RPC-0.9/lib/XML/RPC.pm # How to start the server: # cd samples # SDxmlrpc --precmdfile Xprecmd.txt --port 8000 use strict; use Sys::Hostname; use XML::RPC; my $host= hostname; my $xmlrpc = XML::RPC->new("http://$host:8000/RPC2"); my $result= $xmlrpc->call("txt.find","id-data.*.names.devicename","","","idcp_db"); print "$result\n"; C +++ This is the example in C (file "xmlrpc-c.c" in directory "samples"). You need to have XML-RPC installed, which can be downloaded here http://xmlrpc-c.sourceforge.net. As mentioned in the comment you can create the binary from the c-source with this command:: gcc -Wall xmlrpc-c.c -lxmlrpc_client -o xmlrpc-c Here is the program:: /* This program needs XML-RPC: http://xmlrpc-c.sourceforge.net How to start the server: cd samples SDxmlrpc --precmdfile Xprecmd.txt --port 8000 compile with this command: gcc -Wall xmlrpc-c.c -lxmlrpc_client -o xmlrpc-c */ #include #include #include #include /* only needed for my_hostname */ #include /* only needed for my_hostname */ #include #include #define NAME "Xmlrpc-c Test Client" #define VERSION "1.0" #define PORT 8000 static char *_get_hostname(void) { struct hostent* h; char hostname[1024]; hostname[1023] = '\0'; gethostname(hostname, 1023); h = gethostbyname(hostname); return strdup(h->h_name); } static void dieIfFaultOccurred (xmlrpc_env * const envP) { if (envP->fault_occurred) { fprintf(stderr, "ERROR: %s (%d)\n", envP->fault_string, envP->fault_code); exit(1); } } void print_unpack_string(xmlrpc_env *envP, xmlrpc_value *resultP) /* prints the string * DECREASES the reference counter of resultP */ { const char *ptr; xmlrpc_read_string(envP, resultP, &ptr); dieIfFaultOccurred(envP); puts(ptr); xmlrpc_DECREF(resultP); } int main(int const argc, const char ** const argv) { char serverUrl[256]; xmlrpc_env env; xmlrpc_value * resultP; sprintf(serverUrl, "http://%s:%d/RPC2", _get_hostname(), PORT); xmlrpc_env_init(&env); xmlrpc_client_init2(&env, XMLRPC_CLIENT_NO_FLAGS, NAME, VERSION, NULL, 0); dieIfFaultOccurred(&env); resultP= xmlrpc_client_call(&env, serverUrl, "txt.find", "(ssss)", "id-data.*.names.devicename","","","idcp_db"); dieIfFaultOccurred(&env); print_unpack_string(&env, resultP); xmlrpc_env_clean(&env); xmlrpc_client_cleanup(); return 0; } C++ +++ This is the example in C++ (file "xmlrpc-cpp.cpp" in directory "samples"). You need to have XML-RPC installed, which can be downloaded here http://xmlrpc-c.sourceforge.net. As mentioned in the comment you can create the binary from the c-source with this command:: g++ -Wall --std=c++0x xmlrpc-cpp.cpp -lxmlrpc_client++ -o xmlrpc-cpp Here is the program:: /* This program needs XML-RPC: http://xmlrpc-c.sourceforge.net How to start the server: cd samples SDxmlrpc --precmdfile Xprecmd.txt --port 8000 compile with this command: g++ -Wall --std=c++0x xmlrpc-cpp.cpp -lxmlrpc_client++ -o xmlrpc-cpp */ #include #include /* only needed for my_hostname */ #include /* only needed for my_hostname */ #include #include #include #include #define PORT 8000 static char *_get_hostname(void) { struct hostent* h; char hostname[1024]; hostname[1023] = '\0'; gethostname(hostname, 1023); h = gethostbyname(hostname); return strdup(h->h_name); } static std::string url(std::string host, int port) { std::string st= std::string("http://"); st.append(host); st.append(":"); st.append(std::to_string(port)); st.append("/RPC2"); return st; } int main(int const argc, const char ** const argv) { std::string const serverUrl= url(std::string(_get_hostname()), PORT); xmlrpc_c::clientSimple myClient; xmlrpc_c::value *result_p; result_p= new xmlrpc_c::value(); myClient.call(serverUrl, "txt.find", "ssss", result_p, "id-data.*.names.devicename", "", "", "idcp_db"); std::cout << std::string(xmlrpc_c::value_string(*result_p)) << "\n"; delete result_p; return 0; } Invoking SDxmlrpc ----------------- Here is a short overview on the SDxmlrpc command line options: --version show program's version number and exit -h, --help show this help message and exit --summary print a summary of the function of the program --info Show ip, port and process ID on stderr. --localhost start server on 'localhost' instead of DNSDOMAINNAME. In this case the server can only be contacted from applications running on the same host. --host start server on HOST instead of DNSDOMAINNAME. This may be needed for hosts with more than one network interface. --port=PORT start xmlserver on port PORT -p COMMANDS, --precmd=COMMANDS specify COMMANDS to perform before any other action --precmdfile=FILE specify a FILE to execute before any other action -M, --module specify a MODULE to import at make its functions accessible by XMLRPC -I, --searchpath specify a DIRECTORY to prepend it to the module search path. --no-locking do not lock file accesses --pidfile=PIDFILE specify the PIDFILE where PID's of sub processes will be stored --kill just kill old servers, do not start new ones. --restart restart the already running server Precommands +++++++++++ Precommands are commands that are executed at the start of the SDxmlrpc server before any other command. These commands can be given as a command line parameter (--precmd) or they can be read from a file (--precmdfile). A typical application is to put the command to read a StructuredData file in a file and provide it's name with --precmdfile. Extensions ++++++++++ These are user supplied python modules that can be loaded by the SDxmlrpc. The module name (the filename without ".py") is provided with the command line option "-M". In this case the python module is loaded and it's functions are accessible with the module name as a prefix. You can use command line option "-I" in order to extend the search path for extensions which are basically python modules. Keep in mind that extensions are also searched in all paths specified by the "PYTHONPATH" environment variable. Here is an example: We have a file "myXext.py" with this content:: import StructuredData.SDshelllibTxt as txt import StructuredData.SDshelllibFun as fun def ids(): p= fun.paths("id-data.*",sdc="idcp_db") return fun.poppath(p, no=-1) def show_ids(formatspec="yaml"): return txt.format(ids(), formatspec) We also have a file "Xprecmd.txt" with this content:: # create a named StructuredDataContainer: fun.namedsdc("idcp_db") # read idcp_db StructuredData file: fun.read("idcp_db.cache.SDCyml","","idcp_db") Now we start SDxmlrpc with "-M" to load the extension and with "--precmdfile" to load the sample StructuredData file from the "samples" directory:: SDxmlrpc -M myext --precmdfile Xprecmd.txt --localhost --port 8000 To test the server we use the interactive python shell. We start python by entering "python" on the command line:: $ python Python 2.7.3 (default, Jul 24 2012, 10:05:39) [GCC 4.7.0 20120507 (Red Hat 4.7.0-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import xmlrpclib >>> import pprint >>> s = xmlrpclib.ServerProxy('http://localhost:8000') >>> pprint.pprint(s.myXext.ids()) ['U125/1', 'U125/2', 'U139', 'U2', 'U3', 'U4', 'U41', 'U48', 'U49/1', 'U49/2', 'UE112', 'UE46', 'UE49', 'UE52', 'UE56/1', 'UE56/2', 'UE56R', 'Ubonsai'] >>> print s.myXext.show_ids("yaml") - U125/1 - U125/2 - U139 - U2 - U3 - U4 - U41 - U48 - U49/1 - U49/2 - UE112 - UE46 - UE49 - UE52 - UE56/1 - UE56/2 - UE56R - Ubonsai Process management ++++++++++++++++++ When SDxmlrpc is started it is useful to know it's process id (PID) to be able to restart the server by killing the old one and starting a new one. This is done with the options --pidfile and --kill. --pidfile is used to specify the name of a PID file, this file contains a line with the process id (PID) of the server and the command that was used to start the server. When --pidfile is provided, the process named in this file (and it's children) are killed first. When SDxmlrpc is started, it's PID and command line are put to the PID file. If you dont't want to restart an SDxmlrpc server but just want to kill the old one, use --pidfile together with --kill. Restarting the server +++++++++++++++++++++ The server can forced do a complete restart. In this case it reloads the precommand file and recreates all internal variables. This may be useful when the StructuredData file on disk was changed and the server needs to reload the file. The server can either be restared by sending it signal SIGUSR1 like in:: kill SIGUSR1 PID where PID is the process ID, an integer, of the SDxmlrpc server, or by invoking:: SDxmlrpc --pidfile PIDFILE --restart when PIDFILE is the process ID file that was created when the server was started.