SDpyshell Examples

All these examples use the SDpyshell and the file “idcp_db.cache.SDCyml” of the “samples” directory that is provided with the source of the python StructuredDataStore module. They give you an overview of the capabilities of StructuredData. For more details see also the chapter “Reference”.

Details of the SDpyshell syntax are described in the chapter “Working with the SDpyshell”, it is the same syntax as of python version 2.7.

Start the SDpyshell and read the data file

Change to the “samples” directory of the source distribution:

cd samples

Now we start SDpyshell and read the file “idcp_db.cache.SDCyml”. Note that lines beginning with “>” are commands entered at the SDpyshell prompt:

> SDpyshell
SDpython shell
>>> r("idcp_db.cache.SDCyml")

Reports with get

The “get” command returns the data structure referenced by a given path. Here we want to see everything of the group “global” for the “UE112” device and pretty-print this as a python structure:

>>> get("id-data.UE112.global", "py")
{'description': 'UE112  ,UE112ID7R,idcp13',
 'device_status': 'installed',
 'id-key': 13,
 'instance_no': 0,
 'primary_key': 39,
 'undulator': 'UE112'}

We can also specify YAML as format for the result:

>>> get("id-data.UE112.global", "yaml")
description: UE112  ,UE112ID7R,idcp13
device_status: installed
id-key: 13
instance_no: 0
primary_key: 39
undulator: UE112

get can also print simple scalar values:

>>> get("id-data.UE112.global.device_status", "py")
'installed'

If we don’t want to have simple values printed as python literals, we can omit the format specification “:py”. In this case get uses the “raw” format, which prints the raw values:

>>> get("id-data.UE112.global.device_status")
installed

If provided with a pattern get returns a list if more than one path matches:

>>> get("id-data.*.global.id-key")
[96, 3, 110, 98, 97, 95, 12, 80, 7, 15, 13, 10, 8, 9, 11, 5, 81, 99]

Reports with find

Find accepts wildcards in the path argument. Currently wildcard is just the string “*”. A path with optional wildcards is called a pattern. With this feature we can print data for several paths with a single command.

Here we print the “id-key” property for all devices:

>>> find("id-data.*.global.id-key")
id-data.U125/1.global.id-key : 96
id-data.U125/2.global.id-key : 3
id-data.U139.global.id-key   : 110
id-data.U2.global.id-key     : 98
id-data.U3.global.id-key     : 97
id-data.U4.global.id-key     : 95
id-data.U41.global.id-key    : 12
id-data.U48.global.id-key    : 80
id-data.U49/1.global.id-key  : 7
id-data.U49/2.global.id-key  : 15
id-data.UE112.global.id-key  : 13
id-data.UE46.global.id-key   : 10
id-data.UE49.global.id-key   : 8
id-data.UE52.global.id-key   : 9
id-data.UE56/1.global.id-key : 11
id-data.UE56/2.global.id-key : 5
id-data.UE56R.global.id-key  : 81
id-data.Ubonsai.global.id-key: 99

The find command uses the “aligned” format as a default. It can also return it’s results in YAML or python. Note that find searches all paths in the StructuredData to test if they match the given pattern.

If you enter a path without wildcards, find returns the same as get would do with the “aligned” format. However, find is much slower than get in this case since it tests all paths where get directly uses the path provided.

Reports with rxfind

rxfind tests all paths whether they match a regular expression. Regular expressions are quite powerful for searching strings. Since a regular expression may contain many special characters, we do not use simplified strings here but use single quotes instead:

>>> rxfind('.*UE56/2.*v_max')
id-data.UE56/2.config.v_max_diff    : 0.1
id-data.UE56/2.config.v_max_pos     : 185
id-data.UE56/2.config.v_max_velocity: 2500

Like find, rxfind returns it’s results in “aligned” format as a default.

Reports with findval

This command looks for a given scalar in the StructuredData. Like the other find commands it returns it’s results in “aligned” format as default:

>>> findval(20)
id-data.U125/2.config.v_critical_pos       : 20.0
id-data.U2.config.v_critical_pos           : 20.0
id-data.U3.config.v_critical_pos           : 20.0
id-data.U3.feedback.cc_tables              : 20
id-data.U49/1.config.v_critical_pos        : 20.0
id-data.UE112.config.coil_number           : 20
id-data.UE56/2.feedback.cc_tables          : 20
id-metadata.parameter-info.id-key.order_key: 20

If provided with a pattern findval looks only at paths that match the given pattern:

>>> findval(20, pattern="id-data.*.config.v_critical_pos")
id-data.U125/2.config.v_critical_pos: 20.0
id-data.U2.config.v_critical_pos    : 20.0
id-data.U3.config.v_critical_pos    : 20.0
id-data.U49/1.config.v_critical_pos : 20.0

You can now combine the results of findval from the example above with other commands. In the following example we use the results from the query above. We call “fun.findval” instead of “findval” and store it’s results in variable “n”. We then modify the returned paths with “fun.substpath” and use “get” with it’s results. In the end we get the values of the property “devicename” for all the devices:

>>> n=fun.findval(20, pattern="id-data.*.config.v_critical_pos")
>>> get(fun.substpath(n, "id-data.*.names.devicename"))
['U49ID4R', 'U3IV', 'U2IV', 'U125ID2R']

Reports with rxfindval

This command matches all scalar values with a given regular expression. Here is an example:

>>> rxfindval('.*-ppc')
id-data.U125/2.config.target_arch : vxWorks-ppc603
id-data.U139.config.target_arch   : vxWorks-ppc603
id-data.U3.config.target_arch     : vxWorks-ppc603
id-data.U41.config.target_arch    : vxWorks-ppc603
id-data.U49/1.config.target_arch  : vxWorks-ppc603
id-data.U49/2.config.target_arch  : vxWorks-ppc603
id-data.UE112.config.target_arch  : vxWorks-ppc603
id-data.UE46.config.target_arch   : vxWorks-ppc603
id-data.UE49.config.target_arch   : vxWorks-ppc603
id-data.UE52.config.target_arch   : vxWorks-ppc603
id-data.UE56/1.config.target_arch : vxWorks-ppc603
id-data.Ubonsai.config.target_arch: vxWorks-ppc603

Reports with pr

The pr command is used to print StructuredData or parts of it to the console. Note that this is different from “print”, which is the generic print command to print any value (numbers, strings, data structures) to the console.

pr filters it’s results when a pattern is provided. As a default it returns it’s results in YAML format. pr usually would also print all the type declarations. In this example we want only see the pure data, so we specify only to print the “store” in the format specification:

>>> pr("store", pattern="id-data.UE112.global.**")
id-data:
    UE112:
        global:
            description: UE112  ,UE112ID7R,idcp13
            device_status: installed
            id-key: 13
            instance_no: 0
            primary_key: 39
            undulator: UE112

pr can also return it’s results in csv format:

>>> p("store:csv", pattern="id-data.UE112.global.**")
id-data;UE112;global;description;UE112  ,UE112ID7R,idcp13
id-data;UE112;global;device_status;installed
id-data;UE112;global;id-key;13
id-data;UE112;global;instance_no;0
id-data;UE112;global;primary_key;39
id-data;UE112;global;undulator;UE112

Changing a single value with change

This command is used to change an existing value. Here we read a value, change it and then read it again:

>>> get("id-data.Ubonsai.global.id-key")
99
>>> change("id-data.Ubonsai.global.id-key", 100)
>>> get("id-data.Ubonsai.global.id-key")
100

Note that for the change command, a node for the specified path must already exist. If it doesn’t, you get an error message:

>>> change("id-data.Ubonsai.global.id-keyx", 101)
path "id-data.Ubonsai.global.id-keyx" doesn't exist

Adding a value with put

The put command can create all parts of a path on the fly. It is not necessary that all nodes for the specified path already exist.

Here is the node for path “id-data.U48.global” before the change:

>>> get("id-data.U48.global", "py")
{'description': 'U48    ,U48IV    ,idcp80,located in lund',
 'device_status': 'test',
 'id-key': 80,
 'instance_no': 0,
 'primary_key': 40,
 'undulator': 'U48'}

Now we add a value and create a new map on the fly:

>>> put("id-data.U48.global.newdict.param1", 10)

This is how the node now looks like:

>>> get("id-data.U48.global", "py")
{'description': 'U48    ,U48IV    ,idcp80,located in lund',
 'device_status': 'test',
 'id-key': 80,
 'instance_no': 0,
 'newdict': {'param1': 10},
 'primary_key': 40,
 'undulator': 'U48'}

Working with types

The simplies action with types is to perform a typecheck of a StructuredDataContainer file from the command line. Here is an example:

> SDpyshell -c 'r("idcp_db.cache.SDCyml");typecheck()'
all typechecks succeeded

We can, however, explore and change types in the interacticve shell. First we start the shell and load the file “idcp_db.cache.SDCyml”:

> SDpyshell
SDpython shell
>>> r("idcp_db.cache.SDCyml")

We first print a certain value:

>>> find("id-data.UE46.names.**", "aligned")
id-data.UE46.names.devicename: UE46IT5R
id-data.UE46.names.key       : 10
id-data.UE46.names.name      : UE46
id-data.UE46.names.prefix    : idcp10

We now want to know if a type is defined for this path. The typematch command prints the matching type declaration if it finds one:

>>> typematch("id-data.UE46.names")
id-data.*.names:
    optional_struct:
    - devicename
    - key
    - name
    - prefix

The first line of the result is the matching pattern in the StructuredDataTypes object. The part that follows is the type declaration. We see that this is an “optional_struct” meaning that this is a map where all keys must be keys of the given list of keys but not all keys of the list must be present.

Now we want to see if there are type declarations for the members of this “optional_struct”. The typepaths command just prints the patterns without the type declarations:

>>> typepaths("id-data.*.names.*")
- id-data.*.names.devicename
- id-data.*.names.key
- id-data.*.names.name
- id-data.*.names.prefix

We want to see what the type declaration for path “id-data.*.names.devicename” is:

>>> typeget("id-data.*.names.devicename")
string
...

The “…” is created by converting a simple string to YAML.

The typefind command can be used to print all types that match a pattern:

>>> typefind("id-data.*.names.*")
id-data.*.names.devicename: string
id-data.*.names.key       : integer
id-data.*.names.name      : string
id-data.*.names.prefix    : string

Types can also be modified on the command line. We can for example remove an item in the optional_struct:

>>> typedeleteitem("id-data.*.names", "key")

If we verify the result we see that the item is gone:

>>> typeget("id-data.*.names")
optional_struct:
- devicename
- name
- prefix

We can also add a new item:

>>> typeadditem("id-data.*.names", "newkey")

We again verify the result:

>>> typeget("id-data.*.names")
optional_struct:
- devicename
- name
- newkey
- prefix

We can remove a complete type declaration:

>>> typedelete("id-data.*.names.devicename")

We can add a type, even a complex one. For a struct for example we have to provide a dictionary, we use simplified strings in this case. Note that the space after “:” after “struct” is important. Without it SDpyshell would interpret “:[” as a string literal:

>>> typeput("id-data.*.newnames", {"struct" : ["elm1", "elm2", "elm3" ]})

We verify the result:

>>> typeget("id-data.*.newnames")
struct:
- elm1
- elm2
- elm3

A change of many parameters with export and re-import

First we extract all parameters named “device_status” for all insertion devices, we use the “flat” format in order to make the file more easily editable:

SDpyshell -c 'r("idcp_db.cache.SDCyml");w("PARAMS.TXT", "store:flat", pattern="id-data.*.global.device_status")'

The file “PARAMS.TXT” now looks like this:

id-data.U125/1.global.device_status: test
id-data.U125/2.global.device_status: installed
id-data.U139.global.device_status: installed
id-data.U2.global.device_status: simulated
id-data.U3.global.device_status: test
id-data.U4.global.device_status: test
id-data.U41.global.device_status: installed
id-data.U48.global.device_status: test
id-data.U49/1.global.device_status: installed
id-data.U49/2.global.device_status: installed
id-data.UE112.global.device_status: installed
id-data.UE46.global.device_status: installed
id-data.UE49.global.device_status: installed
id-data.UE52.global.device_status: installed
id-data.UE56/1.global.device_status: installed
id-data.UE56/2.global.device_status: installed
id-data.UE56R.global.device_status: test
id-data.Ubonsai.global.device_status: simulated

Now we can edit this file with a text editor or any program. In this example we want to change all parameters that have the value “test” to the value “installed”. We do this with this perl one-liner:

perl -pi -e 's/: test/: installed/' PARAMS.TXT

Now we want to merge these changes with the existing StructuredData file. We do this with a single command line:

SDpyshell -c 'r("idcp_db.cache.SDCyml"); r("PARAMS.TXT", "store:flat"); w("new.SDCyml")'

The “r” reads alls files “*.SDCyml” in the current directory. The “sread” command reads the file “PARAMS.TXT”. It assumes a StructuredDataStore in YAML and “flat” format. The data is merged with the StructuredData that was read before. Already existing values are replaced with the new ones. The final statement “w” writes the modified StructuredData to the file “new.SDCyml”.

We control the changes with “diff”:

diff idcp_db.cache.SDCyml new.SDCyml
97c97
<                 device_status: test
---
>                 device_status: installed
770c770
<                 device_status: test
---
>                 device_status: installed
944c944
<                 device_status: test
---
>                 device_status: installed
1260c1260
<                 device_status: test
---
>                 device_status: installed
3131c3131
<                 device_status: test
---
>                 device_status: installed
5702d5701
<

Converting a python structure to StructuredData

Let’s assume that we have a file, pythonstruc.py with this content:

{ "key1": 1,
  "key2": { "A": "x",
            "B": "y"
          },
  "key3": [ 1, 2, 3, { "float": 1.23 }]
}

We convert this directly with a single SDpyshell command line:

SDpyshell -c 'r("pythonstruc.py", "store:py"); w("pythonstruc.SDCyml")'

The file pythonstruc.SDCyml now looks like this:

'**SDC-Metadata**':
    version: '1.0'
'**SDC-Store**':
    key1: 1
    key2:
        A: x
        B: y
    key3:
    - 1
    - 2
    - 3
    -   float: 1.23
'**SDC-Types**': {}

Converting a perl structure to StructuredData

Let’s assume that we have a file, perlstruc.pl with this content:

%struc= ( "key1"=> 1,
          "key2"=> { "A"=> "x",
                     "B"=> "y"
                   },
          "key3"=> [ 1, 2, 3, { "float"=> 1.23 }]
        );

In order to convert this we first convert the perl structure to pure YAML with this command line:

perl -MYAML::XS -e 'require "perlstruc.pl";print Dump(\%struc);' > perlstruc.yml

Now we create a StructuredData file like this:

SDpyshell -c 'r("perlstruc.yml", "store"); w("perlstruc.SDCyml")'

The file perlstruc.SDCyml now looks like this:

'**SDC-Metadata**':
    version: '1.0'
'**SDC-Store**':
    key1: 1
    key2:
        A: x
        B: y
    key3:
    - 1
    - 2
    - 3
    -   float: 1.23
'**SDC-Types**': {}