Writing a Spawnpoint Service in Python

Using Python to Write a Spawnpoint Service

This guide will show you how to write a simple program that uses the Bosswave Python bindings and how to deploy this program so that it runs on a Spawnpoint instance.

Installing the Python Bindings

To install the Python bindings, you will need git and pip on your system. Currently, the bindings support only Python 2.

First, clone the bw2python GitHub repository (located at https://github.com/SoftwareDefinedBuildings/bw2python).

From within the newly created bw2python directory, run

pip install .

Note that you may need to use sudo for this command. If you are trying to update the Python bindings to a new version, include the --upgrade flag in your command.

Writing a Simple Counter Driver

An example Python driver is available here. It increments a counter and publishes a Bosswave message once per second until the counter hits a user-specified limit. The most important parts of the code are discussed below.

Imports

from bw2python import ponames
from bw2python.bwtypes import PayloadObject
from bw2python.client import Client

These three import statements are very common in Bosswave Python programs.

1. In Bosswave, each payload object contains information about the format of its contents. ponames allows you to refer to these payload object type identifiers via symbolic names rather than hard-coded constants. For example, a string payload object is identified in dotted form as type 64.0.1.0, but this may be written as ponames.PODFString from Python programs. A complete reference of payload object types is available here.

2. A message's payload objects are exposed as a sequence of PayloadObject instances. A PayloadObject has two attributes to describe its type in dotted and non-dotted form: respectively type_dotted and type_num. A third attribute, contents, exposes the raw data contained within the payload object.

3. The bw2python Client connects to a Bosswave agent on behalf of your program. It provides functions to perform all of the usual Bosswave operations like publishing, querying, and subscribing.

Parameters

Driver code commonly uses external files to define parameters that are specific to individual instances of that driver, such as IP addresses or polling intervals. The counter driver reads a JSON file to produce a dictionary of parameters specifying the URI to publish to and the message to publish upon incrementing the internal counter.

with open("params.json") as f:
    try:
        params = json.loads(f.read())
    except ValueError:
        print "Invalid parameter file"
        sys.exit(1)

Spawnpoint makes it easy to include parameter files in your containers, as discussed below.

Client Initialization

bw_client = Client()

As soon as a new Client instance is initialized, it will attempt to connect to a Bosswave agent. Typically, this is specified by the BW2_AGENT environment variable, which is respected by the Python bindings unless host_name and port arguments are manually specified. If this environment variable is unset and no arguments are given, a Client will try to connect to an agent running on its same host at port 28589.

The BW2_AGENT environment variable is automatically set in Spawnpoint containers, so you may safely omit any arguments when writing code to run on Spawnpoint.

bw_client.setEntityFromEnviron()

This directs the Bosswave client to set its entity (which will be used to identify the client in all future Bosswave operations) from the BW2_DEFAULT_ENTITY environment. You may also direct a client to use an entity contained within a file.

In Spawnpoint, the BW2_DEFAULT_ENTITY variable automatically refers to the entity file you have specified in a service's configuration file (described below), so you typically want to set the client's entity from the environment as was done here.

bw_client.overrideAutoChainTo(True)

This directs the client to automatically construct DOT chains for you. This should almost always be enabled, as it is here.

Publishing Messages

for i in range(num_iters):
    msg = "({}) {}".format(i, message)
    po = PayloadObject(ponames.PODFString, None, msg)
    bw_client.publish(dest_uri, payload_objects=(po,))
    time.sleep(1)

First, you will need to construct the payload objects to send in your Bosswave message. The PayloadObject constructor has three arguments:

1. The type of the PO, in dotted form.

2. The type of the PO, as an integer.

3. The contents of the PO.

Note that only one of (1) or (2) is required. If both are specified, they should be consistent with one another. It is good practice to make use of the symbolic names available from the ponames module here.

A Client's publish function has many optional arguments, but the call seen here covers typical use cases. The first argument is the Bosswave URI to publish to (in this case dest_uri is a parameter read in from an external file) and a sequence of payload objects to embed in the message.

Writing a Configuration File

Spawnpoint containers are described in simple YAML files that contain a sequence of key/value configuration pairs. Here is a YAML file that could be used with the counter driver example:

entity: counter.ent
image: jhkolb/spawnable-py:amd64
cpuShares: 256
memAlloc: 512M
includedFiles: [counter.py, params.json]
run: [python, counter.py, 50]

• entity is the Bosswave entity that will be used within the deployed container for all Bosswave operations. It is automatically set up as the BW2_DEFAULT_ENTITY environment variable within the container.

• image is the Docker image that is used to instantiate the container. jhkolb/spawnable-py:amd64 is an Ubuntu-based container with Python and the Bosswave Python bindings already installed and ready to use.

• cpuShares specifies an allocation of host CPU resources for the container. Spawnpoint offers 1024 shares per host core, and will not accept new containers for deployment if the total allocation among all currently running containers exceeds its quota.

• memAlloc specifies an allocation of host memory for the container. As with CPU shares, new containers are not deployed if the allocation cannot be satisfied.

• includedFiles is a list of files on the host machine that will be included in the container. These are made available in the current working directory of the container's entry point command.

• run specifies the command that will be executed upon container startup. Here, we simply use the Python interpreter to run the counter.py file, which was copied from the host to the container due to the previous includedFiles directive.

Using Version Control

Spawnpoint makes it easy to deploy code that is under version control and hosted on GitHub. A service configuration file may contain a source parameter specifying a repository URL. This repository will then be cloned when the container is initialized. All files and directories within the repository are placed in the container's working directory.

As an example, to deploy a container with code from the demosvc repository:

entity: demo.ent
image: jhkolb/spawnpoint:amd64
source: git+http://github.com/jhkolb/demosvc
build: [go get -d, go build -o demosvc]
run: [./demosvc, 40]
memAlloc: 512M
cpuShares: 1024
includedFiles: [params.yml]

Deploying on a Spawnpoint

The spawnctl command line tool allows you to inspect spawnpoints and deploy containers. Pre-built spawnctl releases are available for the following platforms:

• Mac OS (64-bit)
• Linux (64-bit)

As an example, say we want to deploy our counter driver on a spawnpoint running at the Bosswave URI example.ns/spawnpoint/alpha.

As a first step, we may want to perform a scan to verify that the spawnpoint is running and healthy.

$ spawnctl scan -u example.ns/spawnpoint
Discovered 1 SpawnPoint(s):
[alpha] seen 09 Dec 16 15:50 PST (4.92s) ago at <snip>
  Available Memory: 16384 MB, Available Cpu Shares: 8192
Metadata:
  • lastalive: 2016-12-09 23:50:14.655474351 +0000 UTC
  • arch: amd64
Services:

Next, deploy your code to the spawnpoint using the spawnctl tool. The deploy operation involves three parameters:

• The Bosswave URI of the spawnpoint to deploy to
• The YAML configuration file for the container
• A unique name for the container

No two containers running on the same spawnpoint may share a name.

The Python counter driver could be deployed on the example.ns/spawnpoint/alpha spawnpoint like so:

$ spawnctl deploy -u example.ns/spawnpoint/alpha -c config.yml -n counter

This assumes the following files are present in the current working directory:

• config.yml: A YAML configuration file for the container
• counter.py: The Python source for the program to be executed within the container. This is referenced in the includedFiles parameter of config.yml.
• params.json: This defines instance-specific parameters for the driver and is also referenced in the includedFiles configuration parameter, meaning it will be copied into the container along with counter.py.

Once the deploy process is started, spawnctl will tail the logs for the new service on the target spawnpoint until the user types <CTRL>-c. For example, the output might look like the following:

 Deployment complete, tailing log. Ctrl-C to quit.
[12/09 16:26:41] alpha::counter > [INFO] Booting service
[12/09 16:26:46] alpha::counter > [SUCCESS] Container (re)start successful
...

Verifying Functionality

To check if the counter driver is running properly, we can subscribe to the Bosswave messages it emits. The easiest way to do this is using the bw2 command line tool. Assuming the counter is publishing messages on scratch.ns/counter/out, we can subscribe as follows:

$ bw2 s scratch.ns/counter/out
Message from <snip>/counter/out:
PO 64.0.1.0 len 17 (human readable) contents:
(0) Hello, World!
Message from <snip>/counter/out:
PO 64.0.1.0 len 17 (human readable) contents:
(1) Hello, World!
Message from <snip>/counter/out:
PO 64.0.1.0 len 17 (human readable) contents:
(2) Hello, World!
Message from <snip>/counter/out:
PO 64.0.1.0 len 17 (human readable) contents:
(3) Hello, World!
Message from <snip>/counter/out:
PO 64.0.1.0 len 17 (human readable) contents:
(4) Hello, World!
Message from <snip>/counter/out:
PO 64.0.1.0 len 17 (human readable) contents:
(5) Hello, World!
...

This output indicates that the driver is successfully running on Spawnpoint and publishing Bosswave messages.

Appendix: The Complete Driver Program

import json
import sys
import time

from bw2python import ponames
from bw2python.bwtypes import PayloadObject
from bw2python.client import Client

if len(sys.argv) != 2:
    print "Usage: {} <num_iterations>".format(sys.argv[0])
    sys.exit(1)

with open("params.json") as f:
    try:
        params = json.loads(f.read())
    except ValueError:
        print "Invalid parameter file"
        sys.exit(1)

dest_uri = params.get("dest_uri")
if dest_uri is None:
    print "No 'dest_uri' parameter specified"
    sys.exit(1)
message = params.get("message")
if message is None:
    print "No 'message' parameter specified"
    sys.exit(1)

try:
    num_iters = int(sys.argv[1])
except ValueError:
    print "Invalid number of iterations specified"
    sys.exit(1)

bw_client = Client()
bw_client.setEntityFromEnviron()
bw_client.overrideAutoChainTo(True)

for i in range(num_iters):
    msg = "({}) {}".format(i, message)
    po = PayloadObject(ponames.PODFString, None, msg)
    bw_client.publish(dest_uri, payload_objects=(po,))
    time.sleep(1)

print "Emitted {} messages. Terminating".format(num_iters)