Developer interface

Service instances

class fastscore.suite.Connect(proxy_prefix, auth_secret=None)[source]

An instance of a Connect service.

Typically, an interaction with FastScore starts with:

>>> from fastscore.suite import Connect
>>> connect = Connect("https://localhost:8000")

Afterwards, you can use ‘connect’ to access other FastScore instances. For example:

>>> engine = connect.lookup('engine')
active_sensors

A collection of currently installed sensors indexed by id.

>>> engine = connect.lookup('engine')
>>> engine.active_sensors.keys()
[8]
>>> x = engine.active_sensors[8]
>>> x.id
8
>>> x.tap
manifold.input.records.count
>>> x.uninstall()
>>> engine.active_sensors.ids()
[]
check_health()

Retrieves version information from the instance. A successful reply indicates that the instance is healthy.

>>> connect.check_health()
{
  'id': '366e5030-d773-49cb-8b28-9b1b9d173c79',
  'built_on': 'Thu May 11 12:53:39 UTC 2017',
  'release': '1.5'
}
configure(config)[source]

Sets the FastScore configuration.

>>> with open('config.yaml') as f:
>>>   connect.configure(yaml.load(f))
False
Parameters:config – a dict describing a FastScore configuration.
Returns:True, if an existing configuration has been replaced and False, otherwise.
dump(savefile)[source]

Saves the Connect parameters to a file.

>>> connect.dump(".fastscore")
fleet()[source]

Retrieves metadata for all running instances.

Returns:an array of dicts describing running FastScore instances. Each dict contains the following fields:
  • id: the internal instance id (do not use)
  • api: the service name, e.g. ‘model-manage’
  • release: the instance release, e.g ‘1.5’
  • built_on: the human-readable build date and time
  • host: the host name of the instance REST API
  • port: the port of the instance REST API
  • health: the current health status of the instance.
get(name)[source]

Retrieves a (cached) reference to the named instance.

>>> mm = connect.get('model-manage-1')
>>> mm.name
'model-manage-1'
Parameters:name – a FastScore instance name.
Returns:a FastScore instance object.
get_config(section=None)[source]

Retrieves the current FastScore configuration.

>>> connect.config('db')
{
  'username': 'root',
  'host': 'database',
  'password': 'root',
  'type': 'mysql',
  'port': 3306
}
Parameters:section – gets only the named section of the configuration
Returns:a dict with the FastScore configuration.
get_swagger()

Retrieves the Swagger specification of the API supported by the instance.

>>> connect.get_swagger()
{u'info':...}
static load(savefile)[source]

Recreates a Connect instance from a file.

>>> connect = Connect.load(".fastscore")
login(username, password)[source]

Login to FastScore.

Parameters:
  • username – a user name.
  • password – a password.
lookup(sname)[source]

Retrieves an preferred/default instance of a named service.

>>> engine = connect.lookup('engine')
>>> engine.name
'engine-1'
Parameters:sname – a FastScore service name, e.g. ‘model-manage’.
Returns:a FastScore instance object.
pneumo

Access Pneumo messages.

>>> pneumo = connect.pneumo.socket()
>>> pneumo.recv()
>>> pneumo.close()
>>> connect.pneumo.history()
prefer(sname, name)[source]

Marks the named instance as preferred for a given service.

>>> connect.prefer('engine', 'engine-3')
>>> engine = connect.lookup('engine')
>>> engine.name
'engine-3'
Parameters:
  • sname – a FastScore service name, e.g. ‘model-manage’.
  • name – the name of preferred instance of the given service.
tapping_points

A list of tapping points supported by the instance.

>>> mm.tapping_points
['sys.memory',... ]
target

Gets/Sets the target instance. When set, the target instance also becomes the preferred instance of the service it represents.

>>> engine = connect.get('engine-3')
>>> connect.target = engine
class fastscore.suite.ModelManage(name)[source]

An instance of a Model Manage service. Use Connect to create a ModelManage instance:

>>> mm = connect.lookup('model-manage')
active_sensors

A collection of currently installed sensors indexed by id.

>>> engine = connect.lookup('engine')
>>> engine.active_sensors.keys()
[8]
>>> x = engine.active_sensors[8]
>>> x.id
8
>>> x.tap
manifold.input.records.count
>>> x.uninstall()
>>> engine.active_sensors.ids()
[]
check_health()

Retrieves version information from the instance. A successful reply indicates that the instance is healthy.

>>> connect.check_health()
{
  'id': '366e5030-d773-49cb-8b28-9b1b9d173c79',
  'built_on': 'Thu May 11 12:53:39 UTC 2017',
  'release': '1.5'
}
get_swagger()

Retrieves the Swagger specification of the API supported by the instance.

>>> connect.get_swagger()
{u'info':...}
models

A collection of Model objects indexed by model name.

>>> mm.models.names()
[u'model-1']
>>> model = mm.models['model-1']
>>> model.mtype
'python'
>>> del mm.models['model-1']
>>> mm.models.names()
[]
schemata

A collection of Schema objects indexed schema name. The alternative name for the property is ‘schemas’.

>>> from fastscore import Schema
>>> s = Schema('schema-1')
>>> s.source = '{"type": "string"}'
>>> s.update(mm)
False   # Success; schema created, not updated
>>> mm.schemata.names()
['schema-1']
>>> del mm.schemata['schema-1']
>>> mm.schemata.names()
[]
sensors

A collection of Sensor objects indexed by sensor name.

>>> from fastscore import Sensor
>>> s = Sensor('sensor-1')
>>> s.desc = {'tap': 'manifold.input.records.size',... }
>>> s.update(mm)
False   # Success; sensor created, not updated
>>> mm.sensors.names()
['sensor-1']
>>> del mm.sensors['sensor-1']
>>> mm.sensors.names()
[]
streams

A collection of Stream objects indexed by stream name.

>>> mm.streams.names()
['demo-1','demo-2]
>>> mm.streams['demo-1'].desc
{u'Description': u'A demo stream... }
>>> del.strems['demo-2']
>> mm.streams.names()
['demo-1']
tapping_points

A list of tapping points supported by the instance.

>>> mm.tapping_points
['sys.memory',... ]
class fastscore.suite.Engine(name)[source]

An Engine instance.

active_model

The currently loaded model information.

>>> mm = connect.lookup('model-manage')
>>> engine = connect.lookup('engine')
>>> print stream.active_model.name
>>> print stream.active_model.jets
Returns:An ActiveModelInfo object.
active_sensors

A collection of currently installed sensors indexed by id.

>>> engine = connect.lookup('engine')
>>> engine.active_sensors.keys()
[8]
>>> x = engine.active_sensors[8]
>>> x.id
8
>>> x.tap
manifold.input.records.count
>>> x.uninstall()
>>> engine.active_sensors.ids()
[]
active_streams

A collection of active streams indexed by a slot.

>>> mm = connect.lookup('model-manage')
>>> engine = connect.lookup('engine')
>>> print stream.active_streams[1]
check_health()

Retrieves version information from the instance. A successful reply indicates that the instance is healthy.

>>> connect.check_health()
{
  'id': '366e5030-d773-49cb-8b28-9b1b9d173c79',
  'built_on': 'Thu May 11 12:53:39 UTC 2017',
  'release': '1.5'
}
get_swagger()

Retrieves the Swagger specification of the API supported by the instance.

>>> connect.get_swagger()
{u'info':...}
input(data, slot)[source]

Write data to a REST stream attached to the slot.

Parameters:
  • data – The data to write to the stream.
  • slot – The stream slot.
load_model(model, force_inline=False, embedded_schemas={}, dry_run=False)[source]

Load a model into this engine.

Parameters:
  • model – A Model object.
  • force_inline – If True, force all attachments to load inline. If False, attachments may be loaded by reference.
  • embedded_schemas – A dict of schemas to send with the request to stop the Engine from contacting Model Manage when resolving schemas.
  • dry_run – If True, do not actually load the model, check for errors only.
output(slot)[source]

Reads data from the REST stream attached to the slot.

Parameters:slot – The stream slot.
pause()[source]

Pauses the engine. The result depends on the current state of the engine. A running engine changes its state to PAUSED. An initializing engine will pause upon startup. In all other states the operation is ignored.

policy

Set/get the import policy.

>>> engine = connect.lookup('engine-1')
>>> engine.policy.set('import', 'python', text)
>>> print engine.policy.get('import', 'python')
reset()[source]

Resets the engine. A loaded model is unloaded. All open streams are closed. The engine changes its state to INIT.

scale(factor)[source]

Changes the number of running model instances.

state

The current state of the engine.

Returns:‘INIT’, ‘RUNNING’, ‘PAUSED’, ‘FINISHING’, or ‘FINISHED’.
tapping_points

A list of tapping points supported by the instance.

>>> mm.tapping_points
['sys.memory',... ]
unpause()[source]

Unpauses the engine.

verify_data(sid, rec)[source]

Verify schema against a prepared schema.

Parameters:
  • sid (int) – The schema id.
  • rec (str) – The data record to verify.

Exceptions

class fastscore.FastScoreError(message, caused_by=None)[source]

A FastScore exception.

SDK functions throw only FastScoreError exceptions. An SDK function either succeeds or throws an exception. The return value of a SDK function is always valid.

Pneumo

class fastscore.PneumoSock(proxy_prefix, timeout=None, src=None, type=None, **kwargs)[source]

The Pneumo websocket.

>>> pneumo = connect.pneumo()
>>> pneumo.recv()
LogMsg(src=..., timestamp=..., ...)
close()[source]

Close the Pneumo socket.

recv()[source]

Receives the next Pneumo message.

Models

class fastscore.Model(name, mtype='python', source=None, model_manage=None)[source]

Represents an analytic model. A model can be created directly:

>>> model = fastscore.Model('model-1')
>>> model.mtype = 'python'
>>> model.source = '...'

Or, retrieved from a Model Manage instance:

>>> mm = connect.lookup('model-manage')
>>> model = mm.models['model-1']

A directly-created model must be saved to make attachment and snapshot manipulation functions available:

>>> mm = connect.lookup('model-manage')
>>> model.update(mm)
>>> model.attachments.names()
[]
attachments

A collection of model attachments. See Attachment.

deploy(engine)[source]

Deploy this model to an engine.

Parameters:engine – The Engine instance to use.
mtype

A model type:

  • pfa-json: a PFA model in JSON format.
  • pfa-yaml: a PFA model in YAML format.
  • pfa-pretty: a PrettyPFA model.
  • h2o-java: an H20 model.
  • python: a Python model.
  • python3: a Python 3 model.
  • R: an R model.
  • java: a Java model.
  • c: a C model.
  • octave: an Octave model.
  • sas: a SAS model.
name

A model name, e.g. ‘model-1’.

source

The source code of the model.

Attachments

class fastscore.model.Attachment(name, atype=None, datafile=None, datasize=None, model=None)[source]

Represents a model attachment. An attachment can be created directly but it must (ultimately) associated with the model:

>>> att = fastscore.Attachment('att-1', datafile='/tmp/att1.zip')
>>> model = mm.models['model-1']
>>> att.upload(model)
Parameters:
  • atype – An attachment type. Guessed from the data file name if omitted.
  • datafile – The data file.
  • model – The model instance.
atype

An attachment type.

  • zip A ZIP archive.
  • tgz A gzipped tarball.
datafile

A name of the file that contains the attachment data. The attachment is downloaded when this property is first accessed.

datasize

The size of the attachment. Checking the attachment size does NOT trigger the download.

name

An attachment name.

upload(model=None)[source]

Adds the attachment to the model.

Parameters:model – The model instance. Can be None if the model instance has been provided when the attachemnet was created.

Snapshots

class fastscore.model.Snapshot(snapid, created_on, stype, size, model)[source]

Represents a snapshot of a model state. Do not create directly. Use the model’s snapshots collection:

>>> model = mm.models['model-1']
>>> model.snapshots.browse(count=1)
[{'id': 'yu647a',...}]
>>> snap = model.snapshots['yu']  # prefix is enough
created_on

A date the snapshot has been taken.

id

A snapshot id.

restore(engine)[source]

Restore the model state using the snapshot.

>>> snap = model.snapshots['yu']  # prefix is enough
>>> snap.restore(engine)
size

A size of the snapshot in bytes.

Streams

class fastscore.Stream(name, desc=None, model_manage=None)[source]

A FastScore stream. A stream can be created directly:

>>> stream = Stream('stream-1')
>>> stream.desc = {'Transport':...}

Or, retrieved from a Model Manage instance:

>>> mm = connect.lookup('model-manage')
>>> stream = mm.streams['stream-1']
attach(engine, slot, dry_run=False)[source]

Attach the stream to the engine.

Parameters:slot – The stream slot.
desc

A stream descriptor (a dict).

>>> stream = mm.streams['stream-1']
>>> stream.desc
{'Transport': {'Type': 'discard'}, 'Encoding': 'json'}
name

A stream name.

rate(engine)[source]

Measures the stream throughput outside of a data pipeline.

Parameters:engine – An Engine instance to use.
sample(engine, n=None)[source]

Retrieves a few sample records from the stream.

Parameters:
  • engine – An Engine instance to use.
  • n – A number of records to retrieve (default: 10).
Returns:

An array of base64-encoded records.

update(model_manage=None)[source]

Saves the stream to Model Manage.

Parameters:model_manage – The Model Manage instance to use. If None, the Model Manage instance must have been provided when then stream was created.

Schemata

class fastscore.Schema(name, source=None, model_manage=None)[source]

An Avro schema. It can be created direct

name

A schema name.

source

A schema source, e.g. {‘type’: ‘array’, ‘items’: ‘int’}.

update(model_manage=None)[source]

Saves the schema to Model Manage.

Parameters:model_manage – The Model Manage instance to use. If None, the Model Manage instance must have been provided when then schema was created.
verify(engine)[source]

Asks the engine the check the schema.

Returns:id of the loaded schema. The identifier can be used to validate

data records:

>>> engine = connect.lookup('engine')
>>> sid = schema.verify(engine)
>>> engine.validate_data(sid, rec)

Sensors

class fastscore.Sensor(name, desc=None, model_manage=None)[source]

Represents a FastScore sensor. A sensor can be created directly:

>>> sensor = fastscore.Sensor('sensor-1')
>>> sensor.desc = {'tap': 'manifold.input.records.size',...}

Or, retreieved from Model Manage:

>>> mm = connect.lookup('model-manage')
>>> mm.sensors['sensor-1']
>>> mm.desc
{...}
desc

A sensor descriptor (a dict).

install(target)[source]

Install/attach the sensor.

Parameters:target – The instance to attach the sensor to.
name

A sensor name.

update(model_manage=None)[source]

Saves the sensor to Model Manage.

Parameters:model_manage – The Model Manage instance to use. If None, the Model Manage instance must have been provided when then sensor was created.

Deploy