Home > Product Manuals > Lineage > Lineage

Lineage

FastScore Lineage is a service that intercepts various internally generated events for the purposes of tracking the history of what has happened to various assets within FastScore. Lineage provides REST APIs for accessing information concerning specific events and assets, as well as for tracking various pieces of metadata associated with each.

Events

Metadata events are generated by the various FastScore components (currently Model-Manage and Engine). Events include things like adding a model, schema, or stream descriptor to Model-Manage, loading a model or stream descriptor into an Engine, as well as the various Engine state transitions.

Events are published as a JSON object that contain:

Elements consist of:

Tags are simply a map of key/value pairs that become associated with the enclosing event or element

Here’s an example of what a well-formed event might look like:

{
  "description" : "put model: model1",
  "type" : "put model",
  "elements" : [
    {
      "description" : "model1",
      "type" : "model",
      "tags" : {
        "model-hash" : "abcd1234"
      }
    },
    {
      "description" : "model-manage",
      "type" : "fastscore-service",
    }
  ],
  "tags" : {
    "timestamp" : "5:15"
  }
}

Events arrive at Lineage via Pneumo, the FastScore internal communication system. As such, they can be viewed via the CLI pneumo command.

Running Lineage

Lineage run inside a docker container called ‘fastscore/lineage’

Lineage typically listens on port 8008.

Lineage is configured via a set of ENV variables:

Currently, Lineage requires an instance of ArangoDB as its storage backend. ‘fastscore/lineagedb’ can be used. Support for MySQL is coming soon.

Retrieval APIs

lineage/1/element_types

Returns an array of the various element types that have associated events.

Example output:

[
  "fastscore-service",
  "model",
  "schema",
  "stream"
]

lineage/1/elements

Returns an array of each element, including the description, type and the id of the element. The id of the element can be used to get more information about that specific element using the ‘lineage/1/elements/[id]’ API.

[
  {
    "description": "discard",
    "id": 1367,
    "type": "stream"
  },
  {
    "description": "model-manage-1",
    "id": 325,
    "type": "fastscore-service"
  },
  {
    "description": "yield1234-py",
    "id": 863,
    "type": "model"
  }
]

This results can be filtered by element type by using ‘?element_type=[type]’, where ‘type’ is from the array returned from /1/element_types.

lineage/1/elements/[id]

Returns a detailed view of element specified by the [id].

{
  "description": "yield1234-py",
  "id": 863,
  "type": "model",
  "events": [
    {
      "description": "put model: yield1234-py",
      "id": 839,
      "timestamp": "2018-12-12T21:35:23.685Z",
      "type": "put model"
    }
  ],
  "tags": {
    "model-hash": "4b8558b70f5162d27a5f36f652999c5f",
    "model-type": "Python"
  }
}

Note that the returned JSON object has an array of events, each of which involved the specified element.

Additionally, the ‘tags’ at the end of the element contain only the most recent value associated with any particular key.

lineage/1/events

Returns a list of all the events that lineage has tracked.

[
  {
    "description": "put model: EchoJava-java",
    "id": 262,
    "timestamp": "2018-12-12T21:35:22.456Z",
    "type": "put model"
  },
  {
    "description": "put model: echo-array-double-py",
    "id": 341,
    "timestamp": "2018-12-12T21:35:22.552Z",
    "type": "put model"
  },
  {
    "description": "put model: echo-array-double-py3",
    "id": 415,
    "timestamp": "2018-12-12T21:35:22.641Z",
    "type": "put model"
  }
]

Note that the timestamp indicates the time that Lineage saw the event and persisted it in the configured backend.

lineage/1/events/[id]

Returns a detailed view of the event specified by [id]

{
  "description": "put model: echo-array-double-py3",
  "id": 415,
  "timestamp": "2018-12-12T21:35:22.641Z",
  "type": "put model",
  "elements": [
    {
      "description": "echo-array-double-py3",
      "id": 439,
      "type": "model",
      "tags": [
        {
          "id": 452,
          "key": "model-type",
          "value": "Python3"
        },
        {
          "id": 465,
          "key": "model-hash",
          "value": "278263485a0d64b1edc83c4c30c90fc7"
        }
      ]
    },
    {
      "description": "model-manage-1",
      "id": 325,
      "type": "fastscore-service"
    }
  ],
  "tags": [
    {
      "id": 423,
      "key": "pneumo-timestamp",
      "value": "2018-12-12T21:35:21.613138Z"
    }
  ]
}

In many respects, this will look quite similar to the event that was received and injected into Lineage. The main differences are that the event itself, as well as each element and tag, contain an id. Since the tags’ ids are displayed, tags are no longer represented as a map, but rather as an array of JSON objects.

lineage/1/tags

Returns a list of all the tags

[
  {
    "id": 539,
    "key": "model-hash",
    "value": "a9cd0d2191df64bb1e77d27a9881e827"
  },
  {
    "id": 1136,
    "key": "schema-hash",
    "value": "13f4e51ec271c78a3056c9eed8eef668"
  },
  {
    "id": 571,
    "key": "pneumo-timestamp",
    "value": "2018-12-12T21:35:22.285562Z"
  }
]

lineage/1/tags/[id]

Returns details concerning a specific tag

{
  "id": 452,
  "key": "model-type",
  "value": "Python3",
  "elements": [
    {
      "description": "echo-array-double-py3",
      "id": 439,
      "type": "model"
    },
    {
      "description": "emit-attach1-py3",
      "id": 725,
      "type": "model"
    },
    {
      "description": "echo-py3",
      "id": 656,
      "type": "model"
    }
  ]
}

Note that a tag can be associated with 0 or more elements as well as 0 or more events. In the example above, the ‘model-type:Python3’ key/value pair is associated only with elements, but a brief description of each (including their ids) is included.