The Data Source Journey¶

Start with a data set as Atlas Record.
Create a release. This runs each row of your dataset through publisher transformations you define. If you have things set up correctly, it ends up with a release in the format of the attributes you have defined. This release is not an Atlas Record, and you can’t access it directly. Each release is put into the cloud, and can be accessed from the app.
Set the current release for the data_source. Since there are multiple releases, the current release determines which release is shown when a user requests a data_source.
Publish. As soon as you publish, users will start seeing the data from the current release.

Data Source Definitions¶

Tools within the Sightglass App rely on well-formed and clean data. Each solution can define data sources that tools within that solution can access.

Example:

    "data_sources": [
      {
        "id": "table_name",
        "type": "AtomSet",
        "attributes": {
          ...
        },
        "publisher_transformations": {
          ...
        }
      }
    ]

Attributes¶

Attributes are fields that a tool can use in metrics. Each field defined here is available to tools as the datatype defined here.

Field ids are case sensitive.

Available types:

array
integer
number
string
boolean

Objects are not supported, as there aren’t metrics that support their use.

Array attributes¶

array type fields require a second items object that defines the type of values in the array.

Example:

    "attributes": {
      "field_one": {
        "type": "string"
      },
      "field_two": {
        "type": "array",
        "items": {
          "type": "integer"
        }
      }
    }

Publisher Transformations¶

Metrics expect that the attributes are well-formed and clean. Often, you will need to transform data in some way to ensure that it is as expected.

You can envision your atlas record schema as the input to publisher transformations, and the data_source attributes schema as the desired output.

If the field is already in the desired format, simply pass true as the value to include it as an attribute.

Example:

    "publisher_transformations": {
      "field_one": true
    }

If transformations are needed, pass an array of transformations.

Special Data Types¶

Certain data types require special consideration when being used.

Boolean¶

In order to include an attribute of type boolean in your metrics, you need to start with a specific string in your atlas record and apply an appropriate publisher transform.

The boolean, map_parse_boolean, and map_parse_boolean_array transformations accept any capitalization of true and false as acceptable input.

Example:

"true", "True", and "TRUE" map to true.

Null¶

All map-type transformations recognize null values. Null values include "null" and empty delimiters.

Example:

An atlas record like 1~~3~null split on the ~ delimiter maps to [1,null,3,null]

Rename¶

If the field name in the atlas record doesn’t match the field name in the attribute, you can map the fields using rename.

It is likely simpler to keep the field names the same, and avoid using this transform.

Example:

    "publisher_transformations": {
      "PricePerUnit": [
        {
          "type": "rename",
          "name": "CostPerUnit"
        }
      ]
    }

The above example will take a field called PricePerUnit in the source atlas record’s schema, and make it available as an attribute called CostPerUnit.

Lower Case¶

Strings can be transformed to lower case using lower_case. This is especially recommended for sentiment fields.

Example:

    "publisher_transformations": {
      "field_one": [
        {
          "type": "lower_case",
        }
      ]
    }

The above example will take a field from the atlas record like Positive and transform it to positive.

Upper Case¶

Strings can be transformed to upper case using upper_case.

Example:

    "publisher_transformations": {
      "field_one": [
        {
          "type": "upper_case",
        }
      ]
    }

The above example will take a field from the atlas record like Positive and transform it to POSITIVE.

Boolean¶

Strings can be transformed to Booleans using boolean.

Example:

    "publisher_transformations": {
      "field_one": [
        {
          "type": "boolean",
        }
      ]
    }

The above example will take a field from the atlas record like "true" and transform it to true.

Split¶

Many metrics expect an array of values. Use the split transform to go from a delimited field to an array of values as strings.

You will need to use a delimitter inside your field that is different from the atlas record’s delimitter. Recommended setup is to use psv for the atlas record, and ~ as the delimitter in the field.

Example:

    "publisher_transformations": {
      "field_one": [
        {
          "type": "split",
          "delimiter": "~"
        }
      ]
    }

The above example will take a field from the atlas record like a~b~c and transform it to ['a','b','c'].

Map Parse Integer¶

After splitting an attribute, you may wish to transform each value in the array to an integer.

Example:

    "publisher_transformations": {
      "field_one": [
        {
          "type": "split",
          "delimiter": "~"
        },
        {
          "type": "map_parse_integer"
        }
      ]
    }

The above example will take a field from the atlas record like 1~2~3 and transform it to [1,2,3].

Map Parse Float¶

After splitting an attribute, you may wish to transform each value in the array to a float.

Example:

    "publisher_transformations": {
      "field_one": [
        {
          "type": "split",
          "delimiter": "~"
        },
        {
          "type": "map_parse_float"
        }
      ]
    }

The above example will take a field from the atlas record like 1~2.3~3.14 and transform it to [1.0,2.3,3.14].

Map Parse Boolean¶

After splitting an attribute, you may wish to transform each value in the array to a boolean.

Example:

    "publisher_transformations": {
      "field_one": [
        {
          "type": "split",
          "delimiter": "~"
        },
        {
          "type": "map_parse_boolean"
        }
      ]
    }

The above example will take a field from the atlas record like true~true~false and transform it to [true, true, false].

Map Parse String Array¶

After splitting an array of arrays, you may wish to split the nested arrays.

Example:

    "publisher_transformations": {
      "field_one": [
        {
          "type": "split",
          "delimiter": "|"
        },
        {
          "type": "map_parse_string_array",
          "delimiter": "~"
        }
      ]
    }

The above example will take a field from the atlas record like s1~s2|s3~s4 and transform it to [["s1","s2"],["s3","s4"]].

Map Parse Integer Array¶

After splitting an array of arrays, you may wish to split the nested arrays and transform each value to an integer.

Example:

    "publisher_transformations": {
      "field_one": [
        {
          "type": "split",
          "delimiter": "|"
        },
        {
          "type": "map_parse_integer_array",
          "delimiter": "~"
        }
      ]
    }

The above example will take a field from the atlas record like 1~2|3~4 and transform it to [[1,2],[3,4]].

Map Parse Float Array¶

After splitting an array of arrays, you may wish to split the nested arrays and transform each value to a float.

Example:

    "publisher_transformations": {
      "field_one": [
        {
          "type": "split",
          "delimiter": "|"
        },
        {
          "type": "map_parse_float_array",
          "delimiter": "~"
        }
      ]
    }

The above example will take a field from the atlas record like 1.1~2.2|3.3~4.4 and transform it to [[1.1,2.2],[3.3,4.4]].

Map Parse Boolean Array¶

After splitting an array of arrays, you may wish to split the nested arrays and transform each value to a boolean.

Example:

    "publisher_transformations": {
      "field_one": [
        {
          "type": "split",
          "delimiter": "|"
        },
        {
          "type": "map_parse_boolean_array",
          "delimiter": "~"
        }
      ]
    }

The above example will take a field from the atlas record like true~false|false~true and transform it to [[true,false],[false,true]].

Data Source Usage in Views¶

tool.views.viewId.data_sources = []

AtomList and AtomDetail tools use data_sources to populate dynamic data. The AtomList, and each view, must define the data sources that will be used, and how they will be used.

Multiple data_sources may only be used when only one row is visible, specifically in AtomDetail views.

data_sources is an array of objects with the following attributes:

data_source- The ID of where to find data, as defined by the App config (not the AU2 id). Each AtomList can only have 1 data_source.
primary- Boolean, defaults to false. Each data_sources array must have exactly one primary data_source.
filters - An array of filter objects that define how the data_source will be loaded.

Views With No Data Sources¶

If a view is completely static, with no dynamic fields being loaded from a data source, you can simply omit the data_sources array. There are some caveats-

You must use a data_source for AtomLists, so this can only be used for AtomDetail views.
You can not use any filter, sort or search functionality in the view.

If you wish to have a single data source shared across multiple AtomDetail views in a tool, you can define the data source on the initial_view, and then omit data_sources from subsequent views.

When a user navigate to these views, they will use the data loaded already on the first page, and no additional requests for data will be made. All data across the views will come from a single row in a data_source.

For simple functionality, this works well, reducing load time and network calls. However, this method does not support advanced features like link context, changing filters, or AtomList views.

Data source filters¶

Data source filters are only applicable on views where a single row of data is shown. It is the responsibilty of the solution creator to engineer the data in a way that the filters build a query that will return only one row of data. If more than one row is returned, you may run into issues with unexpected results. In Debug mode, you will be alerted to this issue. In general, the best practice is to filter on a field that only contains unique values.

Filter value comparison is done using an equality test. Partial string comparison or number comparison is not supported.

Static¶

Use to set an initial static filter on a data_source.

Attributes:

type - “static”
data_source_field - the field in the data_source that the value will be checked against
value - The value that will be compared.

Example:

    {
      "type": "static",
      "data_source_field": "country",
      "value": "Belgium"
    }

This will add a country_eq=Belgium filter to the data_source query.

Primary reference¶

Use to set an initial filter on a non-primary data_source, based on a value in the primary data_source.

Attributes:

type - “primary_reference”
data_source_field - the field in the data_source that the value will be checked against
primary_field - The field in the primary data_source whose value will be used

Example:

    {
      "type": "primary_reference",
      "data_source_field": "country",
      "primary_field": "country"
    }

If the value of country in the primary dataset is “Belgium”, this will add a country_eq=Belgium filter to the data_source query.

Link context¶

Use to set an initial filter based on the context passed from an action.

Attributes:

type - “link_context”
data_source_field - the field in the data_source that the value will be checked against
context_name - The name of the context item whose value will be used
allLabel - If the value passed equals the allLabel, no filter will be applied. This allows you to specify that you want to show all values.
name - The name used for display. Currently, this is only used for with filter display.

Example:

    {
      "type": "link_context",
      "data_source_field": "country",
      "context_name": "country",
      "name": "Country"
    }

If the value of country from the context is “Belgium”, this will add a country_eq=Belgium filter to the data_source query.

Example¶

Putting this all together, here is an example data_sources for a view:

    "data_sources": [{
      "data_source": "companies",
      "primary": true
    },{
      "data_source": "countries",
      "filters": [{
        "type": "primary_reference",
        "primary_field": "country",
        "data_source_field": "name"
      }]
    }]

The view will use two data_sources, companies and countries.
companies is the primary data_source, and so it will load first.
After companies loads, it will make the value of its country field available for the country data_source.
The countries data_source will load the row where the name field equals the value of country from the companies data_source.

The Data Source Journey¶

Data Source Definitions¶

Attributes¶

Array attributes¶

Publisher Transformations¶

Special Data Types¶

Boolean¶

Null¶

Rename¶

Lower Case¶

Upper Case¶

Boolean¶

Split¶

Map Parse Integer¶

Map Parse Float¶

Map Parse Boolean¶

Map Parse String Array¶

Map Parse Integer Array¶

Map Parse Float Array¶

Map Parse Boolean Array¶

Data Source Usage in Views¶

Views With No Data Sources¶

Sharing Data Sources Across Views¶

Data source filters¶

Static¶

Primary reference¶

Link context¶

Example¶