Note

This article serves as a temporary procedure for adding fields to V2 Daybreak solutions. Currently, Aunalytics is transitioning all clients to V3 Daybreaks, which import their datamart schema from the Datamart object that is linked to that Daybreak in the config. As such, this procedure will no longer be needed once a client transitions to V3 and is intended only as a temporary guide to support our clients as they await their migration to V3 solutions.

Editing Daybreak V2 Configs

Daybreak™ is a web application created by Aunalytics for interacting with an analytical database (data mart) created by an Aunsight data pipeline. Daybreak was first released in July of 2020 for select clients. At the time, the Daybreak app configurations required extensive coding to produce the desired web app behavior for a given data mart schema. Since that time, the appearance of Datamarts as an Aunsight concept has allowed for a new kind of Daybreak configuration, Daybreak V3, that requires much less management: users simply point the config to a Datamart object and the Daybreak web app can derive all the information it needs to build the web app user interface from the Datamart's config.

Given the benefits of Daybreak V3 configurations, no new V2 Daybreaks will be implemented and all existing V2 applications will eventually be migrated to V3 based on the client's timeline. However, at present, we continue to support our existing V2 clients and therefore many product implementation engineers may occcasionally be tasked with updating a client's Daybreak V2 config to add support for a new field.

This document presents a condensed version of the information published at the time of Daybreak's release for creating new Daybreak apps. That information is still available for purely informational purposes, since no new Daybreaks should be created using the V2 config schema. New Daybreaks should be built from Datamarts and created using the V3 method.

Warning

Don't create new V2 Daybreaks. We implement all new clients on Daybreak V3.

Because our interaction with Daybreak V2 apps is purely in a support/maintenance capacity, this article will focus on just one use case: adding a new field. If you are tasked with doing more extensive modification, please contact the data engineering manager or Product Technical Communications for reassignment or assistance in completing these tasks.

What are Daybreak V2 Configs?

Daybreak configs are JSON objects that are managed by Aunsight's lib-query service. Daybreak configs are used by various platform services and internal teams to do various things:

  • The Daybreak service uses them to configure the web app widgets like field and operation drop downs so that users can interact with the data mart in intelligent ways (e.g. a text field should not show numerical operators like greater than, enumerated fields should show allowed values as check boxes on filters, etc.)
  • The Query Service uses the config to validate and submit queries it recieves from the Daybreak app UI.
  • The Innovation Lab NLA team (ILT-NLA) uses Daybreak configs to interrogate the data mart structure in order to generate a model training dataset suitable for that data mart. Changes to the datamart can then "feed" the training of a new model. The ILT-NLA also stores weighting and synonym settings in the configs as part of the table_header attribute. PIEs/DEs should never try to edit the table_header object in any Daybreak config.

Daybreak configs are composed of various objects, but for the present purpose, the only object we will want to edit is the datamart object.

How to Edit Configs to Add A Field

Step 1: Fetch the Config from Aunsight

The first step in editing a config is to download a copy of the most recent config. This can be done via the toolbelt:

First, set the appropriate context:

au2 context set <ORG GUID> <PROJECT GUID>

Then, download the config for the most recent version of the config:

au2 daybreak version show -i <DAYBREAK ID> -v <VERSION NUMBER> > temp_config.json

The Daybreak organization and project GUIDs, as well as the Daybreak ID (not usually a GUID) and version number can all be discovered by navigating to the Daybreak tool in the Aunsight Web Interface.

When run, this command will output the JSON of the complete config. This should be piped to a temporary working file on your workstation (temp_config.json in this example).

Daybreak V2 configs contain a number of top-level objects. For the purpose of adding a new field or fields to the data model, we only need to examine the datamart object.

The datamart object contains a name and description of the datamart, as well as a tables attribute which is an array of table objects. To add a new field or fields to one of these tables, browse the JSON file in a text editor to find the desired table object you wish to edit.

A table object is an anonymous JSON object identified by its name attribute. You will also notice that the record attribute contains a GUID. This is the GUID of the Atlas record that stores that table's data. Take note of this as you may need to refer to the Atlas record schema for that dataset to understand what you need to add to the configuration.

Step 2: Determine the Source Table Format and State of the Library Cleanup Workflow

Daybreak configs rely on the underlying Atlas record to supply the data that Daybreak will serve up via the web app. These "finished" datasets are created at the end of the Daybreak data pipeline, which usually includes a final Library Cleanup dataflow. This dataflow uses field mapping and select and rename expressions to transform the input datasets into the naming conventions desired requested by the client for their datamart.

As such, it is important to check these mapping expressions in the cleanup dataflow before making changes to the datamart config. To do this, you can either examine the dataflow in the builder and note the field names, or you can run the dataflow and examine the schema of the output dataset. If there is a mistake or mismatch in field names, the Daybreak config will fail. In some cases, a teammate may not have updated the mappings and the field may not be propagated through to the finished dataset linked to the table you are building the Daybreak datamart from. In these cases, it will be necessary to modify the cleanup dataflow accordingly to ensure that all required fields are appropriately mapped.

Warning

Always check the Library Cleanup dataflow for the pipeline you are creating the config for. It may be necessary to add field mappings if the field you need to add is not being written to the output dataset.

Step 3: Edit the datamart object

Once you have confirmed the presence of the new fields in the dataset used as the source for the datamart table, you can add or update fields by creating or updating a the column object in the columns array for that table. To add a new field, create a new object enclosed in the columns array with the following attributes:

{
            "name": "MyField",
            "description": "My Very Special Custom Field",
            "type": "string"

}

In this object, name is the name of the field as it exists in the Atlas record schema and description is a human-readable description that is presented to users in the web app UI.

The type refers to the data type. This information can be inferred from the schema of the source record for the table. There are three data types: - integer - float - string

String types can be further classified by subtype via the format attribute. format types are:

  • "format": "date-time"
  • "format": "currency"

General purpose strings do not need a format specified. For date-time formats, a further dateFormat attribute should be specified with the format of the date. Usually this is "dateFormat": "yyyy-MM-dd HH:mm:ss" (e.g. a datetime format like 2021-10-06 18:57:21).

There are a few optional attributes that may be necessary: - filterable: Determines whether or not a field can be used in a filter. Usually you will want to set this to true. - link: A link indicates a foreign key relationship to the table. link attributes are objects of the form {"table": "AnotherTable", "column": "Foreign_Key_ID"}.

Step 4: Upload the New Config to Aunsight

To upload the config file you have created into Aunsight, run the following Toolbelt command:

au2 daybreak update -i <DAYBREAK ID> -f temp_config.json

Note

Change the --file/-f argument to the path where your updated config is stored.

Step 5: Log into the Daybreak Webapp to Test the New Configuration

Before closing the ticket, it is important to ensure that the new configuration file works. Log into Daybreak and select the context for your project. Daybreak will load your config and attempt to create the web app according to its instructions. There may be an error displayed at loading, but if not, continue into the Data Builder and check each field that was added or changed to make sure it is being displayed properly. Finally, create and run a query on the newly changed fields to ensure that the underlying Atlas record is able to be queried based on the Daybreak datamart configuration.

If all these tests pass, you have succesfully updated your Daybreak V2 webapp config!