Release Overview: April 2021¶
Aunalytics® is excited to announce the April 2021 Daybreak™ release to our clients. This release will provide clients with model and site enhancement information along with any fixes to existing functionality we have included.
Daybreak¶
Daybreak™ for Financial Services 3.0¶
Daybreak for Financial Services 3.0 (DFS 3.0) represents a new generation of Daybreak data model for clients in the financial services industry. Based on insights gleaned from the experiences of numerous client banks and credit unions, Aunalytics has identified numerous improvements that will now be delivered to future and existing clients as Daybreak for Financial Services 3.0.
At its core, DFS 3.0 starts with a next-generation datamart core to support a more streamlined datamart structure and delivery pipeline. Whereas previous versions of Daybreak for Financial Services relied on a monolithically structured datamart, 3.0 has streamlined the logic used to build derived fields into a tiered structure so that datamarts can be implemented faster and customizations handled more dynamically for clients.
In addition to the data model, DFS 3.0 takes advantage of next-generation developments in the Aunsight™ Data Platform to deliver faster and more resilient datamarts. Improvements such as the microprocessing of client data and delivery via storage infrastructure built for ultra-fast query response times adds efficiency and performance to our next-generation data model.
Finally, Aunalytics operations and product documentation teams are working behind the scenes to provide documented standard processes for client implementation that will help educate existing clients on the steps needed to migrate to this version and enable our teams to deliver these upgrades faster and more reliably.
Datamart Restructuring¶
An improved structure for datamarts enables us to deliver the core analytical database tables to new clients faster while enabling clients to build out additional custom fields as desired.
Tier Levels¶
One of the major changes to Daybreak for Financial Services 3.0 is the introduction of a tiered system for organizing fields and tables in the datamart:
| Tier | Purpose | Requirement | Example fields |
|---|---|---|---|
| Tier 0 | Required System fields | Required | CustomerID |
| Tier 1 | Foundational fields | Required | DateofBirth, TaxpayerID, AddressKey (foreign key) |
| Tier 2 | Enhancing Fields | Recommended | Gender, Email, CurrentCreditScore |
| Tier 3 | Customer selected fields | Elective | HasBillPay, HasEStatements, CheckingAccountBalance |
The purpose of this tiering system is to decrease the time to implementation of a standard set of tables common across all clients. Once the initial two tiers are complete, customers can immediately access a basic datamart featuring additional data from tier 2 as that becomes available. Tier 3 enables clients to request custom fields for ad-hoc or client-specific use cases or to add Smart Features on a subscription basis.
By decoupling the layers of the data model, Daybreak for Financial services offers faster time delivery to clients and enhanced ability to provide custom services without complicating the implementation of the foundational datamart structure.
New Tables¶
As part of the restructuring of datamarts into a tiered system, Daybreak for Financial Services will be adding new tables that enhance the relational structure of datamarts:
- Products: Information on the products offered by the client
- Services: Information about the different services a customer has subscribed to
-
Milestones: Information about the stages of the loan approval process
-
Addresses: Customer address and geolocation data
Many of these new tables serve to normalize relationships that had not always been clearly reflected in the 2.0 datamart versions. For example, customers in the previous model were limited to one address, which created various problems when the same customer had accounts listed under different addresses. The new structure makes it easier to accomodate customers who may have a business account listed under one address, and personal accounts under a home address.
Datamart Microprocessing¶
Daybreak for Financial Services 3.0 enables an evolutionary leap in the processing of data for Daybreak: datamart micro-processing. In past versions of the Daybreak for Financial Services model, data is refreshed nightly by processing complete datasets, including fields whose values change infrequently. Datamart microprocessing enables much faster and more frequent datamart refreshes by only building where source data has changed.
Datamart microprocessing leverages transactional workflows in Aunsight™ Golden Record to transfer new or updated data from a source in the form of a dataset log. These small chunks can then be processed on their own and the results integrated into the datamart, greatly reducing unnecessary resource load since only new data is processed in every batch.
In addition to greatly reduced resource footprint for cost and energy savings, microprocessing opens up the possibility of refreshing Daybreak datamarts multiple times in a day. If implemented for clients, this approach would offer a more dynamic experience as datamarts would be accurate to within hours of when their source data has changed.
Point-in-Time Snapshots¶
Daybreak for Financial Services 3.0 also includes datamart snapshots, a new database dimension providing point-in-time information about datamart fields. Snapshots provide an extra dimension to the Daybreak for Financial Services experience, allowing users to retrieve trend data about a field, such as credit score or account balances. In addition to exposing historical data to users in search of trends, the Innovation Lab data science team will have access to historical data for exploration and use in developing new SmartFeatures built on data trends over time.
Dynamic Subscription to Smart Features™¶
Clients on the 3.0 data model will now be able to dynamically subscribe to machine learning generated Smart Features as a part of their tiered datamart. Previously, delivery of new Smart Features required an extensive overhaul of the data pipeline used to generate that datamart. Thanks to its tiered data model, Daybreak for Financial Services 3.0 enables the activation of Smart Features by simply changing the configuration objects for a datamart pipeline.
Dynamic subscription enables new business models for delivering industry intelligent data points as a service. Clients gain the ability to personalize their Daybreak experience by subscribing to the individual machine learning scores and insights they want in their datamart.
Daybreak for Financial Services 2.2 Patch¶
Daybreak for Financial Services version 2 clients have a new update to their data model available to correct a change in the way the transaction summary table is built. Based on changes to Accounts table, the DFS 2.2 patch will change the foreign key for the Transactions summary to point to a uniquely identifying field, AccountPrimaryKey, rather than the previous AccountID which was a non-unique identifier in many client data sources. This change will resolve certain issues for some banking clients whose core systems did not provide unique account IDs (account numbers) for all sub-accounts (e.g. checking versus savings).
Download Insights¶
Last month Aunalytics released Insights, graphical representations of query results in the Daybreak Data Builder. At that time, users could create Insights in one of four types, save them to their query's Insights dashboard and share the query and its Insights with other Daybreak users in the app. This month, users can download the visualizations as PNG or JPG files for inclusion in other applications, or data (CSV) files to create additional visualizations manually. This allows users to import Daybreak Insights into documents and presentations using other tools directly from the app.
Read more about how to download Insights here.
is one of/is not one of Operator for Structured Queries¶
Many Daybreak datamart fields hold enumerated values; in other words, a field can only have one of a limited set of possible values such as ProductType being one of "Mortgage", "Checking," "Savings," etc. To support new query capabilities on such fields, Daybreak now offers is one of and is not one of operators which allow a user to select a subset of values that they would like to find.
For example, if a user wants to select mortgage or HELOC accounts, they can now use the is one of operator on the ProductType field and check the "Mortgage" and "HELOC" boxes from the operands field.
is not one of works similar to is one of but in reverse: it will exclude fields that are checked and return all other results.
Direct Notifications¶
The April release features in-app notifications as a new Aunsight platform component. Daybreak users will experience this as an in-app notification area that provides updates about changes to queries that are shared with them. By default, users will be automatically subscribed to notifications for all queries shared with them. Notifications will enable users to be updated when these queries are changed by their creators.
Notifications appear as an extension of the webapp in the lower left corner of the screen above the account/log out information. When a user visits this part of the app, they will see the last thirty days of activity in a sorted stream of notifications about changes to queries shared with them. Clicking on a message will mark it as read, or users can mark all as read.
Insights for Natural Language Answers¶
Last month Aunalytics released Insights, graphical representations of query results in the Daybreak Data Builder. At that time, users could create Insights in one of four types, save them to their query's Insights dashboard and share the query and its Insights with other Daybreak users in the app. This month, Daybreak Natural Language Answers will feature automatic Insights, graphical representations of the results generated by a natural language question. At present, Daybreak Natural Language Answers will show a summary Insight displaying the number of records in the result set returned by the query. In the future, Insights for Natural Language Answers could be trained to display different visualizations, such as a breakdown of customers by generation (“Show me a list of customers with checking accounts by generation”) or zip code (“Show me mortgage accounts by ZIP code”). Insights for Natural Language Answers lays the groundwork for applications of machine learning to provide answers to clients using a simple, natural language interface.
Aunsight¶
Direct Notifications¶
As with Daybreak, the Aunsight web interface now features notifications as a platform concept. As in Daybreak, certain platform events can be configured to generate notification objects that can be viewed using the new web UI notifications interface.
Aunsight notifications appear in two places. The first is the notifications tool, which is accessed from the User Dashboard (the landing page when a user first logs in). The second location is a notifications quicklink in the upper right corner of the screen near the documentation and service desk links. This quicklink is visible from whatever context a user is working in, and so provides instant visual alerting when new notifications are received.
New Connectors: SmartyStreet and Character Cleaner¶
This month Aunsight premieres two new connectors: SmartyStreet and Character Cleaner.
The SmartStreet geolocation database is used in many Daybreak datamarts to retrieve geolocation information (GPS coordinates and distance calculations) from street addresses using the SmartyStreet API. To further streamline the retrieval of this data for client solutions, Aunsight now has a SmartStreet connector. Engineers can run this connector with a dataset of street addresses in a standard format and recieve a result dataset with geolocation data for those addresses. By using a connector, engineers can now get geolocation data without having to understand or interact with the SmartyStreet API or write custom processes to obtain their data for Daybreak datamarts.
The Character Cleaner connector is a utility process to aid solution engineers with a common use case: finding and replacing certain characters using regular expressions. Previously, data engineers would write custom processes to use the Unix sed utility to perform these transformations. Now, engineers can feed a raw dataset into the Character Cleaner connector with configuration options to find and remove or replace undesired characters (e.g. removing quotation marks like '' or '"') and write the output to a cleaned dataset record. This will provide a valuable tool to increase solution developer productivity.
Job Tracing V1 Framework¶
This release previews the first version of a job tracing framework for Aunsight workloads. The job tracing API provides enhanced capabilities for tracing related Aunsight jobs within a workload. In addition to the platform API changes, this initial release of job tracing will add a trace panel to the jobs page in the Aunsight web interface enabling users to see links to the job showing the root job (usually a workflow), the immediate parent job (another workflow or other component) and any child jobs (jobs kicked off by the present job) as well as the job type and state. This feature provides an easier way for operations to investigate pipeline failures and isolate the root cause of failing components quickly and easily, enabling fast responses to failures in production solutions.
Dataflow Operator Improvements¶
This month a number of new dataflow operators and expressions were created and certain changes were made to existing operators to make them easier to use.
| Issue | Type | Expression/Operator | Purpose |
|---|---|---|---|
| AUN-9784 | New Operator | NonEqui.Join |
This operator allows user to perform a non-equal join operation (a "Theta join") using comparison rather than equality as the condition. |
| AUN-9758 | New Operator | Median |
This operator computes a statistical median from a collection. |
| AUN-12188 | New Expression | CastToBigdecimal |
Casting a double type to chararray transformed the number to scientific notation, which prevented the field from being used as an operand for arithmetical operations. CastToBigdecimal casts numbers to a string (chararray) type, but preserves the decimal notation needed to perform arithmetic. |
| AUN-12617 | New Operator | join.InnerCompound |
Users can now perform joins multiple fields without creating a compound key field |
| AUN-13969 | Updated Operator | A confusing parameter name ('alias') has been changed to 'prefix.' | |
| AUN-14525 | New Expression | ToDateFormatTimezoneCorrected |
A new operator to converts datetime strings to objects with correctly adjusted DST |
Aunsight Golden Record¶
Snowflake Plugin for Transactional Workflows¶
Last month, Aunsight Golden Record released Transactional Workflows, a new capability that can move large datasets from a source database to the datastore of their choice while bypassing the Mapping, Matching, and Merging configuration steps. Initially, this feature came with plugins for performing bulk transfers of data with Amazon S3 and the Aunsight platform. This month, Aunsight Golden record has released another plugin for the Snowflake cloud analytics platform. This feature adds additional capabilities for transactional workfldaows that will enable Aunsight Golden Record to provide enhanced support for customers using that platform.
Release Contents¶
| Issue ID | Description |
|---|---|
| ILT-155 | Data Download by Query |
| DATAINT-522 | Update TX shapes to not use MDM provider in UI |
| DATAINT-515 | Refactor Job Scheduling Logic |
| DATAINT-509 | TXWF - Destination timing options |
| DATAINT-507 | TXWF Statistic Improvements |
| DATAINT-487 | Update UI to validate schemas in draft if connection settings is changed |
| DATAINT-480 | AU TXWF Date format standardization (8601) |
| DATAINT-475 | Improved Password Reset UX |
| DATAINT-447 | Text Area version storage |
| DATAINT-180 | AWS Redshift Transactional Job |
| DATAINT-38 | Aon Mongo Accessibility |
| DATAINT-17 | I/O/R Error Message UX |
| AUN-14840 | Exasol Connector Update: Make DROP transaction safe |
| AUN-14774 | Datamarts: Show names for created by and updated by |
| AUN-14731 | Implement design system font in remaining UIs |
| AUN-14661 | Model service: Make "Publish" button reactive |
| AUN-14657 | Add backend support for tag validation throughout Aunsight |
| AUN-14646 | Add option fields to copy job (update_schema, etc) |
| AUN-14645 | "Move" mode for dispatcher copy |
| AUN-14643 | Daybreak NLP: Display warning when value is empty in enum field |
| AUN-14542 | Design System: Update font throught Aunsight to match the one in Daybreak |
| AUN-14493 | Using entityMart to get links in wf builder, fixing view mode and resource link bugs |
| AUN-14489 | Data Lab: Create more timeframes for resource usage |
| AUN-14488 | Data Lab: Ability to create time frame for resource usage |
| AUN-14486 | Data Lab: Expose more data points in activity history |
| AUN-14461 | Execute functionality for AuQL scripts using lib-aunsight-py |
| AUN-14415 | Create "load datamart" workflow component |
| AUN-14047 | Add restrictions on custom id field for characters and length |
| AUN-13779 | Encompass Refactor |
| AUN-13463 | Processes: Update runtimes automatically when toolbelt or lib-aunsight is updated |
| AUN-13417 | Add additional fields to pull from Encompass |
| AUN-11964 | Data Lab Notebook: Add code snippet title to dialog |
| AUN-11730 | Toolbelt should not execute when invalid options are provided |
| AUN-6178 | Toolbelt create/update secret add field options |
Bug Fixes¶
| Issue ID | Description |
|---|---|
| DATAINT-526 | UI Session Expiration not being enforced |
| DATAINT-516 | [BUG] UI Not properly filtering out connections from Replication and Outputs |
| DATAINT-490 | Advnced tab disappears when editing property |
| DATAINT-485 | Profiling Suggestions Missing |
| DATAINT-482 | UI calling plugin twice for the same request |
| DATAINT-477 | Profiling UI Hotfix |
| DATAINT-463 | [BUG] - Merge Rule Dropdown weird behavior when > 1 rule exists |
| DATAINT-461 | Publish modal shows different TXWF source names on editing table name |
| DATAINT-460 | Ensure resources are cleaned up when draft is deleted |
| DATAINT-456 | AuGR Agent leaving rogue plugin processes running |
| DATAINT-152 | Scheduled Read Jobs getting stuck / not running correctly |
| DATAINT-20 | [PROD-ERROR] MDM Activities getting cancelled and throwing exception across tenants |
| AUN-14864 | Changing the tables port type component for load datamart component |
| AUN-14812 | DF Parquet loader ignores load schema in favor of record's internal schema |
| AUN-14770 | NLP Feedback breaks if NLP model URL does not end with / |
| AUN-14736 | Sonarbug fix for Expression builder |
| AUN-14719 | Query tool queries fail if they end with comment |
| AUN-14628 | Daybreak-Place Holder content for "in the range of, special" needs different formatting |
| AUN-14589 | Workflows - Notification type for roles/members not working in Run WF from modify view |
| AUN-14532 | fixing missing column cause out of bound error |
| AUN-14426 | GraphQL WF References should point to specific job types not TrackerJob |
| AUN-14384 | Losing information while saving WF builder for the exact same state |
| AUN-13849 | Dataflow: Undefined parameter causes launch to hang at "Preparing..." |
| AUN-13824 | Aunsight de-duplication not working |
| AUN-13723 | Dataflow Failure - ERROR 1025 Invalid field projection. Projected field does not exist in schema: |
| AUN-13066 | Dataflow: Improve error message when a record is being used multiple times (for store). |
| AUN-8771 | Lib-Py: dataflow upload loses parameters |