Release Overview¶
Aunalytics is excited to announce the February 2021 Daybreak release to our clients. This release will provide clients with model and site enhancement information along with any fixes to existing functionality we have included.
Daybreak¶
Query Wizard: New “In the Range of” Operator¶
The Query Wizard in the Daybreak Data Builder now supports a new operator for dates: “In the range of.” This operator serves as a placeholder for relative dates calculated from the present date when the query is executed, allowing query creators to define a range of relative dates. For example, a query could search for accounts created in the week immediately preceding when the query is run:
The Query Wizard interface provides a number of predefined date ranges:
- Past ranges:
- Yesterday
- Last week/month/quarter/year
- The last seven/thirty days
- Present ranges:
- Today
- This week/month/quarter/year
- Future ranges:
- Tomorrow
- Next week/the next two weeks/next month/next quarter
In addition to the predefined ranges, users can select “special” from the drop-down menu and define their own ranges using three criteria. Possible special ranges include examples such as the first quarter of last year or the week five weeks into the future.
SQL Builder: Autocomplete functionality¶
The SQL Builder mode of Data Builder now has an autocomplete feature alongside syntax highlighting for assistance in writing SQL query code. The autocomplete feature presents a popup menu with suggestions matching the characters the user has typed. Suggestions are drawn from SQL reserved words (e.g. “SELECT”, “AVG”, or “JOIN”) as well the field names for the Daybreak datamart being queried. So for example, typing “Date” will bring up a list of suggestions beginning with the “DATE” keyword in SQL as well as the datamart field names “DateJoined” and “DateofBirth” from the Daybreak Customer Table. This feature makes writing SQL queries against the Daybreak datamarts much easier since users not need to rely as much on the datamart field names browser in the SQL editor’s schema panel.
Data Builder: Streamlined Interface Buttons¶
The Daybreak Data Builder app is undergoing minor interface changes.
The “New Query” button previously displayed a dropdown menu from which the user could choose the desired query type: Query Wizard or SQL Editor. This drop-down has been replaced by a set of icons for Query Wizard, and SQL editor. Clicking each icon will create a new query in that mode.
Similarly, the action button to close the current query has been renamed from “Cancel” to “Back.” Users will still be prompted to save any modifications before the current query is closed.
Aunsight™ Data Platform¶
Parquet Support¶
Aunsight Atlas records now support a new dataset format: Parquet. Parquet is a columnar dataset format developed by the Apache Software Foundation for Hadoop-based big-data platforms (https://parquet.apache.org/). Unlike CSV files commonly used by Aunsight’s Hadoop storage clusters to date, Parquet structures data internally as columns rather than rows. Accessing and storing data in this way enhances the ability to store and operate on subsets of columns in a table. More importantly, it also allows datasets to be copied directly between the Alluxio object acceleration engine used to store and process data in pipelines and Exasol, the relational database used to process queries from the Daybreak webapp.
In addition to improving the efficiency of Aunsight data pipelines, Parquet will be an important component of a new Aunsight concept: datamarts. Parquet datasets will be used as the basis of datamarts, Aunsight objects which define and manage the structure of data as it moves between Aunsight’s big data storage infrastructure and the relational database structures which Daybreak imposes on data in order to render its structure more familiar to data consumers.
Aunsight™ Golden Record¶
Transactional Workflows¶
Aunsight Golden Record now allows users to configure a Transactional Workflow that can move larger datasets from a source database to the datastore of their choice while bypassing the Mapping, Matching, and Merging configuration steps. This new type of workflow supports bulk data transfers to provide fast, cloud-native data transfer from client data sources into both Amazon S3 and the Aunsight platform.
Minor Improvements: Daybreak¶
| Ticket Number | Ticket Name | Business Value |
|---|---|---|
| AUN-14206 | Daybreak: Create tooltip for gear icon on query wizard | The gear icon in the query wizard now shows “Select columns to show in results” when users hover the cursor over it. |
| AUN-14379 | Daybreak: Add one-to-many to query wizard | Users can now join from the account or transaction table back to the customer table. |
Minor Improvements: Aunsight¶
| Ticket Number | Ticket Name | Business Value |
|---|---|---|
| AUN-11623 | Expression Builder refers to "Bag" but proper nomenclature is "Collection" | Twelve expressions were updated to refer to “Collections” rather than the deprecated term “bags.” |
| AUN-13251 | Dataflow: STRSPLIT Expression improvement | The third argument can now be a field or a static integer. |
| AUN-13770 | Sample rows operation should allow 100% (and at 100%, it should pass through all rows, not sample them) | The operation "Sample Rows" will now work at 100%, allowing dataflow engineers to use “Sample Rows” as a flow control gate. |
| AUN-13840 | Consistent labels across webapp for 'Updated' fields | Various field titles were updated to consistently use the term “updated at”. |
| AUN-14025 | Add undo/redo actions in the expression builder | User will be able to use undo and redo within the expression builder. |
| AUN-14033 | Cached Mode for CopyMulti in the workflow components | Workflow CopyMulti components now have an attribute field to enable cached mode as has already been the case for the Copy component. |
| AUN-14121 | Make runtime images easy to use and maintain in the registry | Packaged process runtimes are now pushed to the Aunsight Harbor registry and made available as base runtimes for other processes. |
| AUN-14354 | Secret Service: Convert password input into multiline text input field | The Secret Service Web UI now uses a multi-line input field allowing multi-line text input for secret content (e.g. for storing SSH keys). |
| AUN-14378 | Processes: Sync file selector and form validator with eachother | Users can now upload files with any contents when the selected runtime does not specify a main file format. |
Minor Improvements and Bug Fixes: Aunsight Golden Record¶
| Change Type | Business Value |
|---|---|
| Enhancement | Scheduled input jobs will now detect records that have been deleted at the source, and propagate those deletions to the appropriate Golden Records. |
| Enhancement | File Reader Plugin - Additional modes 'FTP' and 'SFTP' are now supported which allows users to ingest files directly from an FTP/SFTP server. |
| Enhancement | Oracle Plugin - Users can now bypass discovery to improve performance when connecting to extremely large databases while retaining the ability to create a custom schema input from the selected database. |
| Bug Fix | Improved error message display in the UI for large error messages including the ability to scroll which allows users to view the entire error message directly in the UI. |
| Bug Fix | Updated the Version Browser display to keep the Golden Record in the first position of displayed records when sorting (applies to both horizontal and vertical browsing modes.) |
| Bug Fix | Improved performance of deleted Version and Golden Records reflected in the UI. |
| Bug Fix | Input/Output Graphs on the Monitor page will now default to display the last 5 minutes of data to improve UI performance. |
| Bug Fix | Changing the display name of a Matching step will no longer require a full purge and reload in order to Preview/Publish. |
| Bug Fix | Improved recoverability for scheduled jobs that when they become stuck in a pending state. |
| Bug Fix | Improved reliability of download option for the Rejected Records report. |
Bug Fixes: Aunsight¶
| Ticket Number | Ticket Name |
|---|---|
| AUN-14523 | Activity subscriptions not working |
| AUN-14445 | Workflow name disappears when user in view/modify mode in the project context |
| AUN-14295 | Workflow Builder- multiselecting things twice is possible, causing multiple issues |
| AUN-14283 | Clean up GA event tracking |
| AUN-14248 | Webapp CopyButton - add `type="button"` to copy button |
| AUN-14220 | Jupyter cannot upload or save large files |
| AUN-14205 | Address CloneDeep of Attributes in Data Lab |
| AUN-14204 | Address CloneDeep of Attributes in Data Pipeline |
| AUN-14203 | Address CloneDeep of Attributes in Core Services |
| AUN-14065 | ReadProcess and ListMembers tests are taking unnecessarily long |
| AUN-13268 | Process does not update user task jobs with env vars set by k8s adapter |
| AUN-12834 | Select and Rename Operation not working as expected |