Product Updates - June 29, 2023

Overview of Changes

Pipeline Builder: Dynamic Operators

All pipeline operators except RawQ Topics now support dynamic syntax that can interpret arbitrary tables such as data dictionary tables, lookup tables, and the input schema. 

NOTE: Support for RawQ Topic operators will be added in the next release.

Using a subset of the Jinja template framework, operators can now loop through tables, such as data dictionaries, lookup tables, and tables of constants.

You can use this functionality to replace Autodraft by instead introspecting a table or the input schema. For example, you can configure an Aggregate operator to use a certain function that is based on the data type of every field in its input schema, or you can use a data dictionary table to populate a Parse JSON operator on a data source that is known to have sparse fields.

Dynamic operators can also reference lookup tables and tables of constants, which you can use to set global variables. For example, if you have a constant value that is used in calculations in multiple Apply Expression operators, you can read the constant value from a table, which allows you to propagate a change to the value of that constant across every operator that references it.

To use dynamic syntax, switch an operator from the JSON parser to the Jinja parser in the drop-down window in the Configuration pane. 

NOTE: Operators that use the Jinja parser cannot use Autodraft, and they do not currently support error highlighting.

You add and update tables from the Data Dictionary Tables dev page. After configuring an operator with dynamic syntax, you can resolve the loop and view the configuration that the Jinja syntax creates using the Expand Pipeline dev page.

For more information, including examples, click the information (i) icon next to the Jinja parser option.

Updates/Fixes

• Expressions now support null array literals such as:

(Text[]) null