Dynamic Operators

Dynamic Operators add automation to pipeline development and maintenance by linking the configuration to sources such as Data Dictionary tables, tables of constants, and the operator’s input schema.

Dynamic Operators use a subset of the Jinja language syntax, plus a few custom functions. Sight Machine supports general structures like for and if / elif / else, but not those that are specific to HTML editing, such as include. Also supported are comments, which are delineated by {# … #}.

Creating New Dynamic Operators

All operators support dynamic configuration. To add an operator as dynamic, add the new operator to the canvas and change the syntax to Jinja. Most operators contain a default Jinja configuration to get you started.

Migrating to Dynamic Operators

For operators that you have already configured in JSON, you can always switch the syntax to Jinja. By default, the existing JSON configuration is copied to the Jinja configuration.

Error Handling

Jinja configuration includes error messaging for syntax-related issues and cases in which non-existent attributes are referenced.

For example, if you incorrectly reference the attribute 'nam' instead of 'name', you receive an error like the following:

In some cases, you receive an additional error with the token that triggered the error.

Custom Jinja Functions

Custom functions include:

• try_lookup: Returns the first row that matches the criteria, or null if there is no match.

• lookup: Returns the first row that matches the criteria, or throws an error if there is no match.

• filter: Returns all rows that match the criteria.

• json: Converts the value of a Jinja expression into valid JSON.

• literal: Converts the value of a non-null Jinja expression into a constant value that can be used in an Apply Expression or Filter by Expression operator.

For try_lookup, lookup, and filter, the first argument is the name of the table to filter. After that, the arguments appear in pairs: the even arguments are column names, and the odd arguments are values. Therefore, to get all of the rows from the table my_dictionary in which the column field_type has the value datetime, you would use the following:

rows = filter(tables['my_dictionary'], 'field_type', 'datetime')

About the json Function

The json() function converts the value of a Jinja expression into valid JSON and should be used instead of quoting whenever a JSON option is populated from Jinja. The table below demonstrates how to write a robust dynamic operator configuration:

{% for field in inputs['in'] %}
{
  "from_field": {{ json(field['name']) }},
  "to_field": {{ json(field['name']) }},
  "function":
    {% if row['function'] != null %}
      {{ json(row['function']) }}
    {% else %}
      "last"
    {% endif %}
},
{% endfor %}

{% for field in inputs['in'] %}
{
  "from_field": "{{ field['name'] }}",
  "to_field": "{{ field['name'] }}",
  "function":
    {% if row['function'] != null %}
      "{{ row['function'] }}"
    {% else %}
      "last"
    {% endif %}
},
{% endfor %}

If the json() function is not used, some innocuous-looking templates may fail when special characters are involved. For example, if you have a field named 3" Gap Deviation, and you quote the field name without using the json() function, it produces invalid JSON. Applying json() produces the correct JSON output, as demonstrated in the table below.

The text highlighted in red is not valid JSON, and causes JSON parsing to fail.

Another benefit of using json() is that you do not need to worry about quoting. The output is formatted based on the data type of the argument to json(). Examples of a few different data types are included in the table below.

About the literal Function

Like the json() function, the literal() function converts the value of a non-null Jinja expression into a constant value that can be used in an Apply Expression or Filter by Expression operator. It applies whatever formatting is necessary to produce a valid expression, and then applies any additional formatting that is required to include the expression in a JSON string.

The literal() function should be used whenever a field name or constant in an expression is generated from Jinja. For example, a field named 10' Pipe Flow expands to an invalid expression unless the literal() function is applied.

The literal() function always produces code that evaluates to the same value as its argument. The table below shows the results of applying this function to values of different data types.

Note: The literal() function cannot handle nested objects like arrays, since the Operator Expressions language does not yet support manipulating complex objects.