Build Your Own Template

Why Should I Build a Template?

Creating customizable and reusable templates from existing pipelines gives you more control over your workflow, and helps you become more efficient as you scale. Templates significantly improve pipeline maintenance and readability of a Pipeline by:

• Making it possible to take logically equivalent operator chains that have been copied and pasted from one line or machine to another and manage them centrally.
• Cleaning up the visual congestion on the canvas, turning multiple operators into a single template.

Finding Use Cases for Templates 

Look for cases of logically equivalent operators that are reused.

Step 1: 

Open your pipeline and identify any areas of logic that have been reused.

Upon inspection, you may notice that several facilities and lines use similar logic for their downtime data.

Step 2: 

Identify any differences between the instances of the logic. You can see that the topic varies for each Facility/Line combination.

Step 3: 

Determine what needs to be configurable for each instance of the template. In this case, it's just a different topic name for each instance. You're going to want to edit that operator so it uses dynamic JINJA syntax once you start the template creation process.  

Note: Even though each instance is using the exact same fields, you should make that a dynamic operator and use a data dictionary to reference the key, field, and type so it is easier to maintain long-term.

Building a Template

After you've found a good use case for templates, you can create your first template.

Step 1: 

Make a copy of your workspace and give it a new name.

Step 2: 

Open up the workspace and navigate to the area of your pipeline that you want to use a template for.

Step 3: 

There are two ways to create your template.

You can either Shift+ select multiple operators or Shift+ drag to select multiple operators.

Alternatively, you can right-click an operator and then select the option to create a template from it.

Step 4: 

Click Create a Template.

Building a Template Instance

A template is reusable across a number of instances.  To create your first template, start by creating the first instance.

Step 1: 

Select all the operators that you need for your workflow, and then click Create a Template.

Step 2: 

You are presented with the configuration settings for your new template instance. Name your template instance accordingly and keep in mind that each instance of the template has its own name.

Step 3: 

Since you have already identified what you want to be configurable at a template instance level (in this case, your topic), set that configuration now. Name your only parameter 'template topic' and set it to an empty string for now.

Step 4: 

The operator that you need change is the RawQ Topic operator, as each template instance has a different topic identifier. Select the RawQ operator within the template instance.

Step 5: 

You need any operators that have template-driven configuration to be dynamic. Switch the operator from JSON to Jinja. If you're unfamiliar with Jinja, refer to the info icon for dynamic operators for assistance.

Step 6: 

Reference your template instance options in the RawQ Topic operator. Cut the current value for the Topic name out and set it as your value in your template instance.

Step 7: 

Go back to your operator and reference the 'options', which are part of your template instance options. Use the specific option 'template topic' from those options.

Step 8: 

You can either cancel your template creation or preview the results for your template instance. However, if you want to make this reusable elsewhere, finalize your template instance as a template.

Making a Template Instance Reusable

After you click Finalize Template, you're close to making it reusable.

This guide walks you through the process of finalizing and customizing a template, using a downtime template as an example.

Step 1: 

After you click Finalize Template, the first step is to change the template name. This is the actual template name, not the instance, and it does appear in the Operator Library. Rename the template according to its function. In this example, name it Downtime Template.

Step 2: 

Provide a template description. This appears in the operator drawer. In this example, the description is Template to handle my downtime data.

Step 3: 

Go into the default options, which are used as the default configuration when you drag a new instance of the template onto the canvas. Set the default options according to your needs. In this case, the example value is a fine default.

Step 4: 

Click the Connections tab. 

Because this template has multiple output connections, you can name these for clarity. In this example, the first connection is named 'end down' and the second 'start down', corresponding to the internal operators they come from.

Step 5: 

If you want, you can create your own template documentation to help other users understand how to use it. Other users can access this documentation by selecting the info icon for the template.

Step 6: 

Save your template by clicking Save Template.

After you save the template, it appears updated on the canvas, marked in blue.

Step 7: 

In the operator library, you can now find the downtime template with a quick description available that is available when you hover over it.

The connections have been named as well and are visible on hover.

If you select the template instance itself and go to the info icon for that template instance, you see the Downtime Template's documentation.

Step 8: 

If you want, you can collapse this on the canvas for a more compact and readable view. Double-select the expanded template or select the arrow icons.

Replacing Other Operators With a Template

This guide walks you through the process of using a template to replace a logically equivalent set of operations.

Step 1: 

Navigate to the operator library and select the newly created downtime template. Drag this template onto the canvas.

Step 2: 

Before removing anything else, go into the RawQ Topic that needs to be added to the new template instance and copy the topic name.

Step 3: 

Replace the default template value with the value from the RawQ Topic that you are going to replace.

Step 4: 

Instead of highlighting to create a new template instance, highlight the operators that you want to replace, and then delete them.

Step 5: 

After you configure your downtime template instance, you can connect the template to the downstream merge operator.

Copying Templates Between Workspaces

While templates are workspace-specific, when a workspace is copied, it will include all of the parent templates.  Additionally, if you want to use a template from another workspace, you can copy and paste it into your workspace.  This guide will walk you though the process of moving templates between workspaces using keyboard shortcuts and copy-paste function.

Step 1: After creating a template in your development workspace, select the created template. Use keyboard shortcuts to copy it. Use Command + C (for Mac) or Control + C (for Windows).

Step 2: Navigate to the workspace where the template needs to be added.

Step 3: Use Command + V (for Mac) or Control + V (for Windows) to paste the template. The pasted template appears and is automatically added to the operator library.

By following these steps, it's easy to move templates between workspaces using keyboard shortcuts and the copy-paste function.

Note: The pasted location is based on the location from the "copy from" workspace and pasted templates might not be immediately visible based on zoom/ view area of the canvas.

FAQ

Why should I use templates?

If you have logically equivalent operations that have minor changes between instances, we can manage all of those centrally using templates. This can save time when you need to update something that lives in many different places. For example if you need to make a change related to timestamps, rather than making that change 50 times, using templates you only have to make the change once.  

What is the best place for me to use a template?

Templates are best used where you have logically equivalent operations that are occurring.

Once I’ve saved a template where can I find it?

Saved templates can be found in the operator library.

Are templates workspace specific?

Yes, templates are workspace specific. However, you can copy and paste templates between workspaces.

What are some best practices when using templates?

1. When modifying a template rename your default options.

2. Update the documentation section of the finalize template flow to prevent confusion.

These options can be found in the finalize template flow.