Product Updates - August 18, 2022

Overview of Changes

Pipeline Merge Workflow

Pipeline Builder now supports a Merge Pipeline workflow that compares and combines two Pipelines. You can create a new merge draft from the menu on any Pipeline in the Pipeline Manager. 

If you navigate away from a partially completed merge, you can return to it on the Merges tab.

The merge first creates a draft where you can inspect and validate the merge contents before finishing the operation. Finishing a merge creates a new Pipeline draft that you can then deploy.

The merge produces the comparison based on automatically detecting the two Pipelines’ most recent common ancestor (i.e., a Pipeline they were both copied from in the past). The comparison is displayed as a series of color-coded operators in Graph Mode. If no common ancestor exists (i.e., the Pipelines are brand new or imported), the merge runs the comparison and shows every operator as either added, deleted, or changed.

You can also inspect individual operators and compare and edit their configurations. 

• If an operator has been added, deleted, or edited in only one Pipeline, the merge automatically determines which set of changes to keep. You can override this choice by editing the operator configuration or selecting Use This Version on the Comparison tab of the Edit pane. 
• If both Pipelines have edited the contents of a single operator, the merge renders this as a merge conflict, which introduces an invalid configuration. These conflicts are easy to spot because they have a red bar indicating the validation error. A validation error does not prevent you from finishing the merge operation, but you must fix the error before deploying the resulting Pipeline draft.

Important Notes on Merges

1. The Merge Pipeline feature makes two important, irreversible changes to the way that Pipelines are stored in Mongo.
   • First, the options object has been changed from a JSON object into a string. This string is converted into an object on the ETL side, if the string is valid JSON, but may contain invalid JSON due to user input or merge conflicts which are surfaced as validation errors.
   • Second, the inputs object has been changed into a list. Previously, inputs were stored as key-value pairs where the key was their label ("in", "0", etc.) and the value was the information necessary to determine where that input came from upstream. Now, inputs are a list of objects in which each input has the same upstream information as before, as well as a new name field, which is what now stores the label.
2. In addition, some existing scripts and workflows may involve the copying and pasting of full JSON Pipeline representations into the JSON editor on the Sight Machine platform. If you are using such a script or workflow, be aware that the software currently allows you to paste in a legacy-format Pipeline with no validation errors, and even allows you to deploy that Pipeline. However, the DAG editor does not know how to handle the legacy format, and the Pipeline that you pasted into the JSON editor will not be displayed on either the Draft or Deployed graph editors.
   IMPORTANT: In order for these Pipelines to be visible in the DAG, you must transform your Pipeline to the new format.
3. Regarding operator merge_status in merge drafts, merge statuses are computed with respect to the Common Ancestor. 
   • For instance, changed does not mean that the two Pipelines you are merging have different options for the changed operator in question; it means that one or both of them have changes from the Common Ancestor that did not result in a merge conflict. If an operator has conflicting changes, those are shown on the Configuration tab and the operator is marked with a validation error, as long as there are no other validation errors upstream that prevent ETL from validating the entire Pipeline.
   • Similarly, added does not mean that only one of the merging Pipelines added the operator (both might have).
   • And deleted does not mean that only one of the merging Pipelines deleted the operator (again, both might have).

Operator Co-Pilot: Status Visibility and Actions

The Operator Co-Pilot application now provides status visibility, including information on changes to Levers, Conditions, and Assets that are used in Recipes.

Operators can now receive Cookbook recommendations, even if one or more Levers, Outcomes, or Conditions are no longer available in the platform. In case a Data Field that is associated with a Product or all Levers in the Recipe are no longer available, the Co-Pilot results in an error message that requires you to update or create a new Cookbook.