Chains 103: Nodes

Introduction

As we discussed in our Chains 101: Core Components article, a Command is one of four types of nodes that can exist in a Chain.  In this article, we explore each node type in detail to highlight their role in building Chains that can be more resilient and efficient.  

‍

Commands

As we previously discussed, Commands are the primary nodes in most Chains.  They perform the majority of the data operations.  Consider an example where a Chain retrieves a file from an SFTP directory.  The Download File Command of the SFTP connector performs the operation of retrieving the file.

‍

There are six tabs available when configuring a Command.  Let’s review.

‍

Modify Command 

The modify command tab is required as this defines the core information the command needs to execute its actions.  As an aside, the term modify command can be a little misleading in my humble opinion.  You are not modifying the command or what it does.  Instead you are specifying the information the command needs to perform its task.  I would have labeled the tab Information or Parameters but alas I’m not a UX designer so what do I know.

‍

There are three distinct sets of information that gets specified on the modify command tab.

• Name & Description 
• Connection & Runner  
• Inputs 

‍

Name is perhaps one of the most overlooked configuration elements of a Command.  The name is a powerful way to help document the Chain configuration.  Using a descriptive name outlines the purpose of the command within the Chain.  We recommend that Commands use a convention where the name is the Command Name itself (e.g., Download File) with additional summary information appended.  For example, Download File - Headcount Roster provides insight that the file being downloaded is headcount information.  This is a useful way to build breadcrumbs (remember the Hansel & Gretel story from childhood) to help yourself and other users of the Chain quickly assess each action.  

‍

Like Name, Description is another incredibly powerful way to document the Chain in detail.  Unfortunately, we have noticed that a lot of Chains lack descriptions in the command configuration.  We strongly encourage customers to spend the time adding descriptions to help outline key information about the Command and its operation.  

‍

The Connection and Runner section of the Command is used to specify which Connection the Chain should use at the point of execution.  We discussed Connections in our Chains 101 article and Runners in our Chains 102 article.  

‍

The Inputs section of the Command is where we specify the information that the command needs to perform its action.  For example, we would specify the path to the file in the SFTP server in the File input of the Download File command.  

‍

Notifications

Each command can be configured to send one or more email notifications depending on the command execution result - success, warning, error, paused.  Notifications can also be configured to be sent regardless of the execution result. 

‍

Notifications are optional but a useful way to immediately be notified when a critical step in the Chain fails or simply completes.  

‍

Skip Conditions

Skip conditions provide a mechanism to forego the execution of a particular command based on a set of criteria.  Let’s consider a simple file operation.  We want to delete a file before a new version of the file is downloaded to ensure that we are also processing the most recent version.  Our command to delete the file may include a Skip Condition that instructions the command to be skipped if the file is not found.  As part of the skip condition configuration, we can set the skipped command result.  In this example, the delete file command would be set to Success to allow the next action - file download - to proceed.  

‍

Like Notifications, Skip Conditions are optional but a powerful way to allow Chains to be designed with logic that accounts for different outcomes and scenarios.  

‍

Dynamic Outputs

In our Chains 101 article, we discussed the core concept of Inputs and Outputs.  Every Command generates at least one Output.  Dynamic Outputs provides the ability to create additional outputs to address integration requirements.  

‍

For example, let’s consider the Tabular Transformation Advanced Query command which generates a delimited file of values based on the query.  This is the default output of the command.  The first column of the output contains the fiscal year and the second column contains the fiscal reporting period.  Those values may be needed in subsequent operations of the Chain.  

You could add additional commands to the Chain such as Extract Value to retrieve those values from the Advanced Query output or you can simply create two Dynamic Outputs that retrieve these values from the output Advanced Query command.  

‍

Dynamic Outputs are one of the most useful features of Chains but are often overlooked.  We strongly suggest that you familiarize yourself with this functionality.  

‍

Error Actions

The Error Actions tab is another often overlooked feature.  By default, every command will run for 60 minutes (3600 seconds) before being automatically terminated by the platform.  This setting can be configured at an individual command level on the Error Actions tab.  

‍

There are times when this needs to be increased because a Command is expected to execute longer; however, it can also be used to reduce the timeout when it is known that an execution should not take longer than a set time and when it does, it means that there is a hung process.  By reducing the timeout on the command, the command can fail faster and hung processes can be identified and addressed in a more timely manner.  

‍

In addition to timeouts, retry attempts can be added to a command to force additional executions of the command in the event of a failure.  This adds resiliency to the Chain.  A pause between retry attempts can be specified as well.  The pause is specified in seconds and adds yet another layer of resiliency.  

‍

Finally, certain commands have built in intelligence that allows the command to ignore known errors that would otherwise render the command execution as failure and instead sets the command execution as success.  Consider an example where a Chain uses the Make Directory command to create an archive directory on the SFTP server.  Files are moved to this directory at the end of a Chain’s successful execution.  If this directory already exists (from a prior Chain run) the command would fail; however, the Make Directory command has an error condition to ignore when the directory already exists.  This simple feature can be incredibly useful for building resilient Chains.  

‍

We generally do not recommend the use of Output and Log interpretation as other functionality in the platform offers more robust and flexible functionality to achieve the goals intended by these features.  

‍

Iterators

Commands have the ability to be executed in a loop up to 1000 times.  While there are instances where iteration at the command level may be necessary, we generally recommend that iteration is defined as Command Group which is discussed later in this article.  

‍

Events

One way to think about Events is that they are common Chain operations.  Chains is a low/no code application; however, the concepts of coding that are fundamental to integration are supported by Events.  A conditional is similar to an If-Then statement.  Run Chain is like a function call.  Set Result Text is like echo or print.  

‍

Events are nodes in a Chain in the same way that Commands are.  They use inputs to define how they operate; however, most events do not generate an output.  For example, the Conditional Event uses inputs to evaluate the Conditional’s execution status - Success or Failure - but no output is generated that can be used in subsequent commands.  Instead link conditions, discussed in Chains 101, allow different Chain actions to be performed depending on the result of the Conditional.

‍

Events allow Chains to be built that are more dynamic, embed logic that accounts for different outcomes, and allows user interaction with the Chain during execution.  We strongly recommend learning the functionality of each Event.  

‍

Triggers

Triggers are a special type of function that can only be used in the starting node of a Chain.  They can be used to automatically start (invocation) a Chain execution.  There are two types of invocation triggers - File Events and HTTP Webhooks.  File Events are essentially file watchers and execute a Chain when activity is detected in a specified directory.  File Events can only be used with a GroundRunner.  

‍

HTTP Webhooks are the mechanism to allow 3rd party systems to execute a Chain.  Webhooks require the HTTP connector to be licensed and a valid connection to be created.  The use of webhooks has become much more prevalent in enterprise architecture where dependent systems can interact with one another as various processes are completed.  For example, consider a financial planning application such as Oracle Planning and Budgeting Cloud or Anaplan.  When the budget or forecast cycle is complete, a webhook could invoke a Chain that extracts the data from the planning application and loads it to Workiva to support Budget/Forecast versus Actual reporting.

‍

The Runtime Inputs Trigger allows a Chain to accept values from another Chain or process that executes the Chain.  This is one of the most important features of Chains.  Runtime inputs allow Chains to be reused across different processes by allowing a common integration pattern to be built with values unique to a specific integration to be specified at the time when the Chain is run.  

‍

If your organization has copies of Chains that are identical except certain input values for Commands, Runtime Inputs can be used to consolidate.  By consolidating Chains, an organization not only reduces maintenance efforts but ensures consistency in how data is managed.  

‍

Command Groups

Command Groups have two functions - organizing nodes and iteration (looping) operations.  Chains with a significant number of operations can sometimes feel unwieldy.  Adding a Command Group allows a Chain builder to bucket together nodes that are used to achieve a specific task within the Chain.  

‍

Consider the typical stages of a Chain used for data integration purposes - data extraction, transformation, and loading.  There may be a number of steps needed to perform the transformation and make the data ready to load to the target system.  A Command Group could be used to visually organize these nodes.  It should be noted, a Command Group used for organization purposes has no impact on how the Chain performs its actions.  

‍

Conversely, when a Command Group is configured to enable iteration, there is a material impact on how the Chain operates.  Configuring a Command Group for iteration means that every node (Commands or Events) assigned to the Group will execute according to the loop defined by the Group.  

‍

Let’s consider a common scenario, an SFTP directory needs to be swept and all of the files found need to be loaded to Wdata.  Our first action is to get a list of files in the SFTP directory.  The List Files Command will exist outside of the Command Group for two reasons.  First, we don’t need to list the directory contents multiple times.  Second, and more importantly, we need to use the output of the command - the list of files - to define the loop.  

‍

Now that we have the list of files over which we need to loop (iterate), we can define the actions that need to be executed for each file found in the directory.  In this simple example, we simply download the file from the SFTP directory using the Download File command and then we run a Chain using the Run Chain Event to upload the data to the Wdata table.  Each of these nodes in the Command Group is run for every file found in the directory.  

‍

Let’s take a moment for a quick tangent here.  We described using the Run Chain Event to run our Chain that loads data to Wdata.  Did you know there is a Chain Template available from Workiva to load data to Wdata?  

‍

Back to our regularly scheduled program.  Command Groups currently execute sequentially meaning that if we are iterating over 5 files in the directory, the Commands in the group will be executed for the first file, then again for the 2nd, 3rd, 4th, and 5th file.  There has been discussion of adding parallel iteration to Command Groups but as of this article’s publish date, there is no definitive date for this enhancement.  

‍

One final note about Command Groups.  Command Groups cannot be added to existing Command Groups.  This is important to know especially if a loop is required within a loop.  If this is a requirement of an integration then the Run Chain Event needs to be used to call a child Chain that has the second loop.  We encourage you to contact us if you need any assistance with an integration pattern such as this.  We are happy to help you think about the optimal design ensuring performance and scalability.  

‍

Summary

Whew, that was a lot!  We hope you learned something new by reading this article.  As you can see, there are a number of features within Commands, Events, Triggers, and Command Groups that empower you to build dynamic, flexible, and resilient Chains.  While a Chain with just a handful of Commands configured with just the required inputs will function, leveraging these additional features can elevate a Chain by adding intelligence.  

‍

If you are interested in learning more, please feel free to contact us.  We love showing customers the art of the possible.