Building ETL Workflows

Suite
Customer Experience Employee Experience Strategy & Research
Product
Qualtrics
What's on this page

About Building ETL Workflows

The Qualtrics Workflows platform contains a series of tasks to assist in importing data from third-party destinations into Qualtrics or exporting data from Qualtrics to third-party destinations. These tasks follow the Extract, Transform, Load (ETL) framework. Using ETL tasks, you can create automated and scheduled workflows to bring data from third-party sources into Qualtrics as well as export data from Qualtrics to third-party destinations.

To create an ETL Workflow, you must create 1 or more extractor tasks and 1 or more loader tasks. You are only limited by the overall limit for tasks in 1 workflow.

Qtip: ETL tasks can take up 24 hours to execute. If the task doesn’t succeed within 24 hours, the task will fail.

Qtip: ETL workflows can only run 1 workflow at a time. Simultaneously running workflows may fail.

General Setup for Extractor and Loader Tasks

Qtip: For a general guide on using workflows, see Workflows basic overview.

From the stand-alone Workflows page or the Workflows tab in a project, click Create a workflow.
clicking create a workflow and choosing etl
clicking create a workflow and choosing etl
Select Extract, transform, and load (ETL) data.
ETL workflows usually run on a recurring schedule. Choose a schedule for your workflow. See Scheduled Workflows for more information about setting a workflow’s schedule.
choosing a schedule and clicking save
choosing a schedule and clicking save
Click Save.
Click Data source (ETL extractor) to choose an extractor task to use. See Available Extractor Tasks for a list of tasks you can use. You can add multiple extractors in 1 workflow.
setting up the extractor and loader and adding loops if needed
setting up the extractor and loader and adding loops if needed
If you’d like to transform data before you load it, click Data transformation. This step is optional. See Basic Transform Task for more details.
Click Add a data destination to choose a loader task to use. See Available Loader Tasks for a list of tasks you can use. You can add multiple loaders in 1 workflow.
This step is optional, but useful for alerting you if something goes wrong with your workflow. Go to Settings to set up workflow notifications to get notified if your workflow ever fails.
the settings tab and on/off toggle are located at the top of the page
the settings tab and on/off toggle are located at the top of the page
Don’t forget to turn your workflow on.

Qtip: After setting up your workflow, click Run Immediately to to test your ETL workflow and ensure it’s working.
 

the "run immediately" button at the top of the page
the "run immediately" button at the top of the page

Qtip: You can add additional tasks to your ETL workflows. For example, add a web service task to post to a webhook.

Reloading Data

When there’s been a configuration change between ETL tasks that depend on each other, a button will appear to reload all data with the new configuration. Click Reload the data if you’d like to reprocess the old data.
 

the "reload the data" button an an ETL task that has changed
the "reload the data" button an an ETL task that has changed

Attention: This will reload all data from the task that was edited. Normally, ETL workflows only process new data since the last run. However, if this option is selected, then the next time the workflow runs, all data since the workflow was created will be processed.

Example: When an extract responses from survey task normally runs, only new responses since the last run will be processed. However, if data is reloaded, then the next time the workflow runs, all responses in the survey will be processed.

Extracting Data from a Specified Date Range

With certain ETL workflows, you can extract data from a specific date range. This can be a great way to reprocess data that has been extracted before, or test that your setup is working.

Extracting data from a specific range does not impact other scheduled runs.

Example: A ticket was already processed by your workflow. Since then, though, the data on it has changed. To reprocess the ticket, you use this function.

Qtip: This feature is only available for scheduled ETL workflows.

Supported Extractor Tasks

You can only extract data from a specified date range if you use one of the following tasks:

Qtip: Date range extraction will only work if you have 1 supported extractor task in your workflow.

Run a workflow with a custom date range

Follow these steps to extract data only from your chosen date range.

Create a complete ETL workflow with one of the supported extractor tasks.
ETL workflow
ETL workflow
Click Run Immediately.
Click Advanced settings.
Advanced settings, allowing run workflow with custom date range
Advanced settings, allowing run workflow with custom date range
Select Run workflow with custom date range.
Click Next.
Choose a start date and time.
Choosing a date and time for the custom date range
Choosing a date and time for the custom date range
Qtip: The earliest start you can select is January 1, 1970.
Choose an end date and time.
Qtip: The latest end date and time you can choose is the current one. So long as it’s activated, your workflow will extract future data as it continues to run on its schedule.
Choose a time zone.
Click Yes to confirm.

Troubleshooting a run

Go to your workflow run history. Click the extraction task and view the JSON. In the Output, the CDAP_DateRangeExport fields should reflect the date range you chose. (The CDAP_Worker fields, on the other hand, will show when you ran the test.)

Output of the run history
Output of the run history

Available Extractor Tasks

Here are some of the extractor tasks available at this time:

Qtip: You can also view a list of extractors using the site menu to the left.

Available Loader Tasks

Here are some of the loader tasks available at this time:

Qtip: You can also view a list of loaders using the site menu to the left.

Available Data Transformation Tasks

The following tasks are available to you for transforming the data you process in your ETL workflows:

  • Basic transform task: Change the format of strings and dates, calculate the difference between dates, perform math operations on numeric fields, and more.
  • Merge task: Combine multiple datasets into a single dataset.
  • Redact and substitute data task: Redact data within your workflow based on regex or substitution rules.

Workflow Limits

ETL workflow runs have the following limits:

  • Each run can process up to 200 files
  • Each run can process up to 10GB of data in total
  • Tasks can process files up to 5GB in size

Qtip: Note that each run includes all tasks that are being run within the workflow.

Make sure to also check the documentation for the specific task you are using as some tasks have smaller limits.

Troubleshooting Data Extracting and Loading Tasks

Attention: An ETL workflow is automatically disabled if it meets any of the following criteria in the past 13 days:

  • Every attempted execution has been a failure.
  • There have been at least 10 failures.
  • Every execution is failing for the same reason.
  • The first and last execution were at least 10 days apart.

Incomplete Workflows

Data extractor and loader tasks must be used together. If you are missing a piece of their setup, the workflows editor will alert you.

Example: In this example, we are missing our data loader task. The workflows editor alerts us that our “import task is currently missing a destination.”
 

the "missing a destination" error message
the "missing a destination" error message

Example: In this example, we are missing our data extractor task. The task editor alerts us that “you need to add at least one data source task.”
 

creating a loader task without an extractor. the task alerts you to set up an extractor first
creating a loader task without an extractor. the task alerts you to set up an extractor first

Workflows Failing

If your tasks are failing or not firing properly, the first place you should look is Workflows reporting & history. This will contain information about every time your workflow fired and the results of that workflow.

In reporting & history, each piece of your workflow will have its own entry, making it easy to pinpoint where things went wrong.

Example: In the below example, we are using an extract data from SFTP files task and a add contacts and transactions to XMD task. We can see that the extractor task failed but the loader task succeeded. This means our extractor task is misconfigured, but our loader task is OK.
 

the reporting section. the extract task failed but the load task succeeded
the reporting section. the extract task failed but the load task succeeded

After identifying the problem, you can click View under Details to see more information to help you diagnose how to fix the problem. This will pull up the JSON payload for the task. Scroll down to the Task Output section to find any errors.

Example: In this example, we see that the task failed because the connection had invalid parameters for an extract data from SFTP files task.
 

the task output section of a failed action giving the error why it failed
the task output section of a failed action giving the error why it failed

After editing your workflow and fixing the problem, you can return to reporting & history and click Retry to rerun the workflow.
 

the retry button for retrying a failed workflow
the retry button for retrying a failed workflow

FAQs

Do I have access to all extensions available through the Workflows templates?

Is there a link to all Workflows events and tasks?

What are Workflows templates?

What are workflows? / Where are actions?

Was this helpful?

The feedback you submit here is used only to help improve this page.

That's great! Thank you for your feedback!

Thank you for your feedback!