Extract Data from SFTP Files Task

About Extract Data from SFTP Files Task

The Extract data from SFTP files task allows you to import data from your SFTP server to use in Qualtrics. You can use this task to process just one or multiple files at a time. You can combine this task with other tasks, such as the Load data to XM Directory task, to import data into other places in Qualtrics.

For more information on SFTP setup and troubleshooting, see SFTP Troubleshooting.

Connecting an SFTP Account

Before you can use the Extract data from SFTP Files Task, you must connect an SFTP account for Qualtrics to extract the data from. Accounts may be added by Brand Administrators for other users in the brand to use, or by individual users for their personal use.

There are 3 different authentication options:

• Password: Authenticate with a username and password.
• SSH Key: Authenticate using a Secure Shell (SSH) key that you provide, or a key provided by Qualtrics.
• Encrypted SSH Key: Authenticate using an encrypted SSH key.

Adding an Account as a Brand Administrator

If you are a Brand Administrator, follow the steps in this section to add an SFTP account for some or all users in your license to use.

1. Using the navigation menu in the top left, go to Admin.
   
2. Go to Extensions.
3. Click the Extract data from SFTP files extension.
4. Click Add Account.
   
5. Give the account a name. This name is used for organizational purposes and to identify the account.
   
6. Enter the Hostname of the associated SFTP server.
   Example: Commonly these are in the format of sftp.example.com
7. Enter the Port number. Usually, this is 22.
8. Choose your authentication method. Your options include:
   • Password: Provide a username and password for authentication.
   • SSH key: Provide a username and SSH key for authentication, or use an SSH key generated by Qualtrics.
   • Encrypted SSH key: Provide your own encrypted SSH key with a passphrase for authentication.
9. If you chose password, enter the Username and Password for the SFTP account.
   
10. If authenticating with an SSH key, do one of the following:
    • If using an SSH key generated by Qualtrics, enter a Username and then click Generate SSH keypair.
      You can then click Download public SSH key to download the public key as a .pem file to your computer. The associated private key will be stored by Qualtrics. Click Copy to clipboard to copy the key’s fingerprint to your clipboard. Save both of these in a secure place so you can reference them if needed.
    • If using your own SSH key, click Provide a private key.
      Then, enter your Username and the SSH Key.
      Qtip: You must enter a non password-protected private key in this box. Additionally, the public key must be authorized on your SFTP server.
      Example:
      This is what an SSH key typically looks like (including the begin and end RSA key lines):
      —–BEGIN RSA PRIVATE KEY—–
      a1b2c3d4…
      —–END RSA PRIVATE KEY—–
11. If authenticating with Encrypted SSH key, enter the Username, Encrypted SSH key, and Passphrase associated with your key.
12. Once finished, click Connect Account.

Adding an account as an Individual User

Follow the steps in this section to add an SFTP account for your personal use. Only you will be able to use the account that you connect.

1. Follow the steps in Setting Up an Extract Data from SFTP Files Task section until you get to the step where you select an account to use.
2. Click Add user account.
   
3. Follow the instructions in the “Adding an account as a Brand Administrator” subsection above for more information on connecting your account. The same authentication options and settings are available for you when adding an account as an individual user.

Testing the Connection

After you’ve added an account, you can test the connection from within the Extract data from SFTP files task. At the bottom of the task, click Test connection. This will upload a small test file to your server, read its contents, and then delete the file. 

Downloading public SSH keys

If using an SSH key generated by Qualtrics, you can download the public key for the connection. When viewing the list of SFTP accounts to use, click the three dot menu to the right of an account and then choose Download public SSH key. This will download the key to your computer as a .pem file.

Updating Qualtrics-Generated SSH Keys

If using an SSH key generated by Qualtrics, you can regenerate the key if needed. This is useful if you believe the key has been compromised or if your company requires you to rotate keys after a certain amount of time.

1. When viewing the list of SFTP accounts to use, click the three dot menu to the right of the account and click Edit account.
2. Click the hyperlink to generate a new keypair.
   

Setting up an Extract Data from SFTP Files Task

1. Create an ETL workflow.
2. Select Data source (ETL extractor).
   
3. Choose the Extract data from SFTP files task.
   
4. Choose the SFTP account to extract files from. Any accounts added by Brand Administrators, as well as accounts you’ve added yourself, will be listed here.
   
5. If you want to add new SFTP account credentials, click Add a user account and follow the steps in the connecting an SFTP account section.
6. Click Next.
7. Enter the File name pattern, which is used to identify what files to process by looking at the file’s prefix. All files with your specified prefix will be processed. Enter the desired prefix into the box. You do not need to add an asterisk (*) to the end of the prefix.
   
   Example: Let’s say I have a file called Qualtrics-DataImport.csv. I can enter “Q” or “Qualtrics” as the prefix, depending on if there are other files in the server I want to process.
   Qtip: Click the Use regular expression checkbox if you’d like to use regex for your file name pattern.
8. Enter the Pickup directory or the directory path to find the file on the server. You may enter either a relative path or an absolute path.
   Qtip: This directory path is case sensitive and must point to an existing folder or the workflow will fail. Qualtrics will not be able to access files if they’re in a higher level directory than the one specified in the pickup directory.

   Example: Depending on the path you enter, potential pickup directories could be:

   • /home/username/myFolder (absolute path)
   • myFolder (relative path)
9. If your file is compressed, enable the Are you extracting from a zip file option. Then, enter the Prefix for file within the zip which will be used to find the correct file to import from the zip file.
   Qtip: Only .zip compressions are supported. The files in the .zip file can be encrypted, but the .zip file itself cannot be (for example, filename.csv.pgp.zip is supported but filename.csv.zip.pgp is not).
10. Select a file type.
    
    Your options include: CSV, TSV, JSON, or Excel (.xslx or .xls).

    Qtip: JSON files must have their content enclosed in square brackets, []. For example:

    [
      {
          "fruit": "Apple",
          "size": "Large",
          "color": "Red"
      }
    ]

    Attention: Only UTF-8 files can be used.
11. If processing a CSV, choose the Delimiter for the file. Your options include comma, semicolon, tab, and pipe.
12. If desired, select Decrypt files before importing using PGP. To learn more about this extra means of encryption and data protection, see PGP Encryption.
    Qtip: Make sure the files you want to import are encrypted with Qualtrics’ public key before uploading them to your SFTP server. To do this, you may need to talk to your IT team.
13. If you’d like to skip over files that can’t be processed, select Skip bad files. If this option isn’t selected, the entire workflow will fail when it tries to process the bad file.
    Qtip: “Skip bad files” will not process files that are over the maximum file size and files that can’t be opened, such as those that are corrupted in some way. While this setting reduces the chances a workflow will fail, it will not account for issues that could occur in files that can be opened, such as incorrect headers and formatting.
14. Choose what you’d like to happen after the file is processed. Your options include:
    • Delete the processed file: After processing the file, Qualtrics deletes it from your SFTP server. This option is enabled by default.
    • Keep the processed file: After processing the file, Qualtrics leaves the file on your SFTP server.
    Attention: If your SFTP server is configured to automatically delete or move files after they are touched, the post-processing option you select may not work as intended.
15. Decide if you’d like to Use strict validation, which will automatically reject extracted files if they have certain formatting issues. Read the message in the task for more information about which files will be rejected.
    
    Qtip: Enable Skip malformed rows to upload your file and exclude any rows with formatting issues.
16. Click Choose file and upload a sample file, which will tell Qualtrics the fields you want to import, and will determine the subsequent tasks you can use with this task. The file you upload should contain the fields you want to import as part of this task.
    Qtip: Download the sample files in the task, or download the files in the example files section for templates on how to structure your data based on your use case. The only supported file encoding format is UTF-8. Some spreadsheet software saves as Unicode UTF-8, which is different from UTF-8, if you are getting unexpected errors. Please ensure that you are exporting in UTF-8 format and not Unicode UTF-8.
    
17. Double check that your file is importing the data correctly. If you need to change your settings, click Back at the top of the task.
    

    Qtip: If all your data is being imported as one column, you are using the wrong delimiter. The below screenshot demonstrates what this looks like in the task window.

    

18. If your data looks correct, click Save.
19. Finish building your ETL workflow. When you choose a loader task, you’re deciding the destination of the data you extracted. See available loader tasks for a list of options.
    

Example Files

Below are example files for how to structure your data for import, depending on the type of data you want to import. Click on a file name to download the example file.

• User Data
• Survey Response Data
• Salesforce Contact Data
• Salesforce Account Data
• Salesforce Account Team Member Data
• Employee Data

FAQs