Skip to main content

​
Intro

Amazon S3 is an online file storage web service offered by Amazon Web Services which you can use to store and retrieve any amount of data, any time, from anywhere on the web. Use this connector to import data from an Amazon S3 bucket into Domo. You connect to your Amazon S3 account in the Data Center. This section discusses the fields and menus that are specific to the Amazon S3 connector user interface. To add DataSets, set update schedules, and edit DataSet information, see Adding a DataSet Using a Connector.

​
Prerequisites

To connect to your Amazon S3 account and create a DataSet, you must have the following:
  • Your Access Key
  • Your Secret Key
  • Name of the AWS S3 Bucket you want to retrieve the files from
  • You also need to select the Amazon S3 Region

​
Connecting to Your Amazon S3 Account

This section enumerates the options in the Credentials and Details panes in the Amazon S3 Connector page. The components of the other panes in this page, Scheduling, and Name & Describe Your DataSet, are universal across most connector types and are discussed in great detail in Adding a DataSet Using a Connector.

​
Credentials Pane

This pane contains fields for entering credentials to connect to your Amazon S3 account. The following table describes what is needed for each field:

Field

Description

Access Key

Enter your AWS access key. The AWS Key will be available in the AWS Console under the Security Credentials section or under Users if you are using IAM.

Secret Key

Your AWS secret access key. It will be available when you create your Access Key. You can generate a new secret key in the AWS Console.

Bucket

The name of the AWS S3 Bucket.

Region

Select the Amazon S3 Region.

Once you have entered valid Amazon S3 credentials, you can use the same account any time to create a new Amazon S3 DataSet. You can manage connector accounts in the Accounts tab in the Data Center. For more information about this tab, see Manage Connector/Adapter Accounts.

​
Details Pane

Menu Description
What File Type Would You Like To Import? Select the file type that you want to parse and import.
How would you like to choose your filename? Choose how you would like to select your file.
Parse large XML file If your file is large, select the checkbox to parse it with a stream parser to avoid parsing issues. The recommended file-size is greater than 30MB.
Enter complete filepath Please enter complete filepath.
Example: folder_name/file_name .
List of Files Select the file from the retrieved list of files.
Prefix Prefix limits results to only those keys that begins with a specified prefix.
S3 Path Helper The s3 prefix that will be used to search the file in bucket.
File Name Enter the name of the Amazon S3 Object(file) that you want to import.
File Name Match Type Specify whether the retrieved file starts with or contains the name you have entered in the File Name .
File Compression Type Select the compression type for your file.
Zip File Encoding Select the zip file encoding.
How would you like to choose your file present inside a zip? Choose how would you like to select your file present inside a zip folder.
Enter File Name inside zip Enter the filename present inside the zip.
Example: folder_name/file_name
Match Type For File Name Inside Zip Specify whether the retrieved file should starts with or contains the name you have entered in ENTER FILE NAME INSIDE ZIP .
Select a File From The ZIP Archive Select the file from the retrieved list of files.
Sheet Name Select the sheet from the input file.
Select an Option

Select an Option:

Option Description
Enter Sheet Name Enter the sheet name from the excel file that we need to pull data from.
Select Sheet Name Select the sheet name from the excel file that we need to pull data from.
Are headers present in the file?

Select YES if the file contains headers, else select NO.

Select the delimiting character

Select the delimiting character used by your character separated values (CSV) text. If your delimiter is not listed select ‘Other’.

Specify your delimiter

Enter the character used to delimit your character separated values (CSV) text.

Quote Character

Select the desired quote character for parsing CSV files. (A double quote is the CSV standard).

Enable Parsing for Large Json Files?

Select yes to enable parsing the large Json files.

Subfile Name Enter the name or a portion of the name of the subfiles that you want to import.
Add Filename Column Specify if BATCH_FILE_NAME column should be added to the final output.
Enter Your Data Tag Enter the tag for the data in your file.
Enter Your SubList To Flatten Enter the comma separated lists that you want to flatten out in your data.
Do you require attributes in data? Select ‘Yes’ if you require attributes values as a part of data.
Does your JSON text require a line reader? Select yes if your JSON text includes multiple lines that should be read.
Should the backslash be escaped? Select yes if your JSON text has backslash characters that need to be escaped.
Escape Character Select the desired escape character for parsing CSV files.
Custom Escape Character Enter the desired CSV escape character. Enter Unicode chars in \uxxxx form (e.g. \u005E).
Enter fields to exclude Provides a comma separated list of fields you want to exclude from the import.
Enter your header tag Enter the tag for the header in your JSON text.
Enter XPath expression Enter your XPath expression.
Header Start Row Enter the number for the header row in the data set.
Data Start Row Enter the number for the first data row in the data set.
Footer Enter the number of rows at the end of the data set to skip. For example, to skip the last two rows you would enter 2 .
Blank Rows Specify what should happen if blank rows are encountered.
Empty Column Headers Specify what should happen if empty column headers are encountered. If you choose Add blank columns , column names are automatically generated for the new columns.
File Encoding Select the file encoding. Defaults to UTF-8 .
Use Previous Schema (Append Mode) Select the checkbox to use the previous schema in the subsequent runs of the dataset (applicable only in Append mode). If it’s not selected, the previous schema will not be used.

​
Other Panes

For information about the remaining sections of the connector interface, including how to configure scheduling, retry, and update options, see Adding a DataSet Using a Connector.