Search

Create a Datasource Route

Datasources are used to get file based data from a source, such as an SFTP server or S3 bucket. The data source, once setup and enabled, will poll the location for any new file that fit the criteria set in the datasource.

To create a data source, do these steps:

  1. In the left navigation bar choose Data Source
  2. Click on the Storage Routes from the top navigation tabs
  3. Select the owner you want associated to the route from the "Select Owner" dropdown menu.
  4. Click on the New button.
    • As a side note, you choose "Storage Route" from an existing source. In this step of steps we will configure for an existing SFTP route.
    • If you want to create a new SFTP account and coinciding route you can choose "Hosted SFTP Account".
  5. A dialog box appears. Add a description of the data routes purpose
  6. From the Protocol dropdown, choose SFTP.
  7. Enter the host, port, username, password, and directory. There is also a recursion depth that can be set to look for files under the main folder.
  • When entering the password, make sure to unlock the text box, enter the value, and lock the text box to encrypt it.
  1. Click save to check that the initial set up is working
  2. Click on the "List Files" tab
  3. Click on the "List Files on Route". It should show the files in the location (if there are any). Add some files to the location and list again to verify.
  4. Once verified, click on the "Settings" tab. This provides three rows that can be expanded
    • Filters
      • This is used to select which files will be processed by file name. By default, all files will be chosen if nothing is added here. As an example, if the file names you want to process are like "customer_20251230.csv" and "transactions_20251113.csv" you could create two filters, use the patterns "customer" and "transaction" to process
    • Mappings
      • This is used to map files by filename to output schemas. For instance if the file names are like "customer_20251230.csv", you could use the pattern "customer" and then assign it a schema output (type)
    • Archive Passwords
      • This provides a way to have a list of passwords that can be used to decrypt zip or pgp files. You can add multiple passwords, and they will be used in turn to try to decrypt a file in those formats
  5. Once you are done with filling in those values, click on the "Edit Route" tab. You can now checkbox some important checkboxes:
    1. No Delete: If you do NOT want the files that bettrdata processes to be deleted from the store location, check this box
    2. No Fail: Bettrdata checked a server three times in a row by default if a store is accessible. If it failed, then it will disable the route. This checkbox prevents this from occurring.
    3. Enabled: Enables the route.
  6. Click the Save button to save changes.