Create a Datasource Route

Datasources are used to get file based data from a source, such as an SFTP server or S3 bucket. The data source, once setup and enabled, will poll the location for any new file that fit the criteria set in the datasource.

To create a data source, do these steps:

1. In the left navigation bar choose Data Source
2. Click on the Storage Routes from the top navigation tabs
3. Select the owner you want associated to the route from the "Select Owner" dropdown menu.
4. At this point we can do three things depending on the use case
1. Create new user in Bettrdata
1. Log into Bettrdata
2. Click on the Data Sources link in the left navigation bar
3. Click on Storage Routes in the top tab
4. Select the data owner this data will belong to
5. This use case is where you want to provide a particular user with a share where they will put data. For this do these step
1. Click on New → Hosted SFTP Account
2. Enter the user’s name and email address
3. Click Create
1. This will create a share in Files.com with the username the same as the email address and a home folder also with the same email address, and two folders underneath
• to_<instance>
• Folder where data is dropped to be processed in Bettrdata
• from_<instance>
• Folder where processed data can land
2. Create new user in Files.com
1. If the user you want to create is going to be used for multiple users, you may want to create a user and folders that can be accessed from multiple users. In this case, it may make sense to create a user that is not an email address. Do these steps
1. Log into Files.com
2. Click on Files
3. Under the /home directory create the folder you will share. It is recommended it be the same name as the shared user
4. Under that also create the two folders
• to_<instance>
• Folder where data is dropped to be processed in Bettrdata
• from_<instance>
• Folder where processed data can land
5. After that, create the user
6. Click on User Accounts in the left navigation bar
7. Click on New User
8. Enter the details needed
1. Username
2. Full name as makes sense
3. Company
9. Click on Folder permissions and choose the folder you created for the user and add it
10. Click Create User to finish
3. Connect to existing user share
1. Prep: Put a file in the location /home/{{user name}}/to_<instance> in files.com to test
2. Once the user accounts are created in Files.com, the easiest way to connect to the share with the source route is to use the darkwatch_admin user
3. Log into Bettrdata
4. Click on the Data Sources link in the left navigation bar
5. Click on Storage Routes in the top tab
6. Select the data owner this data will belongs to
7. Click on New → Storage Route
8. Click SFTP
1. Enter the credentials for the darkwatch_admin user
2. Host: files.darkwatch.io
3. Port: 22
4. Directory /home/{{user name}}/to_<instance>
5. Max Recursive: Usually 1
6. Click Save
9. Click on the List Files tab
10. Click List Files on Routes. This should show the file you uploaded to the share earlier
11. If all looks good, click on Edit Route again and check the boxes
• Enabled
• No Fail
• No Delete, if you do not want to remove files from the share as part of the ingest
12. Click Save to finish
5. Re-open the source route to validate. Click on the "List Files" tab
6. Click on the "List Files on Route". It should show the files in the location (if there are any). Add some files to the location and list again to verify.
7. Once verified, click on the "Settings" tab. This provides three rows that can be expanded
• Filters
• This is used to select which files will be processed by file name. By default, all files will be chosen if nothing is added here. As an example, if the file names you want to process are like "customer_20251230.csv" and "transactions_20251113.csv" you could create two filters, use the patterns "customer" and "transaction" to process
• Mappings
• This is used to map files by filename to output schemas. For instance if the file names are like "customer_20251230.csv", you could use the pattern "customer" and then assign it a schema output (type)
• Archive Passwords
• This provides a way to have a list of passwords that can be used to decrypt zip or pgp files. You can add multiple passwords, and they will be used in turn to try to decrypt a file in those formats
8. Once you are done with filling in those values, click on the "Edit Route" tab. You can now checkbox some important checkboxes:
1. No Delete: If you do NOT want the files that bettrdata processes to be deleted from the store location, check this box
2. No Fail: Bettrdata checked a server three times in a row by default if a store is accessible. If it failed, then it will disable the route. This checkbox prevents this from occurring.
3. Enabled: Enables the route.
9. Click the Save button to save changes.