BettrData ↔ Local Sync/Trigger Utility

This utility synchronizes data files from your BettrData instance via API and SFTP, executes user-defined commands (on a local host) on downloaded files, and manages processing status updates.

• ⚙️ Overview
• 🧾 Configuration File — default.ini
• Structure
• 🧠 Configuration Sections
• 🔹 [bettrdata]
• 🔹 [sftp]
• 🔹 [list] — Query Settings
• Example Filters
• 🔹 [processing] — File Handling & Commands
• Jinja2 Command Variables
• Example
• 🔹 [email] — Optional Notifications
• 🔹 [logging]
• 🚀 Runtime Options
• Examples
• 1. Run once for testing
• 2. Run continuously every 5 minutes
• 3. Use custom config file
• ⏱️ Controlling Wait Time & Batch Size
• 🧰 Example End-to-End Setup
• 🪵 Logs & Troubleshooting
• 🧱 File Status Conventions
• 📬 Email Behavior
• 🧩 Typical Workflow

⚙️ Overview

The BettrData Sync Utility automates data retrieval, processing, and synchronization using your BettrData API and SFTP credentials. It can:

• Fetch unprocessed or specific files from your BettrData instance.
• Download files securely via SFTP.
• Run a custom shell command or script for each file.
• Update processing status via configurable attributes.
• Optionally send success/failure email notifications.
• Run continuously (polling) or once (batch mode).

🧾 Configuration File — default.ini

The configuration file defines runtime behavior.

Example path: config/default.ini

Structure

🧠 Configuration Sections

🔹 [bettrdata]

🔹 [sftp]

🔹 [list] — Query Settings

This section defines which files are fetched from BettrData.

Example Filters

💡 WARNING! Do not combine filters using $and; however OR $or is allowed

// AND example:
{
  "status.processed.done": true, 
  "virtual": { "$ne": true },
  "attributes.client_status": { "$exists": false }
}
// OR example:
{ "$or": [
  { "status.processed.done": true },
  { "virtual": { "$ne": true } },
  { "attributes.client_status": { "$exists": false } }
] }

🔹 [processing] — File Handling & Commands

Jinja2 Command Variables

These variables are rendered per file and can be used in your command:

Example

command_1 = c:\scripts\process_file.bat {{filePath}} {{recordCount}} {{md5}}

🔹 [email] — Optional Notifications

Emails are only sent if smtp_host is defined.

🔹 [logging]

🚀 Runtime Options

The script accepts runtime parameters for controlling execution frequency.

python main.py [--config path/to/config.ini] [--once] [--interval N]

Examples

1. Run once for testing

python main.py --config config/dev.ini --once

2. Run continuously every 5 minutes

python main.py --interval 300

3. Use custom config file

python main.py --config c:\client\sync\custom.ini

⏱️ Controlling Wait Time & Batch Size

Example: Fetch 25 records every 5 minutes:

Copy

python main.py --interval 300

and in default.ini:

Copy

[list]
limit = 25

🧰 Example End-to-End Setup

1. Configure default.ini

Set:

• API token and SFTP credentials
• Output directory
• command_1 to run your transformation or load step
• Optional email recipients

2. Test single run

python main.py --once

3. Schedule recurring sync

• Run continuously with -interval
• Or register the .exe build in Windows Task Scheduler / startup

🪵 Logs & Troubleshooting

• Logs are stored under the directory defined in [logging] log_dir.
• Default path: logs/
• Log level is controlled by level.
• DEBUG → most verbose
• INFO → normal operation
• ERROR → only failures

🧱 File Status Conventions

These are stored in the BettrData document under:

attributes[client_status]

(or your custom status_attribute)

📬 Email Behavior

If email is enabled:

• On successful file processing → email sent to success_email
• On failure or exception → email sent to alert_email

SMTP credentials must be valid; otherwise, email sending is silently skipped with a log entry.

🧩 Typical Workflow

1. The script queries BettrData for eligible files (where filter).
2. It downloads each file via SFTP.
3. The command defined in command_1 runs locally.
4. The file’s status is updated in BettrData (STARTED, SUCCESS, FAILED).
5. Logs are written and optional emails sent.
6. Waits interval seconds before repeating (unless -once used).