Customer Epics

Template epics

Deep Dive 2

• Almost all docker are using the base and do not add anything code wise
• Just the launch command is different

Process Overview

There are two names for most processes. Processes loop as part of the looking for new event

• Hunter → processing → indexing, may be passed over for programmatic load
• Trapper → Does the search
• Integration - uses items on the integration job from the integrationRun
• Gatherer → Processing, schema, imports the file, adds the fid
• In Back: Autoconverter - Checks for items with converts and updates the mongo datafile attributes and creates a convert run record
• Cooker → Conversion, loops to look for convertRun records ONLY. Priority comes from convertRun, client priority, and schema. After ran, updates the datafile record and convertrun records
• TransformIO does the actual code to process the file
• Cooker test only runs 50k records, NOT part of ordered process
• Validator - Checks from convert runs, checks file, and sets to review complete. Autovalidate included in this
• Loader → transport files to end point, updates the record and then runs

Service that are not part of an ordered Process

• Mixes are merge
• Proxy → nginix, does not handle cognito, but allows port for api
• Cron → Scheduling of services ?
• Back → API, every service call its
• Back → also includes a set of lightweight services
• Back-ext: client version of back
• Profiler → broken out from back
• Autoprofile: from CCC added, from Admin UI

Docker Additional Information

• Docker name format: if the name has dash instead of periods, then it could have multiple instances
• Networks to talk to eachother
• looping to check on resources
• Docker to Run Code without removing manual changes
• Use docker commit - to stop and restart a docker without losing manual changes
• Address Parser: fullAddress field must exist for address parser
• Flag inside convert type to address into ALL components NOT used
• IntegrationCoordinator: Creates node JS crons timers for loading integration jobs
• Local: is the local sftp service

BettrData Overview

Action/Questions

New Tickets

Invimatic is only New UI

• Ticket: Add query to AB history
• Ticket: Bug for From New SFTP Account, breaks
• Ticket: Bulk: Change type → Change Scema (text change)
• Ticket: Builder, Toggle Hide/Unhide fields
• TIcket: Placeholders in Storage route fields
• Bug Ticket: Change Add to Merge Job: Create New → Create merge job
• Ticket: Change black and white to purge and not purge

Questions

• Talk about multiline
• Global template: How to apply global template as well as client-specific converts
• JSON:
• Logs deep dive
• Go over steps
• Mapping data special cases
• Custom fields
• Show hidden fields

Deep Dive

Waiting to convert may be there if thresholds are not meeting settings

Merges

Have not done this.Merges

Base on keys, which you need to set

Works on converted data

Can check multiple converted job

What happens if omits and select are both included

Is there an order?

Do the converts of a class need to match

Convert Versus Convert Type

Requirements for convert types

• file type
• client
• convert type
• length
• version / qualifier

Convert Type and Output Schema are synonyms

Me

Bulk Processing

Cannot change an active process

Virtual Files

Want

Virtual does not work with multi-line: bug

Compare Header

Good to finding which convert type to use

Needs to be the same file type

Everest multi-convert types

If using validate headers, it will guess from headers

Builder

Global templates

For sales force data or epsilon 401 format

Convert to Template

• Converts a Convert type to a global convert type (template)
• Need to create a client for the global client
• There can be only one template client

Logic

Looks for client specific convert types, and then the global convert types

Validate

Checks if all of the columns are named exactly the same and in the same position

Note

Transform IO and Command IO uses a text file in a structured format

Builder

Build versus Safe

Safe will not change mapped fields

Build will rebuild be position, can end up with new column named

Drop Empty Records

Will drop rows where the MAPPED fields are empty

Admin

Tags

To set a client-specific value

Adds an attribute to the client object in Mongo

Put in client object under attributes field, which is a map

Data Owners

Check with Karim and Aaron if brands are required

Integration Routes

Our manual logic to pull from sources

Good as a start, but can be useful