Search

Home

Release Notes

User Docs

đŸ§©

Workflow Tabs

The Workflow Tabs located across the top of the Data Ops files page provide the ability to easily follow and manage the file processing workflow. You can click on each tab to view and manage files that fall into that particular segment of the workflow.

Tabs Include:

image

All

Default view displaying all owners and files with the ability to search.

My Data

Files with a status of Lock can be found on this tab.

On Hold

Used to place data in a holding state for future review. Data feeds in this state will not continue normal processing if they are in the pipeline. Please note, this will not pause or stop any active processing. The count for the number of files in the on hold status will appear on the On Hold tab in the workflow.

image

Processing

Before processing actually begins, the system places each remote file in the status of Waiting to Process as it pauses to assure that the file is not still growing. If the file remains static (with the same byte count) for two minutes, it is moved onto Processing. However, if during the two minute pause the file size does change, the two minute count will restart and continue until the file size remains static.

The count for the number of files Waiting to Process will appear on the Processing tab in the workflow. These statuses (and others) are also available in the Status column menu on the Manage Files gray bar.

image

Once a file begins Processing, the system will begin to analyze the data, map the file to an Output Schema, if known, and prepare the data for conversion.  During Processing a file can be placed in different statuses or receive various alerts if problems arise during processing.

Note that the following file types are currently supported. If the file doesn’t match the expected format or has issues, it might not be a supported file type.

  • ASCII
  • UTF-8 fixed or delimited
  • ZIP
  • GZIP
  • Excel (xls, xlsx)
  • RAR (short for Roshal Archive)
  • JSON (JavaScript Object Notation), "depth = 1"
  • PGP (Pretty Good Privacy) and GPG (GNU Privacy Guard)
  • DBF (Database File)

Unidentified

This status is assigned to any file that is missing Output Schema, Type, Format, or Length. Format and Length are rarely not detected by the system and if missing it is an indicator that there is an oddity with the data. The count for the number of unidentified files will appear on the Unidentified tab in the workflow.

image

See Output Schemas for more information.

Waiting to Convert

At this status the file has everything it needs to be converted using the Builder.

Files stay in Waiting to Convert status for 1 of 2 reasons.

Either there is:

  • no convert schema setup (builder)
  • Auto Convert is not enabled

For files in the status of Waiting to Convert, the system provides a quick link by clicking on “Waiting to Convert’ to go to the next step in the process using the Builder to build a new convert.

image

The Open in Builder option is also available on the files’ menu icon.

image

See Builder for more information on building a new convert.

Converting

In this status, the data is being actively transformed. There are no user actions required when files are in this state.

  • Convert Complete
  • The Convert process has been completed and standardized output data is ready for downstream processing.

Waiting for Review

A file will bypass the status of “Waiting for Review” automatically moving it onto “Convert Complete”, if all validations have passed.

image

A file will stay in the status of “Waiting to Review” when:

  • A threshold configuration has not been created for the given convert
  • image
  • A threshold validation has failed in the run report
  • image
  • Auto Validate box is not checked in either the run report 
  • image
  • Auto Validate enabled box isn’t checked in Administration > Manage Configurations > System Settings > Conversion > Auto Validate
  • image

If validations have not passed, click on ‘Waiting to Review” in the status column

image

and the system provides a quick link to the next step in the process, which is to view the convert reports. 

image

The View Report is also available in the file hidden menu icon.

See Run for more information on Run Convert.

Merges

Workflow Menu

AllMy DataOn HoldProcessingUnidentifiedWaiting to ConvertConvertingWaiting for ReviewMergesWorkflow Menu