The Workflow Tabs located across the top of the Data Ops files page provide the ability to easily follow and manage the file processing workflow. You can click on each tab to view and manage files that fall into that particular segment of the workflow.
Tabs Include:
All
Default view displaying all owners and files with the ability to search.
My Data
Files with a status of Lock can be found on this tab.
On Hold
Used to place data in a holding state for future review. Data feeds in this state will not continue normal processing if they are in the pipeline. Please note, this will not pause or stop any active processing. The count for the number of files in the on hold status will appear on the On Hold tab in the workflow.
Processing
Before processing actually begins, the system places each remote file in the status of Waiting to Process as it pauses to assure that the file is not still growing. If the file remains static (with the same byte count) for two minutes, it is moved onto Processing. However, if during the two minute pause the file size does change, the two minute count will restart and continue until the file size remains static.
The count for the number of files Waiting to Process will appear on the Processing tab in the workflow. These statuses (and others) are also available in the Status column menu on the Manage Files gray bar.
Once a file begins Processing, the system will begin to analyze the data, map the file to an Output Schema, if known, and prepare the data for conversion. During Processing a file can be placed in different statuses or receive various alerts if problems arise during processing.
Note that the following file types are currently supported. If the file doesnât match the expected format or has issues, it might not be a supported file type.
- ASCII
- UTF-8 fixed or delimited
- ZIP
- GZIP
- Excel (xls, xlsx)
- RAR (short for Roshal Archive)
- JSON (JavaScript Object Notation), "depth = 1"
- PGP (Pretty Good Privacy) and GPG (GNU Privacy Guard)
- DBF (Database File)
Unidentified
This status is assigned to any file that is missing Output Schema, Type, Format, or Length. Format and Length are rarely not detected by the system and if missing it is an indicator that there is an oddity with the data. The count for the number of unidentified files will appear on the Unidentified tab in the workflow.
See Output Schemas for more information.
Waiting to Convert
At this status the file has everything it needs to be converted using the Builder.
Files stay in Waiting to Convert status for 1 of 2 reasons.
Either there is:
- no convert schema setup (builder)
- Auto Convert is not enabled
For files in the status of Waiting to Convert, the system provides a quick link by clicking on âWaiting to Convertâ to go to the next step in the process using the Builder to build a new convert.
The Open in Builder option is also available on the filesâ menu icon.
See Builder for more information on building a new convert.
Converting
In this status, the data is being actively transformed. There are no user actions required when files are in this state.
- Convert Complete
The Convert process has been completed and standardized output data is ready for downstream processing.
Waiting for Review
A file will bypass the status of âWaiting for Reviewâ automatically moving it onto âConvert Completeâ, if all validations have passed.
A file will stay in the status of âWaiting to Reviewâ when:
- A threshold configuration has not been created for the given convert
- A threshold validation has failed in the run report
- Auto Validate box is not checked in either the run reportÂ
- Auto Validate enabled box isnât checked in Administration > Manage Configurations > System Settings > Conversion > Auto Validate
If validations have not passed, click on âWaiting to Reviewâ in the status column
and the system provides a quick link to the next step in the process, which is to view the convert reports.Â
The View Report is also available in the file hidden menu icon.
See Run for more information on Run Convert.