Write Out Logic
For new file stuck
async function checkPipelineHealth(db) { const oneHourAgo = new Date(Date.now() - 60 * 60 * 1000);
// Count datafiles stuck in converting (submitted > 1 hour ago) const convertingQuery = { "status.processed.done": true, types: { $exists: true, $ne: [] }, format: { $exists: true }, length: { $gt: 0 }, "status.converted.pid": { $exists: true }, "status.converted.done": { $in: [null, false] }, "status.converted.createDate": { $lt: oneHourAgo } }; const oldConvertingCount = await db.collection('datafile').countDocuments(convertingQuery);
// Count completed convertruns in the last hour const completedQuery = { "status.exitCode": { $exists: true }, updateDate: { $gte: oneHourAgo } }; const recentCompletedCount = await db.collection('convertrun').countDocuments(completedQuery);
// Problem if there are old converting datafiles and no recent completions. const problem = (oldConvertingCount > 0) && (recentCompletedCount === 0); return problem; }
For When there are files to conver but nothing is converting
// if a convert file has been in a converting state for more than an hour, and there are no convert completes with last twenty minutes // Because files are files
// Load of file in convert state, none in converted state recently (N hours)
// if a convert file has been in a converting state for more than an hour, and there are no convert completes with last twenty minutes // Because files are files
// Load of file in convert state, none in converted state recently (N hours)
// check if there is a convertrun // check if there in no convert run // do copy files get loaded?
// Convert run has gone missing or was never created // {"status.converted.createDate": { "$exists": true }}, // and search for convert run in files and check which one does NOT have a convert run
// Number of convert in motion, number completed
// Create a notion with the various possibility, bad convertstatus, no convert run there should be, // Notion
With a window do we have data in converting and do NOT have any converted. Window needs to be long enough a window for the longest convert
Check the update dates on convertRuns within a period of time
datafile.status.converted.createDate value
Converted
{ "status.processed.done":true, $and:[{types:{$exists:true}}, {types:{$ne:[]}}, {format:{$exists:true}}, {length:{$gt:0}}], format:{$exists:true}, length:{$gt:0}, "status.converted.pid":{$exists:true}, "status.converted.done":{"$in":[null,false]} }
Within time window
Do have convertrun within updateDate > than datafile with updateDate
—-
Problem is False, False
if (a datafile is converting within n time) :: A true else false
if (a convertrun has updated within n time) :: B true else false
// nothing running A:: False B:: False No Error
// just finished running A:: False B:: True No Error
// just started running A:: True B:: True No Error
// File is converting, but no convertrun updates A:: True B:: False ERROR
A:: { "status.processed.done":true, $and:[{types:{$exists:true}}, {types:{$ne:[]}}, {format:{$exists:true}}, {length:{$gt:0}}], format:{$exists:true}, length:{$gt:0}, "status.converted.pid":{$exists:true}, "status.converted.done":{"$in":[null,false]} } Get oldest of above from datafile.status.converted.updateDate Is it within n time?
B:: Are there more than 0