BdpTaskAdapter

In case of setting a high concurrency number that causes submitting too many jobs at one time. We added 1 to a delay count each time a job is sent. The job will be held for a delay interval times this delay count before it is processed. In this way, even if there are so many jobs in a queue are submitted concurrently, the actual commands are delayed and processed one by one with the time interval. Once a job is finished, there will be a small waiting time to determine whether the job submitting events are frequent. If the event is not frequent, meaning that the delayCount remains the same after the waiting duration, the delayCount will be reset to 0.

This is a delay interval in milliseconds for concurrent job submission. This prevents submitting too many jobs concurrently, especially when there are too many concurrent jobs to be submitted. This can be set via options.

Using this property to understand whether the adapter is stopping. You may use it to prevent extra waiting in while loops.

This object stores the progress of batch tasks.

A queue object from https://www.npmjs.com/package/queue

This variable stores all the task objects.

This function emits the 'finish' event when the job exit code is 0 or the 'error' event when else. Note that calling this function does not stop the job process. This function only notify the adapter that the job is finished.

This function returns all jobIds.

This function returns the job definition object by giving the corresponding jobId.

This function initializes the whole adapter process and must be called to get the Adapter object instance.

This function can be called inside BdpTaskAdapter.jobOverrides when you implement the jobOverrides function. This function basically parses the recipe file (in yaml format) and overrides the JobDefinition object. Then, it returns the updated job object. (Note: the original job object will be mutated.)

An async function that construct each task object in the this.jobStore

This function checks job status and return the latest status object. If finished/exited jobs detected, emit finish/error event on the this.runningJobs.get(jobId).jobEmitter eventEmitter.

This private funciton handles the proxy registration and unregistration.

This internal function is called after the jobExitCallback function. This function stores the resulting stdout/stderr files and updates the status of all jobs.

This internal function will be called right after a job has exited (finished or errored) and before the jobExitCallback function. The function gets the remaining stdout/stderr messages, cleans up the streaming resources and unregisters the proxy.

This function loads the index.txt file in the options.taskLogFolder to check if jobs are finished with the exitCode of 0 (normal exit). If the job is finished, then it is skipped. This is used for pipeline resuming if the running pipelines stop unexpectedly.

This function is used to monitor job status and periodically checks job status.

This function trying to fetch remote jobs. This is perticularly useful when adapter process somehow closed and re-run the same adapter process. Instead of directly re-creation of non-fnihsing jobs, adapter trys to fetch the job histroy. Then, the adapter can see if there is a need to re-submit jobs.

An async function that used to run a single job.

The core function to initiate the queue to process tasks.

This private function sends messages to bdp-server with process.send.

This internal function waits until all job object are resolved (having their own exit codes).

Writing task progress to the the progress-index.txt file

You may using this in your adapter class, such as super._setOptions(options).

In case of setting a high concurrency number that causes submitting too many jobs at one time. We added 1 to a delay count each time a job is sent. The job will be held for a delay interval times this delay count before it is processed. In this way, even if there are so many jobs in a queue are submitted concurrently, the actual commands are delayed and processed one by one with the time interval. Once a job is finished, there will be a small waiting time to determine whether the job submitting events are frequent. If the event is not frequent, meaning that the delayCount remains the same after the waiting duration, the delayCount will be reset to 0.

This is a delay interval in milliseconds for concurrent job submission. This prevents submitting too many jobs concurrently, especially when there are too many concurrent jobs to be submitted. This can be set via options.

Using this property to understand whether the adapter is stopping. You may use it to prevent extra waiting in while loops.

This object stores the progress of batch tasks.

A queue object from https://www.npmjs.com/package/queue

This variable stores all the task objects.

This function emits the 'finish' event when the job exit code is 0 or the 'error' event when else. Note that calling this function does not stop the job process. This function only notify the adapter that the job is finished.

This function returns all jobIds.

This function returns the job definition object by giving the corresponding jobId.

This function initializes the whole adapter process and must be called to get the Adapter object instance.

This function can be called inside BdpTaskAdapter.jobOverrides when you implement the jobOverrides function. This function basically parses the recipe file (in yaml format) and overrides the JobDefinition object. Then, it returns the updated job object. (Note: the original job object will be mutated.)

An async function that construct each task object in the this.jobStore

This function checks job status and return the latest status object. If finished/exited jobs detected, emit finish/error event on the this.runningJobs.get(jobId).jobEmitter eventEmitter.

This private funciton handles the proxy registration and unregistration.

This internal function is called after the jobExitCallback function. This function stores the resulting stdout/stderr files and updates the status of all jobs.

This internal function will be called right after a job has exited (finished or errored) and before the jobExitCallback function. The function gets the remaining stdout/stderr messages, cleans up the streaming resources and unregisters the proxy.

This function loads the index.txt file in the options.taskLogFolder to check if jobs are finished with the exitCode of 0 (normal exit). If the job is finished, then it is skipped. This is used for pipeline resuming if the running pipelines stop unexpectedly.

This function is used to monitor job status and periodically checks job status.

This function trying to fetch remote jobs. This is perticularly useful when adapter process somehow closed and re-run the same adapter process. Instead of directly re-creation of non-fnihsing jobs, adapter trys to fetch the job histroy. Then, the adapter can see if there is a need to re-submit jobs.

An async function that used to run a single job.

The core function to initiate the queue to process tasks.

This private function sends messages to bdp-server with process.send.

This internal function waits until all job object are resolved (having their own exit codes).

Writing task progress to the the progress-index.txt file

You may using this in your adapter class, such as super._setOptions(options).