When designing and creating any new batch job that will process large amounts of data, the following methodologies should be understood. Use the methodology below that best aligns with the requirements.

Nextworld batch job methodologies are centered around the following types:

  • Record batch processing - System administrators can enable record batch processing to efficiently process records regardless of the volume of transactions in a table. This is a good candidate if there could be record contention and the process is trying to update the same record multiple times.

Recurring batch job best practices

These best practices ensure that recurring batch jobs are configured and maintained consistently throughout Nextworld.

Recurring batch jobs should be used for an on-going process that occurs on a pre-defined frequency and with a pre-defined set of criteria. 

Recurring batch jobs must:

  • Run as a background task. 
  • Have at least one criteria in your application. This limits the number of records processed. You should never use an open fetch to update all the records in an application.
  • Be schedulable. 
  • Be able to restart if the job ends abnormally.

Recurring batch jobs should:

  •  Use versions so users can save their parameters to run on a periodic basis, either through an application or through the enterprise scheduler. 
  • Process each record with individual logic blocks called outside the transaction boundary. This creates smaller NATE boundaries which process as many records as possible and minimizes the amount of record the user needs to correct. Use a field to determine if the record is successfully processed and include in the fetch to avoid reprocessing the same records. 
  • Consider how multiple jobs which run at the same time, on the same record, will work. 
  • Consider the effects of submitting the same job twice. If the Queue Background Logic Block action is used, give it a Sequential Execution Key so the first execution is completed before the second execution starts.  A field in your table should be set so your second execution will not reprocess the same records. 
  • Consider if the job should run on demand from an application.

Error Handling

When a batch job fails, error handling helps a user determine what went wrong. The error is inserted into an error log that is accessible to the user through an application link. The error log provides several key pieces of information, such as the batch version that failed, the record that failed to update, and the reason the record failed to update.

Data Items

There are several common fields used in batch jobs, including:

  • BatchVersion—Used as the unique identifier for jobs if your job is scheduled. This is a required text field. 
  • Description—Used to specify the purpose for this batch version.  
  • Selection Criteria—Used to specify the selection criteria for your batch job. These fields are often identical to the fields of the table being updated. 
  • Input Parameters—Used to specify any additional data used to help update the records. 

Table

All tables involved in the batch job, such as the Error and Batch Version tables, must be of type Main.

The Batch Version table must include:

  • BatchVersion data item set up as a unique index and as the Primary Search Field on the table. 
  • Selection Criteria to limit the number of records to process.
  • Input Parameters to specify which fields should be updated or altered in the batch process. 

The Error table must include the following data items:

  • BatchVersion
  • BatchJobDateTime included in the index with a descending sort. 
  • ErrorsMessageDescription
  • ErrorsDataItemName
  • ErrorsMessageType
  • ErrorsSeverity
  • ErrorsInputValues
  • ErrorsLogicBlock
  • ErrosSource
  • ErrorsFunctionNumber

Application 

The application layout for the batch job should mimic the layout of the application that is used to manually update the same records. Consider creating sections of the fields as groups. For example, Batch Version, Selection Criteria, and Input Parameters.

Other configurations include the:

  • List Form—Add the BatchVersion data item first, and mark as Primary. All other fields should be listed as available or available-hidden. 
  • Detail Form—Configure the sections. 
    • In the batch job identifier section, include the BatchVersion and Description data items. 
    • In the selection criteria section, if a description field for a table lookup is provided.  verify a synonym exists or is added to the added to the app with the name "<Table Lookup> Description".
    • An input parameters section is optional. 
  • Application Link—Add to manually submit the job. Label the row action based on what the job is doing. For example, Calculate Lead Times. Add an Error log, if capturing and logging errors
  • Application Setting—Select the Read Only Application checkbox. 

The Error Log should be a List Only min app, with a large pop-up size. Configurations for the list form include:

  • BatchVersion listed first - primary
  • BatchJobDateTime - available
  • ErrorsMessageDescription - available
  • ErrorsInputValues - available
  • ErrorsDataItemName - available-hidden
  • ErrorsMessageType - available-hidden
  • ErrorsSeverity - available-hidden
  • ErrorsLogicsBlock - available-hidden
  • ErrorsSource - available-hidden
  • ErrorsFunctionNumber - available-hidden
  • Disable and hide all form and row actions

Logic Blocks

Generally, you need several logic blocks for your batch jobs, including a:

  • Initial logic block—Built as a background task which should not use a NATE container during execution. Additionally, this logic block should:
    • Give an error to the user if there are issues with the batch job.
    • Call the second logic block. 
    • Fetch messages. If they aren't empty, give an error that causes the background task to error and send a notification to the user. There may need to be conditionals to match the received message to the error which should be given. If they are empty, give a success background summary notification to the user.  
  • Fetch records logic block—Finds records that satisfy the selection criteria. This logic block should be called in a new database transaction by the calling logic block in order to find valid records, then pass the records with the correct input parameters to be updated. Additionally, this logic block should:
    • Consider using chunk fetch option  in the Fetch Record action. This helps with high volume processing. 
    • Insert any error messages to the error log, then delete the original message. Error messages are created when a record fails to update. 
    • Call the third logic block. If there is an error on the third logic block, this logic block may choose to not display the error to allow the processing to continue. 
  • Update records logic block—Updates the records passed to it, potentially with help from the input parameters. This logic block should be called in a new database transaction by the logic block which found the records. Additionally, if there are any error messages, this logic block should provide input values information to help the user identify which record failed.  

Scheduling a job

A user may want a batch version to be scheduled to run periodically. This can be accomplished through the Job Scheduler application which allows the logic to be run on records on a recurring scheduled. 

To set up a batch job, in the Job Scheduler application, on the Jobs page, you must:

  • Reference the jobs by the Batch Version when you create the scheduled job record and add jobs. Then:
    • Select Logic Block for the type.
    • Select the name of the initial logic block you created which calls the other logic blocks. This should populate the table field with the table the logic block is built over.
    • Assuming BatchVersion is the primary search field, Logic Block Driver Records should be able to reference records by Batch Version.
    • Do not make the date required, if the batch date is empty, then it should default to today's date.

Recurring batch job naming conventions

These best practices ensure that recurring batch jobs are configured and maintained consistently throughout Nextworld.

Recurring batch jobs 

  • Table name: <field/update (optional)><table>BatchJobs.  For example, LeadTimeItemProductDefinitionsBatchJobs
  • Application name: <process (optional)><table>BatchJobs.  For example,  LeadTimeProductDefinitionsBatchJobs
  • Application setting name: <Application name>Setting
  • Process logic block name:  Process<batch job table>. For example, ProcessLeadTimeProdcutDefinitionsBatchJobs
  • Updating logic block name: <Action><batch job table>. For example, CalculateLeadTimeProductDefintion

Error Log

  • Table name:  <Batch Job Table>ErrorLog. For example, LeadTimeItemOrgUnitsBatchJobsErrorLog
  • Application name: <Batch Job App>ErrorLog. For example, LeadTimeItemOrgUnitsBatchJobsErrorLog 
  • Application setting name: <Error log application>Setting
  • Error handling logic block name:  <Action><batch job table>. For example,  CalculateLeadTimeProductDefinitionsBatchJobs

Ad-hoc batch job best practices

These best practices ensure that ad-hoc batch jobs are configured and maintained consistently throughout Nextworld.

Ad-hoc batch jobs should be used when an end user needs to create a one-time batch process at a specific moment in time for a specific purpose or situation.  This methodology would be used when the situation, point in time, or criteria could always be different anytime someone needs to run the batch. This situation always involves an end user that needs to interact with an application to search and select specific records and use that selection to run a batch job that runs in the background. This method differs from the Recurring batch job best practices that would run a batch job at regular intervals with minimal user interaction. 

Since ad-hoc batch jobs require more user interaction, the application standards between ad-hoc and recurring batch jobs are different. 

Applications should: 

  • Mimic the layout of the application that is used to manually update the same records. Each ad-hoc batch job will look more like the source application, rather than all ad-hoc applications looking like each other. 
  • Give any or all of the following options for an end user to select the records to process.
    • Allow records to be selected using header criteria - Fetch Records action using header criteria
    • Allow records to be selected using the column filter - Fetch Records action using a combination of header criteria and Received Filter
    • Allow records to be selected using the selection radio button - Fetch Records action using a combination of header criteria and Received Selection
      • There is a limitation that only 50 records can be selected from the application, which is why we recommend offering several options for an end user to select a list of records to process. Only having this option is not viable for most batch jobs. 
    • See the Fetch Record action and Logic block expressions for more information.

Many of the same rules apply to ad-hoc batch jobs that apply to recurring batch jobs. 

Ad-hoc batch jobs must:

  • Run as a background task once the records to be used have been selected. However, it's also possible to take into account the number of records selected and only submit the process as a background task when it's a large selection. 
  • Have at least one, hopefully more, required criteria in your application. This limits the number of records processed. You should never allow an end user to select all records in a table. 
  • Be able to restart or re-run the job if it ends abnormally.

Ad-hoc batch jobs should:

  • Consider the transaction boundary for each record and ensure it is as small as possible. 
  • Use a field to determine if the record is successfully processed and include that field in the fetch to avoid reprocessing the same records. 
  • Consider how multiple jobs which run at the same time, on the same record, will work. Race conditions and record locks need to be prevented.
  • Consider the effects of submitting the same job twice. If the Queue Background Logic Block action is used, give it a Sequential Execution Key so the first execution is completed before the second execution starts.  A field in your table should be set so your second execution will not reprocess the same records. 
  • Have error handling that helps a user determine what went wrong. Make sure to include any information that can help the user find additional background jobs that could be kicked off from the UI. It can be hard for a user to find an error from a background task if it is not displayed in the UI itself. Give them direction on how to find that information. 

Some examples of existing ad-hoc applications used for batch processing are:

  • Open Payables Transactions (PaymentCreationRunPayables) - OpenPayables (inline application used by Open Payables Transactions) 
    • Can load details into the Open Payables inline application by entering the required fields on the header. The records are limited by the required fields in the header which are used as minimum filter criteria. 
    • Uses a common context between the header and inline applications. See Common Context configuration for more information.
    • Can further filter the detail list by using the column filters or row selection. If the end user uses the row selection, they will be limited to 50 records.  
    • Once the detail records are loaded, filtered, or selected, the Create Payment Batch for Records action can be used to process the records in a background task. 
  • Escheatments - EscheatmentsBulkPrompt (Logic prompt application) 
    • Can use the list view to filter or select the records and use the Escheatments Bulk Update action.
    •  The Escheatments Bulk Update action uses a logic block prompt to allow the end user to enter in criteria prior to updating the records. See Logic Block Prompts for more information.
      • This logic block prompt allows the user to specify whether the bulk update should Update Selected or Update Filtered records. If Update Selected is used, they will be limited to 50 records. 
      • This logic block prompt also allows the user to specify whether it should move the workflow to Process. This would work similiar to the bulk workflow option in the list view, but they would be limited to 50 records.