View Revisions: Issue #5906
| Summary | 0005906: Non transactional initial load setup can cause load events to be missed ini table_reload_status | ||
|---|---|---|---|
| Revision | 2023-06-29 20:49 by pmarzullo | ||
| Description | When the following parameter is set to false: datareload.batch.insert.transactional=false it is possible for the events that are being inserted into the database for the load are routed and synchronized to the target before the counts in table_reload_status are completed. If this occurs, then the state of those batches that are completed before the table_reload_status is completed will be lost. For example, if there is a load of one table, and a setup batch is created to delete the records for that table before sending the load, the setup batch can be routed and synchronized to the target before the table_reload_status table gets the range of batches that are created for the data load. This will cause the update of the table_reload_status for that setup batch to not update the record because the batch ID for the setup batch does not fall before the start batch ID for the load batches (because both the start_batch_id and end_batch_id are set to 0 when the setup batch is acknowledged). To fix this, we should set the setup, load, and finalize batches, as well as the extract requests, to a status of LS for Load Setup. Then when all information is completely set up in the table_reload_status table for the load, then the status values can be set to NE for the setup and finalize batches, and for the extract requests, and the load batches can be set to NE when not using the initial load extractor job or they can be set to RQ when the initial load extractor job is used. |
||
| Revision | 2023-06-29 20:47 by pmarzullo | ||
| Description | When the following parameter is set to false: datareload.batch.insert.transactional=false it is possible for the events that are being inserted into the database for the load are routed and synchronized to the target before the counts in table_reload_status are completed. If this occurs, then the state of those batches that are completed before the table_reload_status is completed will be lost. For example, if there is a load of one table, and a setup batch is created to delete the records for that table before sending the load, the setup batch can be routed and synchronized to the target before the table_reload_status table gets the range of batches that are created for the data load. This will cause the update of the table_reload_status for that setup batch to not update the record because the batch ID for the setup batch does not fall between the start and end batch IDs for the load batches (because both the start_batch_id and end_batch_id are set to 0 when the setup batch is acknowledged). To fix this, we should set the setup, load, and finalize batches, as well as the extract requests, to a status of LS for Load Setup. Then when all information is completely set up in the table_reload_status table for the load, then the status values can be set to NE for the setup and finalize batches, and for the extract requests, and the load batches can be set to NE when not using the initial load extractor job or they can be set to RQ when the initial load extractor job is used. |
||