View Revisions: Issue #2777
| Summary | 0002777: Initial load waits for extraction that never happens | ||
|---|---|---|---|
| Revision | 2016-09-08 16:18 by elong | ||
| Description | The initial load extractor was enabled in 3.8, which uses a background thread to extract batches. The foreground thread checks if a batch is ready to send. If its staging file is missing, it reschedules the batch for extraction. This happens if the user deletes staging files or the initial load takes a long time and the staging manager deletes the files. There is race condition between the foreground and background thread that causes the batch status to set to RQ for extraction request but the sym_extract_request is set to OK, so extraction will never happen. The user sees this in the log continuously: Batch {} is not ready for delivery. It is currently scheduled for extraction |
||
| Revision | 2016-09-09 13:33 by elong | ||
| Description | The initial load extractor was enabled in 3.8, which uses a background thread to extract batches. The foreground thread checks if a batch is ready to send. If its staging file is missing, it reschedules the batch for extraction. This happens if the user deletes staging files or the initial load takes a long time and the staging manager deletes the files. If one of the first batches has been sent and OK'ed, the extractor will not re-extract batches and gets into an infinite loop Extracting batches for request 4. Starting at batch 96. Ending at batch 106 Batch {} is not ready for delivery. It is currently scheduled for extraction Batch 1-106 is marked as ready but it has been deleted. Rescheduling it for extraction |
||