0004471: DataExtractorService fails to set StagedResource for extracted file to DONE - SymmetricDS - Issue Tracker

ID	Project	Category	View Status	Date Submitted	Last Update

0004471	SymmetricDS	Bug	public	2020-07-09 15:34	2020-11-09 15:14

Reporter	pmarzullo	Assigned To	pmarzullo
Priority	normal
Status	closed	Resolution	fixed
Product Version	3.10.12
Target Version	3.10.13	Fixed in Version	3.10.13

Summary	0004471: DataExtractorService fails to set StagedResource for extracted file to DONE
Description	The DataExtractorService is unable to set the StagedResource for extracted file to the DONE state. The reason for this can be reproduced using the steps below. This is a problem when the number of files that need to be extracted for a table is large, and the synchronization catches up with the extract of the table and the synchronization can not find the extracted file (because the state is CREATE instead of DONE). The synchronization attempts to interrupt the extraction, then resets the batches for the table back to RQ status, and sets the sym_table_reload_request status back to NE. The problem occurs when the SymmetricDS node is running in clustered mode, and the initial load extract is running on another instance of the cluster than the synchronization, and the synchronization can not interrupt the initial load. The batches are reset to RQ, and the sym_table_reload_request is reset to NE, but the initial load extract continues on because it is running on the other node. The initial load extract then marks the rest of the batches for the current table to status NE, and marks the sym_table_reload_request status to OK when it finishes extracting. Then you are in a state where the beginning batches for the current table are in the RQ status, the rest of the batches for the table are in the NE status, and the reload request status is OK. The batches are stuck in the RQ status.
Steps To Reproduce	1. Start an initial load with the initial load extract running in the background. 2. Run the StagingManager job to clean the staging purge. 3. While the StagingManager clean is running, the initial load extract is unable to retrieve the StagedResource because of synchronized method clean() and synchronized block in find() in the StagingManager class. 4. When the StagingManager clean is complete, the file that was written to remains in the CREATE state, and the initial load extractor goes onto the next batch extract.
Tags	initial/partial load, staging

pmarzullo 2020-07-09 16:02 developer ~0001729	The change to the StagedResource was to undo some changes that I interpreted as network file share locking. That is not the case, so the old code is put back. The real change for this is in StagingManager find() method.

pmarzullo 2020-07-09 16:03 developer ~0001730	Code has been merged into 3.11 and 3.12 as well.

SymmetricDS: 3.10 57882a01 2020-07-09 11:56:57 Philip Marzullo Details Diff	0004471: DataExtractorService fails to set StagedResource for extracted file to DONE		Affected Issues 0004471
	mod - symmetric-io/src/main/java/org/jumpmind/symmetric/io/stage/StagedResource.java		Diff File
	mod - symmetric-io/src/main/java/org/jumpmind/symmetric/io/stage/StagingManager.java		Diff File

Date Modified	Username	Field	Change
2020-07-09 15:34	pmarzullo	New Issue
2020-07-09 15:34	pmarzullo	Status	new => assigned
2020-07-09 15:34	pmarzullo	Assigned To	=> pmarzullo
2020-07-09 15:34	pmarzullo	Tag Attached: initial/partial load
2020-07-09 15:34	pmarzullo	Tag Attached: staging
2020-07-09 15:34	pmarzullo	Relationship added	related to 0003250
2020-07-09 16:00		Changeset attached	=> SymmetricDS 3.10 57882a01
2020-07-09 16:02	pmarzullo	Note Added: 0001729
2020-07-09 16:03	pmarzullo	Status	assigned => resolved
2020-07-09 16:03	pmarzullo	Resolution	open => fixed
2020-07-09 16:03	pmarzullo	Fixed in Version	=> 3.10.13
2020-07-09 16:03	pmarzullo	Note Added: 0001730
2020-11-09 15:14	admin	Status	resolved => closed