0004546: Data Extractor gets "java.lang.IllegalStateException: There is no content to read" from staging - SymmetricDS - Issue Tracker

ID	Project	Category	View Status	Date Submitted	Last Update

0004546	SymmetricDS	Bug	public	2020-09-14 19:57	2020-09-23 20:34

Reporter	elong	Assigned To	elong
Priority	normal
Status	closed	Resolution	fixed
Product Version	3.11.11
Target Version	3.11.12	Fixed in Version	3.11.12

Summary	0004546: Data Extractor gets "java.lang.IllegalStateException: There is no content to read" from staging
Description	After optimizing concurrency in 0004472, a race condition was introduced where staging manager can remove a staged resource before it is push or pulled. When purging resources based on batches (stream.to.file.purge.on.ttl.enabled=false), the staging manager gets a list of batches from the database at the beginning of its run. As it looks through staged resources, if the resource does not have a corresponding batch from the list, it will remove the resource. The problem is that resources can be created while the job is in the middle of running, and those batches aren't on the list. To fix, we'll make a note of the start time of the job, and resources must have a last modified time before the job start time to be eligible for purging. 2020-09-11 17:00:00,007 INFO [corp-000] [StagingManager] [corp-000-job-18] Cleaning staging... 2020-09-11 17:00:00,050 INFO [corp-000] [StagingManager] [corp-000-job-18] Purged 224 staging files, freed 187 KB of disk space. 2020-09-11 17:00:00,050 INFO [corp-000] [StagingManager] [corp-000-job-18] Finished cleaning staging in 0 seconds. 2020-09-11 17:02:59,541 ERROR [corp-000] [DataExtractorService] [corp-000-push-default-5] Failed to extract batch 003-333923 StackTraceKey.init [IllegalStateException:427582167] java.lang.IllegalStateException: There is no content to read. Memory buffer was empty and /export/home/postgres/symmetric-ds/tmp/corp-000/outgoing/common/923/0000333923.create was not found. at org.jumpmind.symmetric.io.stage.StagedResource.getReader(StagedResource.java:246) at org.jumpmind.symmetric.service.impl.DataExtractorService.transferFromStaging(DataExtractorService.java:1481) at org.jumpmind.symmetric.service.impl.DataExtractorService.sendOutgoingBatch(DataExtractorService.java:1440) at org.jumpmind.symmetric.service.impl.DataExtractorService.extract(DataExtractorService.java:789) at org.jumpmind.symmetric.service.impl.DataExtractorService.extract(DataExtractorService.java:577) at org.jumpmind.symmetric.service.impl.PushService.pushToNode(PushService.java:206) at org.jumpmind.symmetric.service.impl.PushService.execute(PushService.java:167) at org.jumpmind.symmetric.service.impl.NodeCommunicationService$1.run(NodeCommunicationService.java:519) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Steps To Reproduce	Add break point in debugger in BatchStagingManager to help with timing Run Stage Management job Change data and observe a new batch that is push/pulled Allow Stage Management job to continue, and it will immediately delete the new batch's stage file
Tags	staging

Date Modified	Username	Field	Change
2020-09-14 19:57	elong	New Issue
2020-09-14 19:57	elong	Status	new => assigned
2020-09-14 19:57	elong	Assigned To	=> elong
2020-09-14 19:57	elong	Tag Attached: staging
2020-09-14 19:57	elong	Issue generated from: 0004545
2020-09-14 19:57	elong	Relationship added	related to 0004545
2020-09-14 19:58	elong	Relationship added	related to 0004472
2020-09-14 19:58	elong	Status	assigned => resolved
2020-09-14 19:58	elong	Resolution	open => fixed
2020-09-14 19:58	elong	Fixed in Version	=> 3.11.12
2020-09-14 19:59	elong	Issue cloned: 0004547
2020-09-14 19:59	elong	Relationship added	related to 0004547
2020-09-23 20:34	admin	Status	resolved => closed