View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0006722 | SymmetricDS | Bug | public | 2025-02-10 13:47 | 2025-02-14 14:27 |
| Reporter | elong | Assigned To | elong | ||
| Priority | normal | ||||
| Status | closed | Resolution | fixed | ||
| Product Version | 3.15.0 | ||||
| Target Version | 3.15.13 | Fixed in Version | 3.15.13 | ||
| Summary | 0006722: Purge of batches before data causes appearance of stranded data | ||||
| Description | Purge service is clearing outgoing batches and data events without clearing the corresponding data rows, causing the appearance of stranded data. Stranded data is meant to be data that routing somehow missed and will never route (like a commit that was delayed longer than the window to watch gaps). Starting in 3.15, stranded data is recaptured to attempt to resync and repair it, so the system should be careful to purge evenly without causing the appearance of stranded data. The following problems need fixed: 1. A high batch ID can have a low data ID and vice-versa, so the algorithm of getting the max data ID for each channel's max batch ID doesn't always work. It has to get the max data ID for the entire batch range. 2. A data gap may fill in later, after the purge window for it has already passed, but the purge can still clear outgoing batches, causing the appearance of stranded data. If it also checks for the min data ID of the entire batch range, it can push back the context index for purging data. 3. For the slow purge algorithm (where it joins and checks the batch is OK), it gets the first not-OK batch and adds one to skip it to determine the range, but if that batch is the end of the range, then the context batch ID is not updated to advance the index for the next run. | ||||
| Steps To Reproduce | High concurrency that creates gaps. The SymmetricProH2 demo reproduces the issue within 30 minutes. It seems to also help to disable a channel for some time and re-enable it. | ||||
| Tags | purge | ||||
|
SymmetricDS: 3.15 158f4ccc 2025-02-13 17:45:04 Committer: GitHub Details Diff |
6722: Purge of batches before data causes appearance of stranded data (0000226) 0006722: Purge of batches before data causes appearance of stranded data. Prevent NPE |
Affected Issues 0006722 |
|
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataLoaderService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/PurgeService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/PurgeServiceSqlMap.java | Diff File | ||
| mod - symmetric-core/src/test/java/org/jumpmind/symmetric/service/impl/PurgeServiceTest.java | Diff File | ||
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2025-02-10 13:47 | elong | New Issue | |
| 2025-02-10 13:47 | elong | Status | new => assigned |
| 2025-02-10 13:47 | elong | Assigned To | => elong |
| 2025-02-10 13:47 | elong | Tag Attached: purge | |
| 2025-02-13 17:45 | pbelov | Status | assigned => resolved |
| 2025-02-13 17:45 | pbelov | Resolution | open => fixed |
| 2025-02-13 17:45 | pbelov | Fixed in Version | => 3.15.13 |
| 2025-02-13 18:00 | elong | Changeset attached | => SymmetricDS 3.15 158f4ccc |
| 2025-02-14 14:27 | pbelov | Note Added: 0002627 | |
| 2025-02-14 14:27 | pbelov | Status | resolved => closed |