View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0006685 | SymmetricDS | Bug | public | 2024-12-23 19:36 | 2025-01-03 20:06 |
| Reporter | mdrouard | Assigned To | pbelov | ||
| Priority | normal | ||||
| Status | closed | Resolution | fixed | ||
| Product Version | 3.15.0 | ||||
| Target Version | 3.15.12 | Fixed in Version | 3.15.12 | ||
| Summary | 0006685: Purge Service recaptures an Insert for stranded data after record had been deleted | ||||
| Description | Purge Service recaptures an Insert for stranded data after record had been deleted If a transaction is active for longer than the duration of the parameter routing.stale.dataid.gap.time.ms, data will be marked as stranded. Data in such long database transaction will not be processed normally, instead this data will go through the recapture process when the purge job runs. If there is a stranded Insert (or Update) but the original row had been deleted, then the Purge job will incorrectly keep the original insert (or update) command. This results in data being present on target database and not at the source. | ||||
| Steps To Reproduce | 0. Alter routing.stale.dataid.gap.time.ms to be a period that you are willing to wait to create a stranded data (data gap). 1. Turn off auto-commit on SQL transactions (for testing). 2. Run an insert on a source table and do not commit just yet. 3. Wait for time, exceeding the routing.stale.dataid.gap.time.ms value. 4. Commit the Insert transaction. 5. Run a Delete command targeting the same record that was inserted and commit immediately. 6. Allow the Routing job to run and pick up the data gap. 7. Run manually or wait until the Purge Outgoing job runs. 9. Select from the table on source and client and compare the difference. Observe record missing on the source database but present on the target. | ||||
| Additional Information | Set up: two SymmetricDS engines with client-server push configuration using two catalogs on the same PostgreSQL database: qaserver catalog for SymmetricDS server engine qaclient catalog for SymmetricDS client engine SET routing.stale.dataid.gap.time.ms=8000 (8 seconds) SET Routing job to run every 5 seconds SET Push job to run every 4 seconds SET Purge Outgoing job to only run at midnight -- Table with test data: CREATE TABLE qaserver.atest_6685_stale_gap ( cl1key INTEGER NOT NULL, clVARCHAR500 VARCHAR(500) NOT NULL, PRIMARY KEY (cl1key) ); -- Test slow INSERT: BEGIN transaction; INSERT INTO qaserver.atest_6685_stale_gap VALUES( 44, '44slow'); SELECT pg_sleep(15); COMMIT transaction; -- Test slow UPDATE: BEGIN transaction; UPDATE qaserver.atest_6685_stale_gap SET clVARCHAR500= '44slow-updated44' where cl1key=44; SELECT pg_sleep(15); COMMIT transaction; -- Test slow DELETE: BEGIN transaction; DELETE FROM qaserver.atest_6685_stale_gap where cl1key=44; SELECT pg_sleep(15); COMMIT transaction; -- Compare data in tables: SELECT 'source' as tbl, * FROM qaserver.atest_6685_stale_gap UNION ALL SELECT 'target' as tbl, * FROM qaclient.atest_6685_stale_gap order by 1,2; -- Review batches and data entries: SELECT b.batch_id, case when b.status is null then 'UNrouted' else b.status end as status , d.* FROM qaserver.sym_data d left outer join qaserver.sym_data_event e on e.data_id = d.data_id left outer join qaserver.sym_outgoing_batch b on e.batch_id = b.batch_id WHERE table_name='atest_6685_stale_gap' and d.data_id >=6961 or d.data_id = (SELECT Max(data_id) FROM qaserver.sym_data where table_name<>'atest_6685_stale_gap' ) order by data_id desc ; | ||||
| Tags | purge | ||||
|
SymmetricDS: 3.15 27505f88 2025-01-02 21:25:44 Committer: GitHub Details Diff |
0006685 Fix recapture of stale data when row is missing and refactor reCaptureData 0006685 Fix recapture of stale data when row is missing and refactor reCaptureData * Fix and Refactor DataService.reCaptureData - add a sub-method fetchRecapturedData to process individual stale Data objects * Add 1k record batching to DataService.reCaptureData |
Affected Issues 0006685 |
|
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataService.java | Diff File | ||
| mod - symmetric-core/src/test/java/org/jumpmind/symmetric/service/impl/DataServiceTest.java | Diff File | ||
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2024-12-23 19:36 | mdrouard | New Issue | |
| 2024-12-23 19:36 | mdrouard | Tag Attached: purge | |
| 2024-12-23 19:36 | mdrouard | Description Updated | View Revisions |
| 2024-12-23 19:39 | mdrouard | Assigned To | => pbelov |
| 2024-12-23 19:39 | mdrouard | Status | new => assigned |
| 2024-12-26 22:08 | pbelov | Status | assigned => confirmed |
| 2024-12-26 22:08 | pbelov | Summary | Purge Service missing Delete on long running transactions => Purge Service recaptures an Insert for stranded data after record had been deleted |
| 2024-12-26 22:08 | pbelov | Description Updated | View Revisions |
| 2024-12-26 22:08 | pbelov | Steps to Reproduce Updated | View Revisions |
| 2024-12-27 13:43 | pbelov | Additional Information Updated | View Revisions |
| 2025-01-02 21:27 | pbelov | Status | confirmed => resolved |
| 2025-01-02 21:27 | pbelov | Resolution | open => fixed |
| 2025-01-02 21:27 | pbelov | Fixed in Version | => 3.15.12 |
| 2025-01-02 22:00 | pbelov | Changeset attached | => SymmetricDS 3.15 27505f88 |
| 2025-01-03 20:06 | pbelov | Note Added: 0002573 | |
| 2025-01-03 20:06 | pbelov | Status | resolved => closed |