View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0006513 | SymmetricDS | Bug | public | 2024-06-21 20:20 | 2024-07-05 14:00 |
| Reporter | pmarzullo | Assigned To | pmarzullo | ||
| Priority | normal | ||||
| Status | closed | Resolution | fixed | ||
| Product Version | 3.15.7 | ||||
| Target Version | 3.15.8 | Fixed in Version | 3.15.8 | ||
| Summary | 0006513: File Sync Tracker changes to improve performance and to collect completely the set of files | ||||
| Description | File Sync Tracker does not collect all files in a location when the files show up with a modified date earlier than the sym_context value with key file.sync.fast.scan.track.time If we eliminate the date/time check in FileTriggerFileModifiedListener.addSnapshot() , we will be able to track all files that show up and compare with what is in sym_file_snapshot. File Sync Tracker, when saving the file snapshot records into sym_file_snapshot, executes an update first, and if 0 records are updated, then an insert is attempted. This slows down the file sync tracking tremendously when the number of records in sym_file_snapshot get to be a large number. For example, when the file sync tracker is run for the first time, the update first followed by an insert if necessary is very fast (about 1 second for 1000 records), but as the number of records increase in sym_file_snapshot, it slows down tremendously (100000 records in sym_file_snapshot, it takes over 30 seconds to update/insert 1000 records). We already compared the file system file snapshot with what is in sym_file_snapshot and marked the event types as necessary (CREATE, MODIFY, DELETE). We should not execute the update if the event type is a CREATE and just go to the insert. For example, with 1000000 files on the file system, the amount of time it took with the current algorithm was 6 hours. With the new algorithm, it took 35 minutes. | ||||
| Tags | file sync | ||||
|
|
Patch located here (only good for 3.15.7): https://downloads.jumpmind.com/symmetricds/patches/3.15/patch-0006513.jar |
|
SymmetricDS: 3.15 3b6fd9f5 2024-06-21 20:26:17 Details Diff |
0006513: File Sync Tracker changes to improve performance and to collect completely the set of files |
Affected Issues 0006513 |
|
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/file/FileTriggerFileModifiedListener.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/FileSyncService.java | Diff File | ||
|
SymmetricDS: 3.15 8ca8261c 2024-07-05 13:26:23 Details Diff |
0006513: File Sync Tracker changes to improve performance and to collect completely the set of files |
Affected Issues 0006513 |
|
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/file/FileTriggerFileModifiedListener.java | Diff File | ||
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2024-06-21 20:20 | pmarzullo | New Issue | |
| 2024-06-21 20:20 | pmarzullo | Status | new => assigned |
| 2024-06-21 20:20 | pmarzullo | Assigned To | => pmarzullo |
| 2024-06-21 20:20 | pmarzullo | Tag Attached: file sync | |
| 2024-06-21 20:31 | pmarzullo | Status | assigned => resolved |
| 2024-06-21 20:31 | pmarzullo | Resolution | open => fixed |
| 2024-06-21 20:31 | pmarzullo | Fixed in Version | => 3.15.8 |
| 2024-06-21 21:00 | pmarzullo | Changeset attached | => SymmetricDS 3.15 3b6fd9f5 |
| 2024-06-24 14:51 | pmarzullo | Note Added: 0002468 | |
| 2024-07-05 14:00 | pmarzullo | Changeset attached | => SymmetricDS 3.15 8ca8261c |
| 2024-08-16 17:45 | admin | Status | resolved => closed |