View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0006688 | SymmetricDS Pro | New Feature | public | 2025-01-02 15:34 | 2025-03-17 20:15 |
| Reporter | elong | Assigned To | elong | ||
| Priority | normal | ||||
| Status | closed | Resolution | fixed | ||
| Product Version | 3.16.0 | ||||
| Target Version | 3.16.0 | Fixed in Version | 3.16.0 | ||
| Summary | 0006688: Dynamic reload queue to scale initial load | ||||
| Description | To make it easier for the initial load to scale across multiple threads, introduce a dynamic reload queue that assigns tables to a pool of threads. The number of threads is controlled by a parameter, one for extract threads and one for push/pull threads. All batches for a table are assigned to the same thread so there is no contention for the table. The initial load extract job makes assignments of batches to be extracted and loaded up to the max batches to sync, which allows for adjustment of the parameters during the load. When the reload channel is given the queue name of reload, this feature is enabled, which is the default configuration of the channel. Dynamic queues are internally named reload!1, reload!2, reload!3, etc based on the thread number assignment. The initial.load.queue.use.all.threads parameter controls if all threads can be used by an initial load with a setting of "auto" that uses all threads if table constraints have been deferred. initial.load.extract.thread.per.server.count=20 initial.load.queue.sync.thread.count=20 initial.load.queue.use.all.threads=auto | ||||
| Tags | initial/partial load, performance | ||||
|
SymmetricDS: 3.16 7bab5e18 2025-01-06 21:02:41 Committer: GitHub Details Diff |
0006688: Dynamic reload queue to scale initial load (0000217) 0006688: Dynamic reload queue to scale initial load |
Affected Issues 0006688 |
|
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/AbstractSymmetricEngine.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/cache/CacheManager.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/cache/ICacheManager.java | Diff File | ||
| add - symmetric-core/src/main/java/org/jumpmind/symmetric/cache/OutgoingBatchCache.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/common/Constants.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/common/ParameterConstants.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/db/DatabaseUpgradeListener.java | Diff File | ||
| add - symmetric-core/src/main/java/org/jumpmind/symmetric/ext/IReloadQueueThreadAssigner.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/extract/MultiBatchStagingWriter.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/extract/SelectFromSymDataSource.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/io/stage/SimpleStagingDataWriter.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/model/ExtractRequest.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/model/NodeSecurity.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/model/OutgoingBatch.java | Diff File | ||
| add - symmetric-core/src/main/java/org/jumpmind/symmetric/model/ReadyChannels.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/IIncomingBatchService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/INodeService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/IOutgoingBatchService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/ITriggerRouterService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/ConfigurationService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataExtractorService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataExtractorServiceSqlMap.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataLoaderService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/FileSyncExtractorService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/IncomingBatchService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/InitialLoadService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/NodeCommunicationService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/NodeService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/NodeServiceSqlMap.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/OutgoingBatchService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/OutgoingBatchServiceSqlMap.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/PullService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/PushService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/RouterService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/TriggerRouterService.java | Diff File | ||
| add - symmetric-core/src/main/java/org/jumpmind/symmetric/util/QueueThread.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/web/WebConstants.java | Diff File | ||
| mod - symmetric-core/src/main/resources/symmetric-default.properties | Diff File | ||
| mod - symmetric-core/src/main/resources/symmetric-schema.xml | Diff File | ||
| mod - symmetric-core/src/test/java/org/jumpmind/symmetric/service/impl/DataServiceTest.java | Diff File | ||
| mod - symmetric-core/src/test/java/org/jumpmind/symmetric/service/impl/MockNodeService.java | Diff File | ||
| mod - symmetric-db/src/main/java/org/jumpmind/db/sql/Row.java | Diff File | ||
| mod - symmetric-io/src/main/java/org/jumpmind/symmetric/io/data/writer/DefaultDatabaseWriter.java | Diff File | ||
| mod - symmetric-server/src/main/java/org/jumpmind/symmetric/web/PullUriHandler.java | Diff File | ||
| mod - symmetric-server/src/main/java/org/jumpmind/symmetric/web/PushUriHandler.java | Diff File | ||
|
SymmetricDS: 3.16 680c309b 2025-01-10 18:23:26 Details Diff |
0006688: don't use ready queues if only 1 queue |
Affected Issues 0006688 |
|
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/cache/CacheManager.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/cache/ConfigurationCache.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/cache/ICacheManager.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/cache/OutgoingBatchCache.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/IConfigurationService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/AbstractService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/ConfigurationService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/PullService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/PushService.java | Diff File | ||
| mod - symmetric-server/src/main/java/org/jumpmind/symmetric/web/PullUriHandler.java | Diff File | ||
|
SymmetricDS: 3.16 a46b865f 2025-02-13 13:37:23 evan-miller-jumpmind Details Diff |
0006688: Set default value of trigger.create.before.initial.load parameter back to true |
Affected Issues 0006688 |
|
| mod - symmetric-core/src/main/resources/symmetric-default.properties | Diff File | ||
|
SymmetricDS: 3.16 5233c204 2025-03-03 21:49:28 Details Diff |
0006688: add flush of ready queues |
Affected Issues 0006688 |
|
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/cache/CacheManager.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/cache/ICacheManager.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/cache/OutgoingBatchCache.java | Diff File | ||
|
SymmetricDS: 3.16 32a571f3 2025-03-04 13:36:33 Details Diff |
0006688: don't use ready queues if route on extract |
Affected Issues 0006688 |
|
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/PullService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/PushService.java | Diff File | ||
| mod - symmetric-server/src/main/java/org/jumpmind/symmetric/web/PullUriHandler.java | Diff File | ||
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2025-01-02 15:34 | elong | New Issue | |
| 2025-01-02 15:34 | elong | Status | new => assigned |
| 2025-01-02 15:34 | elong | Assigned To | => elong |
| 2025-01-02 15:34 | elong | Tag Attached: initial/partial load | |
| 2025-01-06 21:03 | elong | Status | assigned => resolved |
| 2025-01-06 21:03 | elong | Resolution | open => fixed |
| 2025-01-06 21:03 | elong | Fixed in Version | => 3.16.0 |
| 2025-01-06 22:00 | elong | Changeset attached | => SymmetricDS 3.16 7bab5e18 |
| 2025-01-10 19:00 | admin | Changeset attached | => SymmetricDS 3.16 680c309b |
| 2025-02-13 14:00 | Changeset attached | => SymmetricDS 3.16 a46b865f | |
| 2025-03-03 22:00 | admin | Changeset attached | => SymmetricDS 3.16 5233c204 |
| 2025-03-04 14:00 | admin | Changeset attached | => SymmetricDS 3.16 32a571f3 |
| 2025-03-17 17:15 | pbelov | Note Added: 0002695 | |
| 2025-03-17 17:15 | pbelov | Status | resolved => closed |
| 2025-03-17 20:15 | elong | Tag Attached: performance |