View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0006688 | SymmetricDS Pro | New Feature | public | 2025-01-02 15:34 | 2025-01-06 22:00 |
| Reporter | elong | Assigned To | elong | ||
| Priority | normal | ||||
| Status | resolved | Resolution | fixed | ||
| Product Version | 3.16.0 | ||||
| Target Version | 3.16.0 | Fixed in Version | 3.16.0 | ||
| Summary | 0006688: Dynamic reload queue to scale initial load | ||||
| Description | To make it easier for the initial load to scale across multiple threads, introduce a dynamic reload queue that assigns tables to a pool of threads. The number of threads is controlled by a parameter, one for extract threads and one for push/pull threads. All batches for a table are assigned to the same thread so there is no contention for the table. The initial load extract job makes assignments of batches to be extracted and loaded up to the max batches to sync, which allows for adjustment of the parameters during the load. When the reload channel is given the queue name of reload, this feature is enabled, which is the default configuration of the channel. Dynamic queues are internally named reload!1, reload!2, reload!3, etc based on the thread number assignment. The initial.load.queue.use.all.threads parameter controls if all threads can be used by an initial load with a setting of "auto" that uses all threads if table constraints have been deferred. initial.load.extract.thread.per.server.count=20 initial.load.queue.sync.thread.count=20 initial.load.queue.use.all.threads=auto | ||||
| Tags | initial/partial load | ||||
|
SymmetricDS: 3.16 7bab5e18 2025-01-06 21:02:41 Committer: GitHub Details Diff |
0006688: Dynamic reload queue to scale initial load (0000217) 0006688: Dynamic reload queue to scale initial load |
Affected Issues 0006688 |
|
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/AbstractSymmetricEngine.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/cache/CacheManager.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/cache/ICacheManager.java | Diff File | ||
| add - symmetric-core/src/main/java/org/jumpmind/symmetric/cache/OutgoingBatchCache.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/common/Constants.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/common/ParameterConstants.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/db/DatabaseUpgradeListener.java | Diff File | ||
| add - symmetric-core/src/main/java/org/jumpmind/symmetric/ext/IReloadQueueThreadAssigner.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/extract/MultiBatchStagingWriter.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/extract/SelectFromSymDataSource.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/io/stage/SimpleStagingDataWriter.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/model/ExtractRequest.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/model/NodeSecurity.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/model/OutgoingBatch.java | Diff File | ||
| add - symmetric-core/src/main/java/org/jumpmind/symmetric/model/ReadyChannels.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/IIncomingBatchService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/INodeService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/IOutgoingBatchService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/ITriggerRouterService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/ConfigurationService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataExtractorService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataExtractorServiceSqlMap.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataLoaderService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/FileSyncExtractorService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/IncomingBatchService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/InitialLoadService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/NodeCommunicationService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/NodeService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/NodeServiceSqlMap.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/OutgoingBatchService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/OutgoingBatchServiceSqlMap.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/PullService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/PushService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/RouterService.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/TriggerRouterService.java | Diff File | ||
| add - symmetric-core/src/main/java/org/jumpmind/symmetric/util/QueueThread.java | Diff File | ||
| mod - symmetric-core/src/main/java/org/jumpmind/symmetric/web/WebConstants.java | Diff File | ||
| mod - symmetric-core/src/main/resources/symmetric-default.properties | Diff File | ||
| mod - symmetric-core/src/main/resources/symmetric-schema.xml | Diff File | ||
| mod - symmetric-core/src/test/java/org/jumpmind/symmetric/service/impl/DataServiceTest.java | Diff File | ||
| mod - symmetric-core/src/test/java/org/jumpmind/symmetric/service/impl/MockNodeService.java | Diff File | ||
| mod - symmetric-db/src/main/java/org/jumpmind/db/sql/Row.java | Diff File | ||
| mod - symmetric-io/src/main/java/org/jumpmind/symmetric/io/data/writer/DefaultDatabaseWriter.java | Diff File | ||
| mod - symmetric-server/src/main/java/org/jumpmind/symmetric/web/PullUriHandler.java | Diff File | ||
| mod - symmetric-server/src/main/java/org/jumpmind/symmetric/web/PushUriHandler.java | Diff File | ||
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2025-01-02 15:34 | elong | New Issue | |
| 2025-01-02 15:34 | elong | Status | new => assigned |
| 2025-01-02 15:34 | elong | Assigned To | => elong |
| 2025-01-02 15:34 | elong | Tag Attached: initial/partial load | |
| 2025-01-06 21:03 | elong | Status | assigned => resolved |
| 2025-01-06 21:03 | elong | Resolution | open => fixed |
| 2025-01-06 21:03 | elong | Fixed in Version | => 3.16.0 |
| 2025-01-06 22:00 | elong | Changeset attached | => SymmetricDS 3.16 7bab5e18 |