View Issue Details

IDProjectCategoryView StatusLast Update
0003356SymmetricDSBugpublic2018-03-28 22:16
Reporterwoehrl01 Assigned Tommichalek  
Status closedResolutionfixed 
Product Version3.9.2 
Target Version3.9.6Fixed in Version3.9.6 
Summary0003356: Add parameter. not respected in 3.9 during loads/reload
DescriptionAfter the upgrade to 3.9, the setting isn't respected anymore. It know sends all the data until the max_batch_to_send limit for the channel is reached.

[central-0] - PullUriHandler - 807875 data and 1000 batches sent during pull request from mobil:node2:2
[central-0] - PullUriHandler - 181286 data and 347 batches sent during pull request from mobil:node2:2
TagsNo tags attached.



2018-01-08 08:48

reporter   ~0001097

I provided a PR which fixes this behaviour:


2018-01-10 16:04

viewer   ~0001101

Hey Woehrl01

I have been reviewing your support ticket and have attempted to reproduce the error that you encountered but was unable to get the logging error messages that you did.

To test I did the following steps:
-Install and configured a new instance of SymmetricDS Pro 3.8.32
-Select a default configuration using the Demo setup
-Change the value for parameter to 1000
-Change the ITEM channel's value for MAX_BATCH_TO_SEND to 2000
-DB Fill the ITEM table on CORP with 2000 rows
-Check log for output statistics to ensure that was updated and is being referenced
**INFO [corp-000] [DataExtractorService] [corp-000-dataextractor-5] Reached the total byte threshold after 1 of 5 batches were extracted for node '002' (extracted 1273 bytes, the max is 1000). The remaining batches will be extracted on a subsequent sync.**
-Stop SymmetricDS Pro and upgrade to ver 3.9.2, restart
-DB Fill the ITEM table on CORP with 2000 rows
-Check log for output statistics to ensure that was updated and is being referenced
**INFO [corp-000] [DataExtractorService] [corp-000-dataextractor-13] Reached the total byte threshold after 1 of 3 batches were extracted for node '001' (extracted 452992 bytes, the max is 1000). The remaining batches will be extracted on a subsequent sync.**

My log output appears to be the same after upgrade. Can you send me additional information on your setup? Do you have = true or any other parameters that are not set to their default values?


2018-01-10 19:05

reporter   ~0001102

Thank you for your feedback kpatierno!
Here are additional infos to reproduce:


I get that output if I do an initial load


2018-01-15 15:29

reporter   ~0001105

This issue is similar to what was seen here,, and appears to have been introduced in this issue,


2018-03-07 19:03

administrator   ~0001144

Hi Lukas,

Can you provide a more detail around the scenario where you see this problem? For example, are you doing a load/initial load/table reload request or anything like that? We have tested normal push/pull functionality with the SymmetricDS "demo" setup as Kenny (kpatierno) described above. When we test, we see the byte threshold get hit and respected, per the logic in the current DataExtractorService:885:

                    if (status.byteExtractCount >= maxBytesToSync && status.batchExtractCount < activeBatches.size()) {
                                "Reached the total byte threshold after {} of {} batches were extracted for node '{}' (extracted {} bytes, the max is {}). "
                                        + "The remaining batches will be extracted on a subsequent sync.",
                                new Object[] { status.batchExtractCount, activeBatches.size(), targetNode.getNodeId(), status.byteExtractCount, maxBytesToSync });
                        status.shouldExtractSkip = true;


2018-03-07 19:49

reporter   ~0001145

Hi Mark,

sure, sorry for missing that information. It happens during an initial load, in a push/pull scenario. (Client is pushing, Server waits for pull).
I'm using the initial load extract job. If I do a simple trigger based sync the batches are respected.

(probably not relevant: The server is sql server and the client is sqlite. And I'm starting symds with the /bin/sym.bat file)

Which kind of additional informations would be helpful for you?


2018-03-10 23:31

reporter   ~0001147

I just thought about this issue a bit. I think the problem in the current implementation (without my PR) arises if the sending of the data is much slower than the extraction of the database.

In that case all the extraction could be already done, before the sending of data limits the transmission,which than transmits all the extracted data. Even so it would be more than the limit.


2018-03-14 15:48

administrator   ~0001150

Lukas, thank you for that additional information. I am reviewing again with these factors in mind.


2018-03-15 16:32

administrator   ~0001152

Lukas, we have reproduced the issue now. We discussed it with the development team, and we are thinking of adding a second parameter for the reload/initial load case. For a lot of SymmetricDS users, initial loads are a much different case than normal data sync, and they might want different thresholds for those cases. Perhaps the new parameter would be, and would have a higher default value than the existing Let us know if you have any concerns with that approach.


2018-03-15 16:47

reporter   ~0001153

Hi Mark, sounds like a great idea, I currently have changed that value to a much higher value (10Mb) especially for the initial load case. I think a second value could give a better fine tuning option.

As I created a pr do you want me to change it, or do you want to provide your own solution? I don't mind either :)


2018-03-21 16:56

administrator   ~0001155

Lukas, we were also thinking of increasing the default to at least 10 MB, so I am glad to hear you were thinking the same thing. If you are able to adjust your PR to use the new parameter, that would be great and I will review and accept it. If you don't get to get to it we will try and implement these changes in the next week or so. Thanks for your continued support of the project.


2018-03-21 20:20

reporter   ~0001156

Mark, I'll update my PR to use a new parameter with a default value of 10 MB. I think I can update it in the next day(s).


2018-03-21 22:01

administrator   ~0001157

Awesome, thank you Lukas.


2018-03-22 13:04

reporter   ~0001158

Mark, I updated the pull request. Let me know if it fit your needs.


2018-03-22 14:25

administrator   ~0001159

Thanks Lukas, that's perfect - it's merged and will be part of the build today or tomorrow. We really appreciate it. Please be aware we also bumped the to 100MB and to 500 MB now, by default. See

Related Changesets

SymmetricDS: 3.9 1b07622b

2018-01-08 03:47:33


Committer: GitHub Details Diff
0003356: not respected Affected Issues
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/ Diff File

Issue History

Date Modified Username Field Change
2018-01-05 14:34 woehrl01 New Issue
2018-01-08 08:48 woehrl01 Note Added: 0001097
2018-01-10 16:04 kpatierno Note Added: 0001101
2018-01-10 19:05 woehrl01 Note Added: 0001102
2018-01-15 15:29 rwfuller Note Added: 0001105
2018-03-07 19:03 mmichalek Note Added: 0001144
2018-03-07 19:49 woehrl01 Note Added: 0001145
2018-03-10 23:31 woehrl01 Note Added: 0001147
2018-03-14 15:48 mmichalek Note Added: 0001150
2018-03-14 18:12 mmichalek Summary not respected in 3.9 => not respected in 3.9 during loads/reloads
2018-03-15 16:32 mmichalek Note Added: 0001152
2018-03-15 16:47 woehrl01 Note Added: 0001153
2018-03-21 16:56 mmichalek Note Added: 0001155
2018-03-21 20:20 woehrl01 Note Added: 0001156
2018-03-21 22:01 mmichalek Note Added: 0001157
2018-03-22 13:04 woehrl01 Note Added: 0001158
2018-03-22 14:14 mmichalek Assigned To => mmichalek
2018-03-22 14:14 mmichalek Status new => resolved
2018-03-22 14:14 mmichalek Resolution open => fixed
2018-03-22 14:14 mmichalek Fixed in Version => 3.9.6
2018-03-22 14:14 mmichalek Target Version => 3.9.6
2018-03-22 14:14 mmichalek Summary not respected in 3.9 during loads/reloads => Add parameter. not respected in 3.9 during loads/reload
2018-03-22 14:25 mmichalek Note Added: 0001159
2018-03-22 15:00 woehrl01 Changeset attached => SymmetricDS 3.9 1b07622b
2018-03-28 22:16 mmichalek Status resolved => closed