View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0003356 | SymmetricDS | Bug | public | 2018-01-05 14:34 | 2018-03-28 22:16 |
| Reporter | woehrl01 | Assigned To | mmichalek | ||
| Priority | normal | ||||
| Status | closed | Resolution | fixed | ||
| Product Version | 3.9.2 | ||||
| Target Version | 3.9.6 | Fixed in Version | 3.9.6 | ||
| Summary | 0003356: Add initial.load.transport.max.bytes.to.sync parameter. transport.max.bytes.to.sync not respected in 3.9 during loads/reload | ||||
| Description | After the upgrade to 3.9, the setting transport.max.bytes.to.sync isn't respected anymore. It know sends all the data until the max_batch_to_send limit for the channel is reached. Output: [central-0] - PullUriHandler - 807875 data and 1000 batches sent during pull request from mobil:node2:2 [central-0] - PullUriHandler - 181286 data and 347 batches sent during pull request from mobil:node2:2 | ||||
| Tags | No tags attached. | ||||
|
|
I provided a PR which fixes this behaviour: https://github.com/JumpMind/symmetric-ds/pull/62 |
|
|
Hey Woehrl01 I have been reviewing your support ticket and have attempted to reproduce the error that you encountered but was unable to get the logging error messages that you did. To test I did the following steps: -Install and configured a new instance of SymmetricDS Pro 3.8.32 -Select a default configuration using the Demo setup -Change the value for parameter transport.max.bytes.to.sync to 1000 -Change the ITEM channel's value for MAX_BATCH_TO_SEND to 2000 -DB Fill the ITEM table on CORP with 2000 rows -Check log for output statistics to ensure that transport.max.bytes.to.sync was updated and is being referenced **INFO [corp-000] [DataExtractorService] [corp-000-dataextractor-5] Reached the total byte threshold after 1 of 5 batches were extracted for node '002' (extracted 1273 bytes, the max is 1000). The remaining batches will be extracted on a subsequent sync.** -Stop SymmetricDS Pro and upgrade to ver 3.9.2, restart -DB Fill the ITEM table on CORP with 2000 rows -Check log for output statistics to ensure that transport.max.bytes.to.sync was updated and is being referenced **INFO [corp-000] [DataExtractorService] [corp-000-dataextractor-13] Reached the total byte threshold after 1 of 3 batches were extracted for node '001' (extracted 452992 bytes, the max is 1000). The remaining batches will be extracted on a subsequent sync.** My log output appears to be the same after upgrade. Can you send me additional information on your setup? Do you have stream.to.file.enabled = true or any other parameters that are not set to their default values? |
|
|
Thank you for your feedback kpatierno! Here are additional infos to reproduce: Configuration: stream.to.file.enabled=true initial.load.use.extract.job.enabled=true initial.load.extract.and.send.when.staged=true I get that output if I do an initial load |
|
|
This issue is similar to what was seen here, https://sourceforge.net/p/symmetricds/discussion/739236/thread/22260ca7/, and appears to have been introduced in this issue, http://www.symmetricds.org/issues/view.php?id=2577 |
|
|
Hi Lukas, Can you provide a more detail around the scenario where you see this problem? For example, are you doing a load/initial load/table reload request or anything like that? We have tested normal push/pull functionality with the SymmetricDS "demo" setup as Kenny (kpatierno) described above. When we test, we see the byte threshold get hit and respected, per the logic in the current DataExtractorService:885: if (status.byteExtractCount >= maxBytesToSync && status.batchExtractCount < activeBatches.size()) { log.info( "Reached the total byte threshold after {} of {} batches were extracted for node '{}' (extracted {} bytes, the max is {}). " + "The remaining batches will be extracted on a subsequent sync.", new Object[] { status.batchExtractCount, activeBatches.size(), targetNode.getNodeId(), status.byteExtractCount, maxBytesToSync }); status.shouldExtractSkip = true; } |
|
|
Hi Mark, sure, sorry for missing that information. It happens during an initial load, in a push/pull scenario. (Client is pushing, Server waits for pull). I'm using the initial load extract job. If I do a simple trigger based sync the batches are respected. (probably not relevant: The server is sql server and the client is sqlite. And I'm starting symds with the /bin/sym.bat file) Which kind of additional informations would be helpful for you? |
|
|
I just thought about this issue a bit. I think the problem in the current implementation (without my PR) arises if the sending of the data is much slower than the extraction of the database. In that case all the extraction could be already done, before the sending of data limits the transmission,which than transmits all the extracted data. Even so it would be more than the limit. |
|
|
Lukas, thank you for that additional information. I am reviewing again with these factors in mind. |
|
|
Lukas, we have reproduced the issue now. We discussed it with the development team, and we are thinking of adding a second parameter for the reload/initial load case. For a lot of SymmetricDS users, initial loads are a much different case than normal data sync, and they might want different thresholds for those cases. Perhaps the new parameter would be intial.load.transport.max.bytes.to.sync, and would have a higher default value than the existing transport.max.bytes.to.sync. Let us know if you have any concerns with that approach. |
|
|
Hi Mark, sounds like a great idea, I currently have changed that value to a much higher value (10Mb) especially for the initial load case. I think a second value could give a better fine tuning option. As I created a pr do you want me to change it, or do you want to provide your own solution? I don't mind either :) |
|
|
Lukas, we were also thinking of increasing the default to at least 10 MB, so I am glad to hear you were thinking the same thing. If you are able to adjust your PR to use the new parameter, that would be great and I will review and accept it. If you don't get to get to it we will try and implement these changes in the next week or so. Thanks for your continued support of the project. |
|
|
Mark, I'll update my PR to use a new parameter with a default value of 10 MB. I think I can update it in the next day(s). |
|
|
Awesome, thank you Lukas. |
|
|
Mark, I updated the pull request. Let me know if it fit your needs. |
|
|
Thanks Lukas, that's perfect - it's merged and will be part of the build today or tomorrow. We really appreciate it. Please be aware we also bumped the transport.max.bytes.to.sync to 100MB and initial.load.transport.max.bytes.to.sync to 500 MB now, by default. See https://www.symmetricds.org/issues/view.php?id=3502 |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2018-01-05 14:34 | woehrl01 | New Issue | |
| 2018-01-08 08:48 | woehrl01 | Note Added: 0001097 | |
| 2018-01-10 16:04 | kpatierno | Note Added: 0001101 | |
| 2018-01-10 19:05 | woehrl01 | Note Added: 0001102 | |
| 2018-01-15 15:29 | rwfuller | Note Added: 0001105 | |
| 2018-03-07 19:03 | mmichalek | Note Added: 0001144 | |
| 2018-03-07 19:49 | woehrl01 | Note Added: 0001145 | |
| 2018-03-10 23:31 | woehrl01 | Note Added: 0001147 | |
| 2018-03-14 15:48 | mmichalek | Note Added: 0001150 | |
| 2018-03-14 18:12 | mmichalek | Summary | transport.max.bytes.to.sync not respected in 3.9 => transport.max.bytes.to.sync not respected in 3.9 during loads/reloads |
| 2018-03-15 16:32 | mmichalek | Note Added: 0001152 | |
| 2018-03-15 16:47 | woehrl01 | Note Added: 0001153 | |
| 2018-03-21 16:56 | mmichalek | Note Added: 0001155 | |
| 2018-03-21 20:20 | woehrl01 | Note Added: 0001156 | |
| 2018-03-21 22:01 | mmichalek | Note Added: 0001157 | |
| 2018-03-22 13:04 | woehrl01 | Note Added: 0001158 | |
| 2018-03-22 14:14 | mmichalek | Assigned To | => mmichalek |
| 2018-03-22 14:14 | mmichalek | Status | new => resolved |
| 2018-03-22 14:14 | mmichalek | Resolution | open => fixed |
| 2018-03-22 14:14 | mmichalek | Fixed in Version | => 3.9.6 |
| 2018-03-22 14:14 | mmichalek | Target Version | => 3.9.6 |
| 2018-03-22 14:14 | mmichalek | Summary | transport.max.bytes.to.sync not respected in 3.9 during loads/reloads => Add initial.load.transport.max.bytes.to.sync parameter. transport.max.bytes.to.sync not respected in 3.9 during loads/reload |
| 2018-03-22 14:25 | mmichalek | Note Added: 0001159 | |
| 2018-03-22 15:00 | woehrl01 | Changeset attached | => SymmetricDS 3.9 1b07622b |
| 2018-03-28 22:16 | mmichalek | Status | resolved => closed |