View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0003341 | SymmetricDS | Improvement | public | 2017-12-20 19:06 | 2018-02-14 20:08 |
| Reporter | mmichalek | Assigned To | mmichalek | ||
| Priority | normal | ||||
| Status | closed | Resolution | fixed | ||
| Product Version | 3.8.33 | ||||
| Target Version | 3.9.1 | Fixed in Version | 3.9.1 | ||
| Summary | 0003341: Improve PostgreSQL bulk loader to tolerate NULL bytes in incoming data | ||||
| Description | The PostgreSQL bulk loader will now filter out NULL bytes from string data before loading. This prevents errors like the following when there are null bytes in the data. See the attached insert statement for a simple statement that will reproduce the issue using the demo data set. 2017-12-20 13:54:58,122 ERROR [store-003] [DataLoaderService] [store-003-dataloader-6] Failed to load batch 000-30 StackTraceKey.init [SqlException:2182076017] org.jumpmind.db.sql.SqlException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 Where: COPY item, line 1 at org.jumpmind.db.sql.AbstractSqlTemplate.translate(AbstractSqlTemplate.java:300) at org.jumpmind.db.sql.AbstractSqlTemplate.translate(AbstractSqlTemplate.java:291) at org.jumpmind.symmetric.io.PostgresBulkDatabaseWriter.endCopy(PostgresBulkDatabaseWriter.java:184) at org.jumpmind.symmetric.io.PostgresBulkDatabaseWriter.end(PostgresBulkDatabaseWriter.java:200) at org.jumpmind.symmetric.io.data.writer.NestedDataWriter.end(NestedDataWriter.java:68) at org.jumpmind.symmetric.io.data.writer.NestedDataWriter.end(NestedDataWriter.java:68) at org.jumpmind.symmetric.io.data.writer.TransformWriter.end(TransformWriter.java:471) at org.jumpmind.symmetric.io.data.DataProcessor.forEachTableInBatch(DataProcessor.java:176) at org.jumpmind.symmetric.io.data.DataProcessor.process(DataProcessor.java:116) at org.jumpmind.symmetric.service.impl.DataLoaderService$LoadIntoDatabaseOnArrivalListener$1.call(DataLoaderService.java:988) at org.jumpmind.symmetric.service.impl.DataLoaderService$LoadIntoDatabaseOnArrivalListener$1.call(DataLoaderService.java:1) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 Where: COPY item, line 1 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2455) at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:1049) at org.postgresql.core.v3.QueryExecutorImpl.endCopy(QueryExecutorImpl.java:892) at org.postgresql.core.v3.CopyInImpl.endCopy(CopyInImpl.java:40) at org.jumpmind.symmetric.io.PostgresBulkDatabaseWriter.endCopy(PostgresBulkDatabaseWriter.java:178) ... 12 more 2017-12-20 13:54:58,128 ERROR [corp-000] [AcknowledgeService] [qtp1262353316-300] The outgoing batch 003-30 failed: ERROR: invalid byte sequence for encoding "UTF8": 0x00 Where: COPY item, line 1 | ||||
| Tags | No tags attached. | ||||
|
|
|
|
|
There is a patch for 3.8.33 at: https://www.jumpmind.com/downloads/symmetricds/patches/3.8/patch-0003341.jar |
|
SymmetricDS: 3.8 70b80988 2017-12-20 14:09:23 Details Diff |
0003341: Improve PostgreSQL bulk loader to tolerate NULL bytes in incoming data |
Affected Issues 0003341 |
|
| mod - symmetric-client/src/main/java/org/jumpmind/symmetric/io/PostgresBulkDatabaseWriter.java | Diff File | ||
|
SymmetricDS: 3.9 515e3eb0 2017-12-20 14:09:23 Details Diff |
0003341: Improve PostgreSQL bulk loader to tolerate NULL bytes in incoming data |
Affected Issues 0003341 |
|
| mod - symmetric-client/src/main/java/org/jumpmind/symmetric/io/PostgresBulkDatabaseWriter.java | Diff File | ||
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2017-12-20 19:06 | mmichalek | New Issue | |
| 2017-12-20 19:06 | mmichalek | Status | new => assigned |
| 2017-12-20 19:06 | mmichalek | Assigned To | => mmichalek |
| 2017-12-20 19:06 | mmichalek | File Added: insert-null-bytes.sql | |
| 2017-12-20 19:17 | mmichalek | Status | assigned => resolved |
| 2017-12-20 19:17 | mmichalek | Resolution | open => fixed |
| 2017-12-20 19:17 | mmichalek | Fixed in Version | => 3.9.1 |
| 2017-12-20 19:19 | mmichalek | Note Added: 0001089 | |
| 2017-12-20 20:00 | mmichalek | Changeset attached | => SymmetricDS 3.8 70b80988 |
| 2017-12-22 20:06 | mmichalek | Status | resolved => closed |
| 2018-02-14 20:08 | mmichalek | Changeset attached | => SymmetricDS 3.9 515e3eb0 |