View Issue Details

IDProjectCategoryView StatusLast Update
0006880SymmetricDSBugpublic2025-06-08 00:43
Reporterelong Assigned Toelong  
Prioritynormal 
Status closedResolutionfixed 
Product Version3.15.0 
Target Version3.15.17Fixed in Version3.15.17 
Summary0006880: Deadlock from updating common batch stats in clustered environment
DescriptionDeadlocks occurring when updating the batch statistics on common batches, like those on heartbeat and monitor channels. The conditions needed are a clustered node (cluster.lock.enabled=true) and separate staging (cluster.staging.enabled=false). Each node in the cluster is extracting a batch and updating the statistics for all batches with the same batch ID, which is competing with the other nodes. Instead, each node that is using the batch from staging should look up the first common batch that was extracted to copy its statistics.

2025-05-07 03:41:29,317 ERROR [corp-00000] [DataExtractorService] [qtp1576526619-1330] Failed to extract batch 02533-1107530134 StackTraceKey.init [SqlException:766039076] org.jumpmind.db.sql.SqlException: ERROR: deadlock detected
  Detail: Process 3729 waits for ShareLock on transaction 2957219400; blocked by process 11161.
Process 11161 waits for ShareLock on transaction 2957219719; blocked by process 5455.
Process 5455 waits for ExclusiveLock on tuple (11197,10) of relation 21618 of database 16402; blocked by process 3729.
  Hint: See server log for query details.
  Where: while updating tuple (11197,10) in relation "sym_outgoing_batch"
    at org.jumpmind.db.sql.AbstractSqlTemplate.translate(AbstractSqlTemplate.java:308)
    at org.jumpmind.db.sql.AbstractSqlTemplate.translate(AbstractSqlTemplate.java:297)
    at org.jumpmind.db.sql.JdbcSqlTemplate.execute(JdbcSqlTemplate.java:492)
    at org.jumpmind.db.sql.JdbcSqlTemplate.update(JdbcSqlTemplate.java:348)
    at org.jumpmind.symmetric.service.impl.OutgoingBatchService.updateCommonBatchExtractStatistics(OutgoingBatchService.java:183)
    at org.jumpmind.symmetric.service.impl.DataExtractorService.extractOutgoingBatch(DataExtractorService.java:1010)
    at org.jumpmind.symmetric.service.impl.DataExtractorService.extractBatch(DataExtractorService.java:855)
    at org.jumpmind.symmetric.service.impl.DataExtractorService.lambda$extract$1(DataExtractorService.java:630)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
Caused by: org.postgresql.util.PSQLException: ERROR: deadlock detected
  Detail: Process 3729 waits for ShareLock on transaction 2957219400; blocked by process 11161.
Process 11161 waits for ShareLock on transaction 2957219719; blocked by process 5455.
Process 5455 waits for ExclusiveLock on tuple (11197,10) of relation 21618 of database 16402; blocked by process 3729.
  Hint: See server log for query details.
  Where: while updating tuple (11197,10) in relation "sym_outgoing_batch"
    at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675)
    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365)
    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355)
    at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:490)
    at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:408)
    at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:166)
    at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:155)
    at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
    at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
    at org.jumpmind.db.sql.JdbcSqlTemplate$5.execute(JdbcSqlTemplate.java:377)
    at org.jumpmind.db.sql.JdbcSqlTemplate$5.execute(JdbcSqlTemplate.java:348)
    at org.jumpmind.db.sql.JdbcSqlTemplate.execute(JdbcSqlTemplate.java:490)
    ... 9 more
Tagscluster

Relationships

related to 0006881 closedelong Deadlock from updating common batch stats in clustered environment 

Activities

pbelov

2025-06-08 00:43

manager   ~0003020

Included in the 3.15.17 release

Related Changesets

SymmetricDS: 3.15 7090e8fa

2025-05-12 17:37:42

elong


Committer: GitHub Details Diff
0006880: Deadlock from updating common batch stats in clustered (0000274)

0006880: Deadlock from updating common batch stats in clustered environment
Affected Issues
0006880
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/IOutgoingBatchService.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataExtractorService.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/OutgoingBatchService.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/OutgoingBatchServiceSqlMap.java Diff File

SymmetricDS: 3.16 2643d461

2025-05-12 17:37:42

elong


Committer: pbelov Details Diff
0006880: Deadlock from updating common batch stats in clustered (0000274)

0006880: Deadlock from updating common batch stats in clustered environment
Affected Issues
0006880
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/IOutgoingBatchService.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/DataExtractorService.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/OutgoingBatchService.java Diff File
mod - symmetric-core/src/main/java/org/jumpmind/symmetric/service/impl/OutgoingBatchServiceSqlMap.java Diff File

Issue History

Date Modified Username Field Change
2025-05-09 16:44 elong New Issue
2025-05-09 16:44 elong Status new => assigned
2025-05-09 16:44 elong Assigned To => elong
2025-05-09 16:44 elong Tag Attached: cluster
2025-05-09 16:46 elong Status assigned => resolved
2025-05-09 16:46 elong Resolution open => fixed
2025-05-09 16:46 elong Fixed in Version => 3.15.17
2025-05-09 16:46 elong Issue cloned: 0006881
2025-05-09 16:46 elong Relationship added related to 0006881
2025-05-12 18:00 elong Changeset attached => SymmetricDS 3.15 7090e8fa
2025-05-12 18:00 pbelov Changeset attached => SymmetricDS 3.16 2643d461
2025-06-08 00:43 pbelov Note Added: 0003020
2025-06-08 00:43 pbelov Status resolved => closed