View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0006881SymmetricDSBugpublic2025-05-09 16:462025-05-09 16:46
Reporterelong Assigned Toelong  
Prioritynormal 
Status assignedResolutionopen 
Product Version3.16.0 
Target Version3.16.3 
Summary0006881: Deadlock from updating common batch stats in clustered environment
DescriptionDeadlocks occurring when updating the batch statistics on common batches, like those on heartbeat and monitor channels. The conditions needed are a clustered node (cluster.lock.enabled=true) and separate staging (cluster.staging.enabled=false). Each node in the cluster is extracting a batch and updating the statistics for all batches with the same batch ID, which is competing with the other nodes. Instead, each node that is using the batch from staging should look up the first common batch that was extracted to copy its statistics.

2025-05-07 03:41:29,317 ERROR [corp-00000] [DataExtractorService] [qtp1576526619-1330] Failed to extract batch 02533-1107530134 StackTraceKey.init [SqlException:766039076] org.jumpmind.db.sql.SqlException: ERROR: deadlock detected
  Detail: Process 3729 waits for ShareLock on transaction 2957219400; blocked by process 11161.
Process 11161 waits for ShareLock on transaction 2957219719; blocked by process 5455.
Process 5455 waits for ExclusiveLock on tuple (11197,10) of relation 21618 of database 16402; blocked by process 3729.
  Hint: See server log for query details.
  Where: while updating tuple (11197,10) in relation "sym_outgoing_batch"
    at org.jumpmind.db.sql.AbstractSqlTemplate.translate(AbstractSqlTemplate.java:308)
    at org.jumpmind.db.sql.AbstractSqlTemplate.translate(AbstractSqlTemplate.java:297)
    at org.jumpmind.db.sql.JdbcSqlTemplate.execute(JdbcSqlTemplate.java:492)
    at org.jumpmind.db.sql.JdbcSqlTemplate.update(JdbcSqlTemplate.java:348)
    at org.jumpmind.symmetric.service.impl.OutgoingBatchService.updateCommonBatchExtractStatistics(OutgoingBatchService.java:183)
    at org.jumpmind.symmetric.service.impl.DataExtractorService.extractOutgoingBatch(DataExtractorService.java:1010)
    at org.jumpmind.symmetric.service.impl.DataExtractorService.extractBatch(DataExtractorService.java:855)
    at org.jumpmind.symmetric.service.impl.DataExtractorService.lambda$extract$1(DataExtractorService.java:630)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
Caused by: org.postgresql.util.PSQLException: ERROR: deadlock detected
  Detail: Process 3729 waits for ShareLock on transaction 2957219400; blocked by process 11161.
Process 11161 waits for ShareLock on transaction 2957219719; blocked by process 5455.
Process 5455 waits for ExclusiveLock on tuple (11197,10) of relation 21618 of database 16402; blocked by process 3729.
  Hint: See server log for query details.
  Where: while updating tuple (11197,10) in relation "sym_outgoing_batch"
    at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675)
    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365)
    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355)
    at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:490)
    at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:408)
    at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:166)
    at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:155)
    at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
    at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
    at org.jumpmind.db.sql.JdbcSqlTemplate$5.execute(JdbcSqlTemplate.java:377)
    at org.jumpmind.db.sql.JdbcSqlTemplate$5.execute(JdbcSqlTemplate.java:348)
    at org.jumpmind.db.sql.JdbcSqlTemplate.execute(JdbcSqlTemplate.java:490)
    ... 9 more
Tagscluster

Relationships

related to 0006880 resolvedelong Deadlock from updating common batch stats in clustered environment 

Activities

There are no notes attached to this issue.

Issue History

Date Modified Username Field Change
2025-05-09 16:46 elong New Issue
2025-05-09 16:46 elong Status new => assigned
2025-05-09 16:46 elong Assigned To => elong
2025-05-09 16:46 elong Tag Attached: cluster
2025-05-09 16:46 elong Issue generated from: 0006880
2025-05-09 16:46 elong Relationship added related to 0006880