View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0006989 | SymmetricDS Pro | Improvement | public | 2025-07-07 20:52 | 2025-07-28 17:30 |
| Reporter | pbelov | Assigned To | pbelov | ||
| Priority | normal | ||||
| Status | closed | Resolution | fixed | ||
| Product Version | 3.15.0 | ||||
| Target Version | 3.15.19 | Fixed in Version | 3.15.19 | ||
| Summary | 0006989: Limit number of changes requested by the Log Miner from PostgreSQL | ||||
| Description | Limit number of changes requested by the Log Miner from PostgreSQL to prevent out-of-memory conditions and long-running cursors consuming lots of database resources. Update: with this change Log Miner for PostgreSQL is now more resilient when WAL backlog is putting pressure on resources available to both database engine and SymmetricDS. Administrators can now log.miner.max.request.rows parameter to dial down number of changes PostgreSQL will send to SymmetricDS during one run of the Log miner job. Also, SymmetricDS will mark WAL entries as "consumed" much sooner, allowing PostgreSQL to release resources. Administrators should consider tuning the log.miner.flush.commit.size parameter, which controls how often new data is committed to SymmetricDS runtime table, which also initiates marking WAL entries as "consumed". | ||||
| Steps To Reproduce | Currently PostgresTransactionMiner calls the pg_logical_slot_peek_changes without a limit to number of changes returned. Postgres allows to specify an upper limit on the number of changes to avoid depleting resources: https://www.postgresql.org/docs/current/functions-admin.html | ||||
| Additional Information | Ticket [UEM-49611-335] | ||||
| Tags | dialect: postgresql, log miner, performance | ||||
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2025-07-07 20:52 | pbelov | New Issue | |
| 2025-07-07 20:52 | pbelov | Status | new => assigned |
| 2025-07-07 20:52 | pbelov | Assigned To | => pbelov |
| 2025-07-08 01:42 | pbelov | Status | assigned => confirmed |
| 2025-07-09 13:18 | pbelov | Summary | Limit number of changes peeked by the PostgreSQL Log Miner => Limit number of changes requested by the Log Miner from PostgreSQL |
| 2025-07-09 13:18 | pbelov | Description Updated | View Revisions |
| 2025-07-09 16:52 | pbelov | Status | confirmed => resolved |
| 2025-07-09 16:52 | pbelov | Fixed in Version | => 3.15.19 |
| 2025-07-09 16:52 | pbelov | Target Version | => 3.15.19 |
| 2025-07-10 14:13 | pbelov | Description Updated | View Revisions |
| 2025-07-10 14:56 | pbelov | Status | resolved => closed |
| 2025-07-10 14:56 | pbelov | Note Added: 0003181 | |
| 2025-07-28 17:30 | pbelov | Tag Attached: dialect: postgresql | |
| 2025-07-28 17:30 | pbelov | Tag Attached: log miner | |
| 2025-07-28 17:30 | pbelov | Tag Attached: performance |