View Revisions: Issue #6616
| Summary | 0006616: Save column references as numeric values for faster look-up in AbstractDatabaseWriter.getRowData() | ||
|---|---|---|---|
| Revision | 2024-10-11 12:54 by pbelov | ||
| Description | Background: AbstractDatabaseWriter.getRowData() is used to prepare array of string values for import into target table. It does so by lookup up source columns by name and copying values from the source array to the target array. Currently getRowData() is searching for source columns (to copy data from) for every single data row. This adds up to about 1 second overhead per 500K rows. Originally discovered while working on bulk load, but affects all load batches. Proposed solution is to save column references as numeric values (source column 0000001 ==> target column 0000001) and re-use these numeric references to copy values from source to target skipping additional column name look-ups (for every row of data, after the first). |
||
| Revision | 2024-10-11 14:26 by elong | ||
| Description | Background: AbstractDatabaseWriter.getRowData() is used to prepare array of string values for import into target table. It does so by lookup up source columns by name and copying values from the source array to the target array. Currently getRowData() is searching for source columns (to copy data from) for every single data row. This adds up to about 1 second overhead per 500K rows. Originally discovered while working on bulk load, but affects all load batches. Proposed solution is to save column references as numeric values (source column #0000001 ==> target column #0000001) and re-use these numeric references to copy values from source to target skipping additional column name look-ups (for every row of data, after the first). |
||
| Revision | 2024-10-11 14:27 by elong | ||
| Description | Background: AbstractDatabaseWriter.getRowData() is used to prepare array of string values for import into target table. It does so by lookup up source columns by name and copying values from the source array to the target array. Currently getRowData() is searching for source columns (to copy data from) for every single data row. This adds up to about 1 second overhead per 500K rows. Originally discovered while working on bulk load, but affects all load batches. Proposed solution is to save column references as numeric values (source column 1 ==> target column 1) and re-use these numeric references to copy values from source to target skipping additional column name look-ups (for every row of data, after the first). |
||
| Revision | 2024-10-11 12:54 by pbelov | ||
| Additional Information | Given: S = number of columns in the source table, T = number of columns in the target table, N = number of rows in the data load batch, Current algorithm cost is: O( S * T * N) Proposed algorithm cost is: O( S * T ) + O( N ) |
||
| Revision | 2024-10-11 16:06 by pbelov | ||
| Additional Information | Given: S = number of columns in the source table, T = number of columns in the target table, N = number of rows in the data load batch, Current algorithm cost is: O( S * T * N) Proposed algorithm cost is: O( S * T ) + O( N ); For large N this cost growth is linear |
||