CTM-661: add db indexes for retrieveSnapshot() - #2126
Conversation
davidangb
changed the title
|
|
The analysis makes sense to me – indexes can make a huge difference; but I'm not really familiar with Postgres query plans. Practical question, how long will this changeset take to apply on the production instance? |
@aednichols I don't know for sure, but my best guess is seconds or tens of seconds. --------- new index snapshot_table_parent_id_idx on snapshot_table.parent_id
select count(1) from snapshot_table;
-- 75,195
select count(distinct(parent_id)) from snapshot_table;
-- 30,795
--------- new index snapshot_column_table_id_idx on snapshot_column.table_id
select count(1) from snapshot_column;
-- 1,866,360
select count(distinct(table_id)) from snapshot_column;
-- 75,195
--------- new index snapshot_map_table_source_id_idx on snapshot_map_table.source_id
select count(1) from snapshot_map_table;
-- 75,149
select count(distinct(source_id)) from snapshot_map_table;
-- 30,759
--------- new index snapshot_map_column_map_table_id_idx on snapshot_map_column.map_table_id
select count(1) from snapshot_map_column;
-- 1,864,637
select count(distinct(map_table_id)) from snapshot_map_column;
-- 75,149
At these scales, Postgres should be pretty efficient. 1.8M rows is not trivial but also is not particularly scary. |
There was a problem hiding this comment.
Than you, this looks great. The google query insights shows that the sequential scans the snapshot_column and snapshot_map_column tables were big factors in the load on the db during one of the recent spikes. The join between snapshot_table and snapshot_column and then also between snapshot_map_table and snapshot_map_column were also pretty big.
As far as applying these changes, we have done this sort of change before to help with the load on the DB and I don't think it has caused an issue before (see #2030 and #1861)
3 participants
Jira ticket: https://broadworkbench.atlassian.net/browse/CTM-661
Addresses
TDR's Postgres instance saw its CPU pegged at 100% for ~2 hours. This has happened multiple times in the past week.
Summary of changes
This PR adds indexes to four columns used by
GET /api/repository/v1/snapshots/{id}, to retrieve a single snapshot.Details
The most recent CPU spike was triggered by a surge in traffic from
azul-ucsc-0-hammerbox@platform-anvil-prod.iam.gserviceaccount.com.Claude quantified this traffic as:
This increased traffic to
GET /api/repository/v1/snapshots/{id}in turn caused the db to struggle because the SQL powering this API is inefficient. Two separate queries -SnapshotTableDao.sqlSelectTableandSnapshotMapTablesDao.retrieveMapTables- each perform full table scans on two columns each. The size of the tables in question is not large, but under this kind of load they struggled mightily.Running
explainagainst the two queries proves this out - look for the two""Node Type"": ""Seq Scan""callouts in each of these:SnapshotTableDao.sqlSelectTable
"QUERY PLAN" "[ { ""Plan"": { ""Node Type"": ""Sort"", ""Parallel Aware"": false, ""Async Capable"": false, ""Startup Cost"": 1542.01, ""Total Cost"": 1542.12, ""Plan Rows"": 43, ""Plan Width"": 124, ""Actual Startup Time"": 14.640, ""Actual Total Time"": 14.665, ""Actual Rows"": 171, ""Actual Loops"": 1, ""Output"": [""t.id"", ""t.name"", ""t.row_count"", ""t.primary_key"", ""c.id"", ""c.name"", ""c.type"", ""c.array_of"", ""c.required"", ""t.ctid"", ""c.ordinal""], ""Sort Key"": [""t.ctid"", ""c.ordinal""], ""Sort Method"": ""quicksort"", ""Sort Space Used"": 48, ""Sort Space Type"": ""Memory"", ""Shared Hit Blocks"": 873, ""Shared Read Blocks"": 0, ""Shared Dirtied Blocks"": 0, ""Shared Written Blocks"": 0, ""Local Hit Blocks"": 0, ""Local Read Blocks"": 0, ""Local Dirtied Blocks"": 0, ""Local Written Blocks"": 0, ""Temp Read Blocks"": 0, ""Temp Written Blocks"": 0, ""I/O Read Time"": 0.000, ""I/O Write Time"": 0.000, ""Temp I/O Read Time"": 0.000, ""Temp I/O Write Time"": 0.000, ""Plans"": [ { ""Node Type"": ""Hash Join"", ""Parent Relationship"": ""Outer"", ""Parallel Aware"": false, ""Async Capable"": false, ""Join Type"": ""Inner"", ""Startup Cost"": 215.10, ""Total Cost"": 1540.85, ""Plan Rows"": 43, ""Plan Width"": 124, ""Actual Startup Time"": 2.067, ""Actual Total Time"": 14.418, ""Actual Rows"": 171, ""Actual Loops"": 1, ""Output"": [""t.id"", ""t.name"", ""t.row_count"", ""t.primary_key"", ""c.id"", ""c.name"", ""c.type"", ""c.array_of"", ""c.required"", ""t.ctid"", ""c.ordinal""], ""Inner Unique"": true, ""Hash Cond"": ""(c.table_id = t.id)"", ""Shared Hit Blocks"": 865, ""Shared Read Blocks"": 0, ""Shared Dirtied Blocks"": 0, ""Shared Written Blocks"": 0, ""Local Hit Blocks"": 0, ""Local Read Blocks"": 0, ""Local Dirtied Blocks"": 0, ""Local Written Blocks"": 0, ""Temp Read Blocks"": 0, ""Temp Written Blocks"": 0, ""I/O Read Time"": 0.000, ""I/O Write Time"": 0.000, ""Temp I/O Read Time"": 0.000, ""Temp I/O Write Time"": 0.000, ""Plans"": [ { - ""Node Type"": ""Seq Scan"", ""Parent Relationship"": ""Outer"", ""Parallel Aware"": false, ""Async Capable"": false, ""Relation Name"": ""snapshot_column"", ""Schema"": ""public"", ""Alias"": ""c"", ""Startup Cost"": 0.00, ""Total Cost"": 1199.72, ""Plan Rows"": 47972, ""Plan Width"": 61, ""Actual Startup Time"": 0.009, ""Actual Total Time"": 9.593, ""Actual Rows"": 49377, ""Actual Loops"": 1, ""Output"": [""c.id"", ""c.name"", ""c.type"", ""c.array_of"", ""c.required"", ""c.ordinal"", ""c.table_id""], ""Shared Hit Blocks"": 720, ""Shared Read Blocks"": 0, ""Shared Dirtied Blocks"": 0, ""Shared Written Blocks"": 0, ""Local Hit Blocks"": 0, ""Local Read Blocks"": 0, ""Local Dirtied Blocks"": 0, ""Local Written Blocks"": 0, ""Temp Read Blocks"": 0, ""Temp Written Blocks"": 0, ""I/O Read Time"": 0.000, ""I/O Write Time"": 0.000, ""Temp I/O Read Time"": 0.000, ""Temp I/O Write Time"": 0.000 }, { ""Node Type"": ""Hash"", ""Parent Relationship"": ""Inner"", ""Parallel Aware"": false, ""Async Capable"": false, ""Startup Cost"": 215.04, ""Total Cost"": 215.04, ""Plan Rows"": 5, ""Plan Width"": 79, ""Actual Startup Time"": 0.753, ""Actual Total Time"": 0.755, ""Actual Rows"": 18, ""Actual Loops"": 1, ""Output"": [""t.id"", ""t.name"", ""t.row_count"", ""t.primary_key"", ""t.ctid""], ""Hash Buckets"": 1024, ""Original Hash Buckets"": 1024, ""Hash Batches"": 1, ""Original Hash Batches"": 1, ""Peak Memory Usage"": 10, ""Shared Hit Blocks"": 145, ""Shared Read Blocks"": 0, ""Shared Dirtied Blocks"": 0, ""Shared Written Blocks"": 0, ""Local Hit Blocks"": 0, ""Local Read Blocks"": 0, ""Local Dirtied Blocks"": 0, ""Local Written Blocks"": 0, ""Temp Read Blocks"": 0, ""Temp Written Blocks"": 0, ""I/O Read Time"": 0.000, ""I/O Write Time"": 0.000, ""Temp I/O Read Time"": 0.000, ""Temp I/O Write Time"": 0.000, ""Plans"": [ { - ""Node Type"": ""Seq Scan"", ""Parent Relationship"": ""Outer"", ""Parallel Aware"": false, ""Async Capable"": false, ""Relation Name"": ""snapshot_table"", ""Schema"": ""public"", ""Alias"": ""t"", ""Startup Cost"": 0.00, ""Total Cost"": 215.04, ""Plan Rows"": 5, ""Plan Width"": 79, ""Actual Startup Time"": 0.138, ""Actual Total Time"": 0.739, ""Actual Rows"": 18, ""Actual Loops"": 1, ""Output"": [""t.id"", ""t.name"", ""t.row_count"", ""t.primary_key"", ""t.ctid""], ""Filter"": ""(t.parent_id = 'e40bea8e-be23-441d-9a8b-a35abb213d2c'::uuid)"", ""Rows Removed by Filter"": 5610, ""Shared Hit Blocks"": 145, ""Shared Read Blocks"": 0, ""Shared Dirtied Blocks"": 0, ""Shared Written Blocks"": 0, ""Local Hit Blocks"": 0, ""Local Read Blocks"": 0, ""Local Dirtied Blocks"": 0, ""Local Written Blocks"": 0, ""Temp Read Blocks"": 0, ""Temp Written Blocks"": 0, ""I/O Read Time"": 0.000, ""I/O Write Time"": 0.000, ""Temp I/O Read Time"": 0.000, ""Temp I/O Write Time"": 0.000 } ] } ] } ] }, ""Query Identifier"": -5805976077017921812, ""Planning"": { ""Shared Hit Blocks"": 302, ""Shared Read Blocks"": 0, ""Shared Dirtied Blocks"": 0, ""Shared Written Blocks"": 0, ""Local Hit Blocks"": 0, ""Local Read Blocks"": 0, ""Local Dirtied Blocks"": 0, ""Local Written Blocks"": 0, ""Temp Read Blocks"": 0, ""Temp Written Blocks"": 0, ""I/O Read Time"": 0.000, ""I/O Write Time"": 0.000, ""Temp I/O Read Time"": 0.000, ""Temp I/O Write Time"": 0.000 }, ""Planning Time"": 1.357, ""Triggers"": [ ], ""Execution Time"": 14.919 } ]"SnapshotMapTablesDao.retrieveMapTables
"QUERY PLAN" "[ { ""Plan"": { ""Node Type"": ""Hash Join"", ""Parallel Aware"": false, ""Async Capable"": false, ""Join Type"": ""Inner"", ""Startup Cost"": 176.81, ""Total Cost"": 1457.32, ""Plan Rows"": 43, ""Plan Width"": 96, ""Actual Startup Time"": 2.166, ""Actual Total Time"": 14.619, ""Actual Rows"": 171, ""Actual Loops"": 1, ""Output"": [""smt.id"", ""smt.from_table_id"", ""smt.to_table_id"", ""smc.id"", ""smc.from_column_id"", ""smc.to_column_id""], ""Inner Unique"": true, ""Hash Cond"": ""(smc.map_table_id = smt.id)"", ""Shared Hit Blocks"": 778, ""Shared Read Blocks"": 0, ""Shared Dirtied Blocks"": 0, ""Shared Written Blocks"": 0, ""Local Hit Blocks"": 0, ""Local Read Blocks"": 0, ""Local Dirtied Blocks"": 0, ""Local Written Blocks"": 0, ""Temp Read Blocks"": 0, ""Temp Written Blocks"": 0, ""I/O Read Time"": 0.000, ""I/O Write Time"": 0.000, ""Temp I/O Read Time"": 0.000, ""Temp I/O Write Time"": 0.000, ""Plans"": [ { - ""Node Type"": ""Seq Scan"", ""Parent Relationship"": ""Outer"", ""Parallel Aware"": false, ""Async Capable"": false, ""Relation Name"": ""snapshot_map_column"", ""Schema"": ""public"", ""Alias"": ""smc"", ""Startup Cost"": 0.00, ""Total Cost"": 1153.70, ""Plan Rows"": 48270, ""Plan Width"": 64, ""Actual Startup Time"": 0.009, ""Actual Total Time"": 5.756, ""Actual Rows"": 49341, ""Actual Loops"": 1, ""Output"": [""smc.id"", ""smc.map_table_id"", ""smc.from_column_id"", ""smc.to_column_id""], ""Shared Hit Blocks"": 671, ""Shared Read Blocks"": 0, ""Shared Dirtied Blocks"": 0, ""Shared Written Blocks"": 0, ""Local Hit Blocks"": 0, ""Local Read Blocks"": 0, ""Local Dirtied Blocks"": 0, ""Local Written Blocks"": 0, ""Temp Read Blocks"": 0, ""Temp Written Blocks"": 0, ""I/O Read Time"": 0.000, ""I/O Write Time"": 0.000, ""Temp I/O Read Time"": 0.000, ""Temp I/O Write Time"": 0.000 }, { ""Node Type"": ""Hash"", ""Parent Relationship"": ""Inner"", ""Parallel Aware"": false, ""Async Capable"": false, ""Startup Cost"": 176.75, ""Total Cost"": 176.75, ""Plan Rows"": 5, ""Plan Width"": 48, ""Actual Startup Time"": 0.589, ""Actual Total Time"": 0.590, ""Actual Rows"": 18, ""Actual Loops"": 1, ""Output"": [""smt.id"", ""smt.from_table_id"", ""smt.to_table_id""], ""Hash Buckets"": 1024, ""Original Hash Buckets"": 1024, ""Hash Batches"": 1, ""Original Hash Batches"": 1, ""Peak Memory Usage"": 10, ""Shared Hit Blocks"": 107, ""Shared Read Blocks"": 0, ""Shared Dirtied Blocks"": 0, ""Shared Written Blocks"": 0, ""Local Hit Blocks"": 0, ""Local Read Blocks"": 0, ""Local Dirtied Blocks"": 0, ""Local Written Blocks"": 0, ""Temp Read Blocks"": 0, ""Temp Written Blocks"": 0, ""I/O Read Time"": 0.000, ""I/O Write Time"": 0.000, ""Temp I/O Read Time"": 0.000, ""Temp I/O Write Time"": 0.000, ""Plans"": [ { - ""Node Type"": ""Seq Scan"", ""Parent Relationship"": ""Outer"", ""Parallel Aware"": false, ""Async Capable"": false, ""Relation Name"": ""snapshot_map_table"", ""Schema"": ""public"", ""Alias"": ""smt"", ""Startup Cost"": 0.00, ""Total Cost"": 176.75, ""Plan Rows"": 5, ""Plan Width"": 48, ""Actual Startup Time"": 0.057, ""Actual Total Time"": 0.577, ""Actual Rows"": 18, ""Actual Loops"": 1, ""Output"": [""smt.id"", ""smt.from_table_id"", ""smt.to_table_id""], ""Filter"": ""(smt.source_id = 'ec740600-3e27-4613-bbe2-1c647bbdfe19'::uuid)"", ""Rows Removed by Filter"": 5604, ""Shared Hit Blocks"": 107, ""Shared Read Blocks"": 0, ""Shared Dirtied Blocks"": 0, ""Shared Written Blocks"": 0, ""Local Hit Blocks"": 0, ""Local Read Blocks"": 0, ""Local Dirtied Blocks"": 0, ""Local Written Blocks"": 0, ""Temp Read Blocks"": 0, ""Temp Written Blocks"": 0, ""I/O Read Time"": 0.000, ""I/O Write Time"": 0.000, ""Temp I/O Read Time"": 0.000, ""Temp I/O Write Time"": 0.000 } ] } ] }, ""Query Identifier"": 971855624158779861, ""Planning"": { ""Shared Hit Blocks"": 273, ""Shared Read Blocks"": 0, ""Shared Dirtied Blocks"": 0, ""Shared Written Blocks"": 0, ""Local Hit Blocks"": 0, ""Local Read Blocks"": 0, ""Local Dirtied Blocks"": 0, ""Local Written Blocks"": 0, ""Temp Read Blocks"": 0, ""Temp Written Blocks"": 0, ""I/O Read Time"": 0.000, ""I/O Write Time"": 0.000, ""Temp I/O Read Time"": 0.000, ""Temp I/O Write Time"": 0.000 }, ""Planning Time"": 1.889, ""Triggers"": [ ], ""Execution Time"": 14.765 } ]"We should probably figure out why
azul-ucsc-0-hammerbox@platform-anvil-prod.iam.gserviceaccount.comneeds to call retrieveSnapshot this many times, including repeated queries to the same snapshot, but we should also fix our db.