add infrastructure progress report 2504/5

progress-reports/infrastructure/2504-infrastructure-progress-report.md

@@ -0,0 +1,32 @@
+# Tracker Monthly Report
+
+This is a progress update for 3/2025 on infrastructure provided by sudoblock including development of the tracker, indexer, and operation of RPC endpoints being used to support the ICON ecosystem. 
+
+### Summary
+
+For the frontend this month was a light month after clearing out a lot of the backlogs for the frontend. Most work has been on the backend was we're in the middle of a final migration after realizing some bad designs the current DB is running on. This month was mostly working through some of the nuances of the migration of these production workloads. 
+
+### [icon-transformer](https://github.com/sudoblockio/icon-transformer)
+
+#### Open Issues
+
+| Number | Name | Status | Created | Notes |
+| --- | --- | --- | --- | --- | 
+| [76](https://github.com/sudoblockio/icon-transformer/pull/76) | chore(main): release 0.3.5 | open | 2023-05-29T19:14:58Z | | 
+
+### Infrastructure 
+
+- Ran numerous benchmarks to figure out optimal server configuration parameters 
+  - Able to double performance of the nodes for DB / blockchain applications 
+  - Figured out real bottlenecks and working towards new deployment strategy 
+- Added new active alarms as just scripts 
+  - Before was just probes to see if endpoint was active. Now we send RPC payloads and do other custom logic on http probes which make up our status pages and alarms 
+
+## Downtime Incidents
+
+| Date | Total Downtime | Cause | Resolution |
+| --- | --- | --- | --- |
+
+> None to be reported 
+
+> Report generated with [tackle](https://github.com/robcxyz/tackle-box)

progress-reports/infrastructure/2505-infrastructure-progress-report.md

@@ -0,0 +1,51 @@
+# Tracker Monthly Report
+
+This is a progress update for 4/2025 on infrastructure provided by sudoblock including development of the tracker, indexer, and operation of RPC endpoints being used to support the ICON ecosystem.
+
+### Summary
+
+On the frontend we started scoping out wallet connect and what we need to do on the frontend and in Hanna to support the protocol. On the backend, we're still doing migrations from last month + further optimizations needed to run many applications on the same system without so called noisy neighbor issues where adjacent workloads could interfere with others. Finally after working through this we can say we are working with all the available knobs to tune our systems and get the most performance - something that has been a long learning experience. 
+
+### [icon-tracker-frontend](https://github.com/sudoblockio/icon-tracker-frontend)
+
+#### Closed Issues
+
+| Number | Name | Status | Created | Notes |
+| --- | --- | --- | --- | --- |
+| [561](https://github.com/sudoblockio/icon-tracker-frontend/pull/561) | fix: added apr column in voting page #560 | closed | 2025-04-19T11:35:50Z | | 
+| [560](https://github.com/sudoblockio/icon-tracker-frontend/issues/560) | APR in voting page | closed | 2025-04-19T11:35:01Z | | 
+| [556](https://github.com/sudoblockio/icon-tracker-frontend/pull/556) | chore: added logging for Sentry ErrorBoundary | closed | 2025-02-18T15:08:06Z | | 
+| [544](https://github.com/sudoblockio/icon-tracker-frontend/issues/544) | When staking in the tracker you need to leave a small amount for the Tx fee | closed | 2025-02-06T18:30:28Z | | 
+| [541](https://github.com/sudoblockio/icon-tracker-frontend/issues/541) | Show APR/APY values for validators | closed | 2025-02-05T13:57:39Z | | 
+| [525](https://github.com/sudoblockio/icon-tracker-frontend/issues/525) | Validate that the new global bond rate value is reflected in the governance page | closed | 2025-01-20T13:18:08Z | | 
+| [513](https://github.com/sudoblockio/icon-tracker-frontend/issues/513) | Display projected voter APR on tracker | closed | 2024-12-09T14:35:28Z | | 
+
+#### Open Issues
+
+| Number | Name | Status | Created | Notes |
+| --- | --- | --- | --- | --- | 
+| [563](https://github.com/sudoblockio/icon-tracker-frontend/pull/563) | feat: added total available voted amt in voting page #552 | open | 2025-04-21T12:19:47Z | | 
+| [562](https://github.com/sudoblockio/icon-tracker-frontend/pull/562) | chore(main): release 0.6.8 | open | 2025-04-21T11:52:11Z | | 
+
+### [icon-governance](https://github.com/sudoblockio/icon-governance)
+
+#### Open Issues
+
+| Number | Name | Status | Created | Notes |
+| --- | --- | --- | --- | --- | 
+| [60](https://github.com/sudoblockio/icon-governance/issues/60) | Update bond_percent formula | open | 2025-02-01T19:34:34Z | |
+
+### Infrastructure
+
+- Still working on migration figuring out new way to operate nodes to get another ~30% performance 
+  - Each VM running all the applications is currently being migrated again 
+- After this will need one last migration potentially at the end of the year to upgrade some components but that should be it 
+- As a result of these migrations we believe we have de-bottlenecked everything we can allowing us to run roughly 2 times more workloads and push at least 5 times as many requests per second as before the optimizations 
+
+## Downtime Incidents
+
+| Date | Total DoCrash in wntime | Cause | Resolution |
+| --- | --- | --- | --- |
+| 4/29 | ~5-10 minutes | Kernel crash on node running many VMs | Understood original issue better - came up during some maintentance / migration - Fixed procedure so issue won't happen again. |
+
+> Report generated with [tackle](https://github.com/robcxyz/tackle-box)

progress-reports/infrastructure/2506-infrastructure-progress-report.md

@@ -0,0 +1,64 @@
+# Tracker Monthly Report
+
+This is a progress update for 5/2025 on infrastructure provided by sudoblock including development of the tracker, indexer, and operation of RPC endpoints being used to support the ICON ecosystem.
+
+### Summary
+
+This past month was very busy on the ops side as we figured out solutions to several bottlenecks which required redeploying every service we are running on our cluster. This included rolling several k8s clusters and every blockchain VM and at this time is almost finished. While the tracker itself doesn't need the newly gained performance, because of these changes we saw a decrease in memory usage of about 25% or 50gb which is helpful when we are running 4 copies of the tracker (prod/dev environments with backups). Most notably though this has allowed us to run the ICON nodes much more efficiently and of course every other chain we might run. 
+
+### [icon-tracker-frontend](https://github.com/sudoblockio/icon-tracker-frontend)
+
+#### Closed Issues
+
+| Number | Name | Status | Created | Notes |
+| --- | --- | --- | --- | --- |
+| [566](https://github.com/sudoblockio/icon-tracker-frontend/pull/566) | fix: monthly inflation box overflow in Governance page #565 | closed | 2025-05-22T09:03:36Z | | 
+| [563](https://github.com/sudoblockio/icon-tracker-frontend/pull/563) | feat: added total available voted amt in voting page #552 | closed | 2025-04-21T12:19:47Z | | 
+| [562](https://github.com/sudoblockio/icon-tracker-frontend/pull/562) | chore(main): release 0.7.0 | closed | 2025-04-21T11:52:11Z | | 
+
+#### Open Issues
+
+| Number | Name | Status | Created | Notes |
+| --- | --- | --- | --- | --- | 
+| [565](https://github.com/sudoblockio/icon-tracker-frontend/issues/565) | Governance page style | open | 2025-05-22T08:59:18Z | | 
+| [564](https://github.com/sudoblockio/icon-tracker-frontend/issues/564) | Modify  | open | 2025-05-16T10:25:32Z | | 
+| [524](https://github.com/sudoblockio/icon-tracker-frontend/issues/524) | Jan Sprint | open | 2025-01-20T12:11:35Z | | 
+
+### [icon-governance](https://github.com/sudoblockio/icon-governance)
+
+#### Closed Issues
+
+| Number | Name | Status | Created | Notes |
+| --- | --- | --- | --- | --- |
+| [60](https://github.com/sudoblockio/icon-governance/issues/60) | Update bond_percent formula | closed | 2025-02-01T19:34:34Z | | 
+
+#### Open Issues
+
+| Number | Name | Status | Created | Notes |
+| --- | --- | --- | --- | --- | 
+| [62](https://github.com/sudoblockio/icon-governance/pull/62) | chore(main): release 0.10.7 | open | 2025-02-13T08:11:01Z | | 
+
+### [balanced-backend](https://github.com/balancednetwork/balanced-backend)
+
+#### Closed Issues
+
+| Number | Name | Status | Created | Notes |
+| --- | --- | --- | --- | --- |
+| [83](https://github.com/balancednetwork/balanced-backend/pull/83) | fix: add skip on wETH address which does not exist #82 | closed | 2025-05-09T10:04:52Z | | 
+| [78](https://github.com/balancednetwork/balanced-backend/pull/78) | chore(main): release 0.8.4 | closed | 2024-11-21T09:16:44Z | | 
+
+#### Open Issues
+
+| Number | Name | Status | Created | Notes |
+| --- | --- | --- | --- | --- | 
+| [84](https://github.com/balancednetwork/balanced-backend/issues/84) | getTotalCollateral missing at non-near head block heights for loans contract | open | 2025-05-18T10:28:26Z | | 
+| [82](https://github.com/balancednetwork/balanced-backend/issues/82) | wETH has non-existant address | open | 2025-05-09T09:59:30Z | | 
+
+## Downtime Incidents
+
+| Date | Total Downtime | Cause | Resolution |
+| --- | --- | --- | --- |
+| 5/14 | 11 min | Upstream issue with BGP | Ordered a second cross-connect and added redundant peer |
+| 5/20 | No downtime but over 8 hours saw about 10% of requests failing | We forgot to remove 2 nodes from a LB pool which normally is ok for 1 node but 2 caused issues | Do the right config |
+
+> Report generated with [tackle](https://github.com/robcxyz/tackle-box)

progress-reports/infrastructure/tackle.yaml

@@ -3,7 +3,6 @@
   help: Pulls issues from github and builds a table so that monthly reports can be generated
   month->: input What number month are you writing report in ISO format (ie Feb=02)?
   day->: input What day to run till in ISO format (ie->01)? --default 01
-  github_token->: password Enter your github token --default {{get_env('GITHUB_TOKEN')}}
 
   exec:
     repo_names: