-
Notifications
You must be signed in to change notification settings - Fork 26
feat: fee algo performance metrics #215
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
segfault-magnet
wants to merge
15
commits into
master
Choose a base branch
from
feat/fee_algo_performance_metrics
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
fbfe2ea
initial version of fee analyzing
segfault-magnet a93a02e
fix simulation
segfault-magnet f225639
add finalization time
segfault-magnet 06c2826
plot l2 behind buildup
segfault-magnet 42efe52
cleanup
segfault-magnet 3cc2f14
remove dep
segfault-magnet 1b38d24
cleanup
segfault-magnet b175a21
move into separate files
segfault-magnet 22fb251
fix imports
segfault-magnet c87bc85
group imports
segfault-magnet 2ad5fb9
cleanup
segfault-magnet f30e2ad
use u64
segfault-magnet c6177ba
fix oom
segfault-magnet 463a07d
update integrity
segfault-magnet 0472b79
Merge branch 'master' into feat/fee_algo_performance_metrics
hal3e File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,267 +1,5 @@ | ||
| use std::time::Duration; | ||
|
|
||
| use actix_web::{HttpResponse, Responder, ResponseError, web}; | ||
| use anyhow::Result; | ||
| use eth::HttpClient; | ||
| use services::{ | ||
| fee_metrics_tracker::service::calculate_blob_tx_fee, | ||
| fees::{Api, FeesAtHeight, SequentialBlockFees, cache::CachingApi}, | ||
| state_committer::{AlgoConfig, SmaFeeAlgo}, | ||
| types::{DateTime, Utc}, | ||
| }; | ||
| use thiserror::Error; | ||
| use tracing::{error, info}; | ||
|
|
||
| use super::{ | ||
| models::{FeeDataPoint, FeeParams, FeeResponse, FeeStats}, | ||
| state::AppState, | ||
| utils::last_n_blocks, | ||
| }; | ||
|
|
||
| #[derive(Error, Debug)] | ||
| pub enum FeeError { | ||
| #[error("Internal Server Error: {0}")] | ||
| InternalError(String), | ||
|
|
||
| #[error("Bad Request: {0}")] | ||
| BadRequest(String), | ||
| } | ||
|
|
||
| impl ResponseError for FeeError { | ||
| fn error_response(&self) -> HttpResponse { | ||
| match self { | ||
| FeeError::InternalError(message) => { | ||
| HttpResponse::InternalServerError().body(message.clone()) | ||
| } | ||
| FeeError::BadRequest(message) => HttpResponse::BadRequest().body(message.clone()), | ||
| } | ||
| } | ||
| } | ||
|
|
||
| pub async fn index_html() -> impl Responder { | ||
| let contents = include_str!("index.html"); | ||
| HttpResponse::Ok() | ||
| .content_type("text/html; charset=utf-8") | ||
| .body(contents) | ||
| } | ||
|
|
||
| struct FeeHandler { | ||
| state: web::Data<AppState>, | ||
| params: FeeParams, | ||
| config: AlgoConfig, | ||
| seq_fees: SequentialBlockFees, | ||
| last_block_height: u64, | ||
| last_block_time: DateTime<Utc>, | ||
| sma_algo: SmaFeeAlgo<CachingApi<HttpClient>>, | ||
| } | ||
|
|
||
| impl FeeHandler { | ||
| async fn new(state: web::Data<AppState>, params: FeeParams) -> Result<Self, FeeError> { | ||
| let ending_height = Self::resolve_ending_height(&state, ¶ms).await?; | ||
| let start_height = ending_height.saturating_sub(params.amount_of_blocks); | ||
| let config = Self::parse_config(¶ms)?; | ||
| let seq_fees = Self::fetch_fees(&state, start_height, ending_height).await?; | ||
| let last_block = Self::get_last_block_info(&state, &seq_fees).await?; | ||
| let sma_algo = SmaFeeAlgo::new(state.fee_api.clone(), config); | ||
|
|
||
| Ok(Self { | ||
| state, | ||
| params, | ||
| config, | ||
| seq_fees, | ||
| last_block_height: last_block.0, | ||
| last_block_time: last_block.1, | ||
| sma_algo, | ||
| }) | ||
| } | ||
|
|
||
| async fn get_fees_response(&self) -> Result<FeeResponse, FeeError> { | ||
| let data = self.calculate_fee_data().await?; | ||
| let stats = self.calculate_statistics(&data); | ||
| Ok(FeeResponse { data, stats }) | ||
| } | ||
|
|
||
| async fn resolve_ending_height( | ||
| state: &web::Data<AppState>, | ||
| params: &FeeParams, | ||
| ) -> Result<u64, FeeError> { | ||
| if let Some(val) = params.ending_height { | ||
| Ok(val) | ||
| } else { | ||
| state.fee_api.current_height().await.map_err(|e| { | ||
| error!("Error fetching current height: {:?}", e); | ||
| FeeError::InternalError("Failed to fetch current height".into()) | ||
| }) | ||
| } | ||
| } | ||
|
|
||
| fn parse_config(params: &FeeParams) -> Result<AlgoConfig, FeeError> { | ||
| AlgoConfig::try_from(params.clone()).map_err(|e| { | ||
| error!("Error parsing config: {:?}", e); | ||
| FeeError::BadRequest("Invalid configuration parameters".into()) | ||
| }) | ||
| } | ||
|
|
||
| async fn fetch_fees( | ||
| state: &web::Data<AppState>, | ||
| start: u64, | ||
| end: u64, | ||
| ) -> Result<SequentialBlockFees, FeeError> { | ||
| state.fee_api.fees(start..=end).await.map_err(|e| { | ||
| error!("Error fetching sequential fees: {:?}", e); | ||
| FeeError::InternalError("Failed to fetch sequential fees".into()) | ||
| }) | ||
| } | ||
|
|
||
| async fn get_last_block_info( | ||
| state: &web::Data<AppState>, | ||
| seq_fees: &SequentialBlockFees, | ||
| ) -> Result<(u64, DateTime<Utc>), FeeError> { | ||
| let last_block = seq_fees.last(); | ||
| let last_block_time = state | ||
| .fee_api | ||
| .inner() | ||
| .get_block_time(last_block.height) | ||
| .await | ||
| .map_err(|e| { | ||
| error!("Error fetching last block time: {:?}", e); | ||
| FeeError::InternalError("Failed to fetch last block time".into()) | ||
| })? | ||
| .ok_or_else(|| { | ||
| error!("Last block time not found"); | ||
| FeeError::InternalError("Last block time not found".into()) | ||
| })?; | ||
| info!("Last block time: {}", last_block_time); | ||
| Ok((last_block.height, last_block_time)) | ||
| } | ||
|
|
||
| async fn calculate_fee_data(&self) -> Result<Vec<FeeDataPoint>, FeeError> { | ||
| let mut data = Vec::with_capacity(self.seq_fees.len()); | ||
|
|
||
| for block_fee in self.seq_fees.iter() { | ||
| let fee_data = self.process_block_fee(block_fee).await?; | ||
| data.push(fee_data); | ||
| } | ||
|
|
||
| Ok(data) | ||
| } | ||
|
|
||
| async fn process_block_fee(&self, block_fee: &FeesAtHeight) -> Result<FeeDataPoint, FeeError> { | ||
| let current_fee_wei = calculate_blob_tx_fee(self.params.num_blobs, &block_fee.fees); | ||
| let short_fee_wei = self | ||
| .fetch_fee(block_fee.height, self.config.sma_periods.short) | ||
| .await?; | ||
| let long_fee_wei = self | ||
| .fetch_fee(block_fee.height, self.config.sma_periods.long) | ||
| .await?; | ||
|
|
||
| let acceptable = self | ||
| .sma_algo | ||
| .fees_acceptable( | ||
| self.params.num_blobs, | ||
| self.params.num_l2_blocks_behind, | ||
| block_fee.height, | ||
| ) | ||
| .await | ||
| .map_err(|e| { | ||
| error!("Error determining fee acceptability: {:?}", e); | ||
| FeeError::InternalError("Failed to determine fee acceptability".into()) | ||
| })?; | ||
|
|
||
| let block_gap = self.last_block_height - block_fee.height; | ||
| let block_time = self.last_block_time - Duration::from_secs(12 * block_gap); | ||
|
|
||
| let convert = |wei| format!("{:.4}", (wei as f64) / 1e18); | ||
|
|
||
| Ok(FeeDataPoint { | ||
| block_height: block_fee.height, | ||
| block_time: block_time.to_rfc3339(), | ||
| current_fee: convert(current_fee_wei), | ||
| short_fee: convert(short_fee_wei), | ||
| long_fee: convert(long_fee_wei), | ||
| acceptable, | ||
| }) | ||
| } | ||
|
|
||
| async fn fetch_fee( | ||
| &self, | ||
| current_height: u64, | ||
| period: std::num::NonZeroU64, | ||
| ) -> Result<u128, FeeError> { | ||
| let fees = self | ||
| .state | ||
| .fee_api | ||
| .fees(last_n_blocks(current_height, period)) | ||
| .await | ||
| .map_err(|e| { | ||
| error!("Error fetching fees for period: {:?}", e); | ||
| FeeError::InternalError("Failed to fetch fees".into()) | ||
| })? | ||
| .mean(); | ||
| Ok(calculate_blob_tx_fee(self.params.num_blobs, &fees)) | ||
| } | ||
|
|
||
| fn calculate_statistics(&self, data: &[FeeDataPoint]) -> FeeStats { | ||
| let total_blocks = data.len() as f64; | ||
| let acceptable_blocks = data.iter().filter(|d| d.acceptable).count() as f64; | ||
| let percentage_acceptable = if total_blocks > 0.0 { | ||
| (acceptable_blocks / total_blocks) * 100.0 | ||
| } else { | ||
| 0.0 | ||
| }; | ||
|
|
||
| let gap_sizes = self.compute_gap_sizes(data); | ||
| let percentile_95_gap_size = Self::calculate_percentile(&gap_sizes, 0.95); | ||
| let longest_unacceptable_streak = gap_sizes.into_iter().max().unwrap_or(0); | ||
|
|
||
| FeeStats { | ||
| percentage_acceptable, | ||
| percentile_95_gap_size, | ||
| longest_unacceptable_streak, | ||
| } | ||
| } | ||
|
|
||
| fn compute_gap_sizes(&self, data: &[FeeDataPoint]) -> Vec<u64> { | ||
| let mut gap_sizes = Vec::new(); | ||
| let mut current_gap = 0; | ||
|
|
||
| for d in data { | ||
| if !d.acceptable { | ||
| current_gap += 1; | ||
| } else if current_gap > 0 { | ||
| gap_sizes.push(current_gap); | ||
| current_gap = 0; | ||
| } | ||
| } | ||
|
|
||
| if current_gap > 0 { | ||
| gap_sizes.push(current_gap); | ||
| } | ||
|
|
||
| gap_sizes | ||
| } | ||
|
|
||
| fn calculate_percentile(gaps: &[u64], percentile: f64) -> u64 { | ||
| if gaps.is_empty() { | ||
| return 0; | ||
| } | ||
|
|
||
| let mut sorted_gaps = gaps.to_vec(); | ||
| sorted_gaps.sort_unstable(); | ||
|
|
||
| let index = ((sorted_gaps.len() as f64) * percentile).ceil() as usize - 1; | ||
| sorted_gaps[index.min(sorted_gaps.len() - 1)] | ||
| } | ||
| } | ||
|
|
||
| pub async fn get_fees(state: web::Data<AppState>, params: web::Query<FeeParams>) -> impl Responder { | ||
| let handler = match FeeHandler::new(state.clone(), params.into_inner()).await { | ||
| Ok(h) => h, | ||
| Err(e) => return e.error_response(), | ||
| }; | ||
|
|
||
| match handler.get_fees_response().await { | ||
| Ok(response) => HttpResponse::Ok().json(response), | ||
| Err(e) => e.error_response(), | ||
| } | ||
| } | ||
| pub mod block_time_info; | ||
| pub mod error; | ||
| pub mod fee; | ||
| pub mod index; | ||
| pub mod simulate; | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| use actix_web::HttpResponse; | ||
| use serde_json::json; | ||
| use services::fees::Api; | ||
| use tracing::error; | ||
|
|
||
| use crate::{state::AppState, utils::ETH_BLOCK_TIME}; | ||
| pub async fn get_block_time_info(state: actix_web::web::Data<AppState>) -> HttpResponse { | ||
| let current_height = match state.fee_api.current_height().await { | ||
| Ok(h) => h, | ||
| Err(e) => { | ||
| error!("Error fetching current height: {:?}", e); | ||
| return HttpResponse::InternalServerError() | ||
| .body("Could not fetch current block height"); | ||
| } | ||
| }; | ||
|
|
||
| let last_block_time = match state.fee_api.inner().get_block_time(current_height).await { | ||
| Ok(Some(t)) => t, | ||
| _ => { | ||
| return HttpResponse::InternalServerError() | ||
| .body("Last block time not found".to_string()); | ||
| } | ||
| }; | ||
|
|
||
| HttpResponse::Ok().json(json!({ | ||
| "last_block_height": current_height, | ||
| "last_block_time": last_block_time.to_rfc3339(), | ||
| "block_interval": ETH_BLOCK_TIME | ||
| })) | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| use actix_web::{HttpResponse, ResponseError}; | ||
| use thiserror::Error; | ||
| use tracing::error; | ||
|
|
||
| #[derive(Error, Debug)] | ||
| pub enum FeeError { | ||
| #[error("Internal Server Error: {0}")] | ||
| InternalError(String), | ||
| #[error("Bad Request: {0}")] | ||
| BadRequest(String), | ||
| } | ||
|
|
||
| impl ResponseError for FeeError { | ||
| fn error_response(&self) -> HttpResponse { | ||
| match self { | ||
| FeeError::InternalError(msg) => HttpResponse::InternalServerError().body(msg.clone()), | ||
| FeeError::BadRequest(msg) => HttpResponse::BadRequest().body(msg.clone()), | ||
| } | ||
| } | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mostly moved into a separate file.