Skip to content

[c-930] use metrics v2 to trace relevant code paths #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
maxim-inj merged 17 commits into from
Apr 10, 2026
Merged

[c-930] use metrics v2 to trace relevant code paths #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
3161698
fix: leftover go mod tidy
dbrajovic Mar 20, 2026
dbaa145
chore: wire up metrics v2
dbrajovic Mar 20, 2026
d0704a8
chore: trace CListMempool
dbrajovic Mar 20, 2026
d633dc6
impl: extend node ctor to accept a metrics meter
dbrajovic Mar 23, 2026
2348fb5
chore: trace CListMempool
dbrajovic Mar 30, 2026
a91a6e6
chore: trace state store
dbrajovic Mar 30, 2026
12c20a0
chore: trace consensus Receive handler
dbrajovic Mar 30, 2026
31ce1d9
chore: trace state sync reactor
dbrajovic Mar 30, 2026
74c0343
chore: trace consensus handleMsg method
dbrajovic Mar 30, 2026
54c7b49
chore: fix test compilation
dbrajovic Mar 31, 2026
813ce3f
fix: set node meter from arg
dbrajovic Mar 31, 2026
ab25bba
fix: use proper ctx for recheckTxs
dbrajovic Mar 31, 2026
d1e31b5
Merge branch 'v1.x-inj' into c-930/trace-everything
dbrajovic Apr 6, 2026
5c4282e
Merge branch 'v1.x-inj' into c-930/trace-everything
maxim-inj Apr 10, 2026
55e059a
chore: refactor changes to make diff smaller
maxim-inj Apr 10, 2026
d876c8b
chore: stabilize tests
maxim-inj Apr 10, 2026
aeb863b
fix: fix test 04 in CI
maxim-inj Apr 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions abci/tutorials/abci-v2-forum-app/forum.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ func main() {
cfg.DefaultDBProvider,
nm.DefaultMetricsProvider(config.Instrumentation),
logger,
nil,
)

defer func() {
Expand Down
43 changes: 27 additions & 16 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
module github.com/cometbft/cometbft

go 1.23
toolchain go1.23.7
go 1.23.9

require (
github.com/BurntSushi/toml v1.4.0
Expand Down Expand Up @@ -33,15 +32,16 @@ require (
github.com/snikch/goodman v0.0.0-20171125024755-10e37e294daa
github.com/spf13/cobra v1.9.1
github.com/spf13/viper v1.19.0
github.com/stretchr/testify v1.10.0
golang.org/x/crypto v0.36.0
golang.org/x/net v0.37.0
google.golang.org/grpc v1.71.0
github.com/stretchr/testify v1.11.1
golang.org/x/crypto v0.41.0
golang.org/x/net v0.43.0
google.golang.org/grpc v1.75.0
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

python - <<'PY'
import json
import urllib.request

queries = [
    ("google.golang.org/grpc", "1.75.0"),
    ("go.opentelemetry.io/otel/sdk", "1.38.0"),
]

for pkg, version in queries:
    body = json.dumps({
        "package": {"ecosystem": "Go", "name": pkg},
        "version": version,
    }).encode()
    req = urllib.request.Request(
        "https://api.osv.dev/v1/query",
        data=body,
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req) as resp:
        payload = json.load(resp)
    print(json.dumps({
        "package": pkg,
        "version": version,
        "vulns": payload.get("vulns", []),
    }, indent=2))
PY

Repository: InjectiveLabs/cometbft

Length of output: 2648

🏁 Script executed:

cat go.mod | head -50

Repository: InjectiveLabs/cometbft

Length of output: 1787

🏁 Script executed:

sed -n '38p;141p' go.mod

Repository: InjectiveLabs/cometbft

Length of output: 147

🌐 Web query:

GO-2026-4762 GHSA-p77j-4mvh-x3m3 google.golang.org/grpc authorization bypass

💡 Result:

GO-2026-4762 (GHSA-p77j-4mvh-x3m3, CVE-2026-33186) is a critical authorization bypass vulnerability (CVSS 9.1) in google.golang.org/grpc versions before 1.79.3. The gRPC-Go server accepts HTTP/2 requests with a :path pseudo-header missing the leading slash (e.g., "Service/Method" instead of "/Service/Method"). It routes these to the correct handler, but authorization interceptors (e.g., google.golang.org/grpc/authz) evaluate the raw non-canonical path. Deny rules using canonical paths (with "/") fail to match, allowing bypass if a fallback allow rule exists. Impacts servers using path-based authorization interceptors with deny rules for canonical paths and fallback allows. Attackers send crafted HTTP/2 frames. Fixed in v1.79.3: rejects non-canonical :path with codes.Unimplemented before interceptors. Upgrade to v1.79.3 or later. Mitigate with validating interceptors or default-deny policies.

Citations:

🌐 Web query:

GO-2026-4394 GHSA-9h8m-3fm2-qjrq go.opentelemetry.io/otel PATH hijack RCE

💡 Result:

GO-2026-4394 (GHSA-9h8m-3fm2-qjrq, CVE-2026-24051) is a high-severity vulnerability in the OpenTelemetry Go SDK (go.opentelemetry.io/otel/sdk) enabling arbitrary code execution (RCE) via PATH hijacking on macOS/Darwin systems. Summary: The resource detection code in sdk/resource/host_id.go executes the 'ioreg' system command without an absolute path, relying on the PATH environment variable. An attacker able to modify PATH (local low-priv attacker) can place a malicious executable named 'ioreg' in a directory prioritized in PATH, leading to RCE in the application's context when resource.WithHostID or similar host detection is used. Affected versions: v1.21.0 to v1.39.0 (Go modules all versions in that range). Patched in: v1.40.0 via commit d45961b (uses absolute path for ioreg on macOS). Severity: High (CVSS 7.0: AV:L/AC:H/PR:L/UI:N/S:U/C:H/I:H/A:H). Official sources confirm the issue only affects macOS due to ioreg usage; other platforms use file reads or different safe methods. Mitigation: Upgrade to v1.40.0 or later. Avoid using resource.WithHostID if possible. Do not run in environments where PATH is attacker-controllable.

Citations:

Upgrade the vulnerable dependency pins before merging.

google.golang.org/grpc v1.75.0 contains a critical authorization-bypass vulnerability (GO-2026-4762 / GHSA-p77j-4mvh-x3m3, CVSS 9.1). Servers using path-based authorization interceptors are at risk when HTTP/2 requests with non-canonical :path pseudo-headers bypass deny rules. Upgrade to v1.79.3 or later.

go.opentelemetry.io/otel/sdk v1.38.0 contains a high-severity PATH-hijacking RCE vulnerability (GO-2026-4394 / GHSA-9h8m-3fm2-qjrq, CVSS 7.0) on macOS systems due to unqualified command execution. Upgrade to v1.40.0 or later.

Both versions are now in this diff.

🧰 Tools
🪛 OSV Scanner (2.3.5)

[CRITICAL] 38-38: google.golang.org/grpc 1.75.0: Authorization bypass in gRPC-Go via missing leading slash in :path in google.golang.org/grpc

(GO-2026-4762)

[CRITICAL] 38-38: google.golang.org/grpc 1.75.0: gRPC-Go has an authorization bypass via missing leading slash in :path

(GHSA-p77j-4mvh-x3m3)

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@go.mod` at line 38, Replace the vulnerable module versions in go.mod: bump
google.golang.org/grpc from v1.75.0 to v1.79.3 (or later) and bump
go.opentelemetry.io/otel/sdk from v1.38.0 to v1.40.0 (or later); after updating
the module lines for these symbols, run go mod tidy and re-run tests/build to
ensure dependency graph and imports resolve cleanly.

)

require github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7

require (
github.com/InjectiveLabs/metrics/v2 v2.0.0-beta.8
github.com/Masterminds/semver/v3 v3.3.1
github.com/go-git/go-git/v5 v5.14.0
github.com/goccmack/goutil v1.2.3
Expand All @@ -51,9 +51,9 @@ require (
github.com/oasisprotocol/curve25519-voi v0.0.0-20220708102147-0a8a51822cae
github.com/supranational/blst v0.3.14
golang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8
golang.org/x/sync v0.12.0
gonum.org/v1/gonum v0.15.1
google.golang.org/protobuf v1.36.5
golang.org/x/sync v0.16.0
gonum.org/v1/gonum v0.16.0
google.golang.org/protobuf v1.36.8
)

require (
Expand All @@ -64,6 +64,7 @@ require (
github.com/Nvveen/Gotty v0.0.0-20120604004816-cd527374f1e5 // indirect
github.com/ProtonMail/go-crypto v1.1.5 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/cloudflare/circl v1.6.0 // indirect
github.com/cockroachdb/errors v1.11.3 // indirect
Expand All @@ -85,7 +86,7 @@ require (
github.com/getsentry/sentry-go v0.31.1 // indirect
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
github.com/go-git/go-billy/v5 v5.6.2 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-sql-driver/mysql v1.7.1 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
Expand All @@ -95,6 +96,7 @@ require (
github.com/google/flatbuffers v25.2.10+incompatible // indirect
github.com/google/go-cmp v0.7.0 // indirect
github.com/gotestyourself/gotestyourself v2.2.0+incompatible // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.2 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/imdario/mergo v0.3.15 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
Expand Down Expand Up @@ -131,13 +133,20 @@ require (
github.com/xanzy/ssh-agent v0.3.3 // indirect
go.etcd.io/bbolt v1.4.0 // indirect
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
go.opentelemetry.io/otel v1.34.0 // indirect
go.opentelemetry.io/otel/metric v1.34.0 // indirect
go.opentelemetry.io/otel/trace v1.34.0 // indirect
go.opentelemetry.io/otel v1.38.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.38.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.38.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.38.0 // indirect
go.opentelemetry.io/otel/metric v1.38.0 // indirect
go.opentelemetry.io/otel/sdk v1.38.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.38.0 // indirect
go.opentelemetry.io/otel/trace v1.38.0 // indirect
go.opentelemetry.io/proto/otlp v1.7.1 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/sys v0.31.0 // indirect
golang.org/x/text v0.23.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250115164207-1a7da9e5054f // indirect
golang.org/x/sys v0.35.0 // indirect
golang.org/x/text v0.28.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250825161204-c5933d9347a5 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250825161204-c5933d9347a5 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
Expand All @@ -152,3 +161,5 @@ retract (
// superseeded by v0.38.3 because of ASA-2024-001
[v0.38.0, v0.38.2]
)

replace github.com/cometbft/cometbft/api => ./api
86 changes: 50 additions & 36 deletions go.sum
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion go.work
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
go 1.23
go 1.23.9

use (
.
Expand Down
79 changes: 38 additions & 41 deletions internal/clist/clist_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ package clist
import (
"fmt"
"runtime"
"sync"
"sync/atomic"
"testing"
"time"

Expand Down Expand Up @@ -73,55 +73,56 @@ func TestGCFifo(t *testing.T) {
t.Skipf("Skipping on non-amd64 machine")
}

const numElements = 1000000
l := New()
const numElements = 100000
finalized := func() *atomic.Int64 {
l := New()
finalized := new(atomic.Int64)

// Use a WaitGroup to wait for all finalizers to run.
var wg sync.WaitGroup
wg.Add(numElements)
type value struct {
Int int
}

type value struct {
Int int
}
for i := 0; i < numElements; i++ {
v := new(value)
v.Int = i
l.PushBack(v)
runtime.SetFinalizer(v, func(_ *value) {
finalized.Add(1)
})
}

for i := 0; i < numElements; i++ {
v := new(value)
v.Int = i
l.PushBack(v)
runtime.SetFinalizer(v, func(_ *value) {
wg.Done()
})
}
for el := l.Front(); el != nil; {
next := el.Next()
l.Remove(el)
el.DetachPrev()
el.DetachNext()
el = next
}

for el := l.Front(); el != nil; {
next := el.Next()
l.Remove(el)
el = next
}
if l.Len() != 0 {
t.Errorf("expected list to be empty, got %v elements", l.Len())
}

// Wait for all finalizers to run.
wg.Wait()
return finalized
}()

if l.Len() != 0 {
t.Errorf("expected list to be empty, got %v elements", l.Len())
}
assert.Eventually(t, func() bool {
runtime.GC()
return finalized.Load() == numElements
}, 30*time.Second, 50*time.Millisecond)
}

// This test is quite hacky because it relies on SetFinalizer
// which isn't guaranteed to run at all.
// This test exercises random removals. Finalizer-based GC assertions are
// covered by TestGCFifo and proved too nondeterministic here.
func TestGCRandom(t *testing.T) {
t.Helper()
if runtime.GOARCH != "amd64" {
t.Skipf("Skipping on non-amd64 machine")
}

const numElements = 1000000
const numElements = 100000
l := New()

// Use a WaitGroup to wait for all finalizers to run.
var wg sync.WaitGroup
wg.Add(numElements)

type value struct {
Int int
}
Expand All @@ -130,9 +131,6 @@ func TestGCRandom(t *testing.T) {
v := new(value)
v.Int = i
l.PushBack(v)
runtime.SetFinalizer(v, func(_ *value) {
wg.Done()
})
}

els := make([]*CElement, 0, numElements)
Expand All @@ -143,12 +141,11 @@ func TestGCRandom(t *testing.T) {
for _, i := range cmtrand.Perm(numElements) {
el := els[i]
l.Remove(el)
_ = el.Next()
el.DetachPrev()
el.DetachNext()
els[i] = nil
}

// Wait for all finalizers to run.
wg.Wait()

if l.Len() != 0 {
t.Errorf("expected list to be empty, got %v elements", l.Len())
}
Expand Down
2 changes: 1 addition & 1 deletion internal/consensus/byzantine_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,7 +288,7 @@ func TestByzantinePrevoteEquivocation(t *testing.T) {
assert.Equal(t, prevoteHeight, ev.Height())
}
}
case <-time.After(20 * time.Second):
case <-time.After(60 * time.Second):
t.Fatalf("Timed out waiting for validators to commit evidence")
}
}
Expand Down
12 changes: 12 additions & 0 deletions internal/consensus/reactor.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
package consensus

import (
"context"
"fmt"
"math/rand"
"reflect"
"sync"
"sync/atomic"
"time"

injmetrics "github.com/InjectiveLabs/metrics/v2"
cmtcons "github.com/cometbft/cometbft/api/cometbft/consensus/v1"
"github.com/cometbft/cometbft/internal/bits"
cstypes "github.com/cometbft/cometbft/internal/consensus/types"
Expand Down Expand Up @@ -51,6 +53,7 @@ type Reactor struct {
initialHeight int64 // under rsMtx

Metrics *Metrics
meter injmetrics.Meter
}

type ReactorOption func(*Reactor)
Expand All @@ -63,6 +66,7 @@ func NewReactor(consensusState *State, waitSync bool, options ...ReactorOption)
rs: consensusState.GetRoundState(),
initialHeight: consensusState.state.InitialHeight,
Metrics: NopMetrics(),
meter: injmetrics.NewNilMeter(),
}
conR.BaseReactor = *p2p.NewBaseReactor("Consensus", conR)
if waitSync {
Expand Down Expand Up @@ -265,6 +269,9 @@ func (conR *Reactor) Receive(e p2p.Envelope) {
panic(fmt.Sprintf("Peer %v has no state", e.Src))
}

_, stopFn := conR.meter.FuncTimingCtx(context.Background(), "Receive", injmetrics.Tag("msg", fmt.Sprintf("%T", msg)))
defer stopFn()

switch e.ChannelID {
case StateChannel:
switch msg := msg.(type) {
Expand Down Expand Up @@ -1079,6 +1086,11 @@ func ReactorMetrics(metrics *Metrics) ReactorOption {
return func(conR *Reactor) { conR.Metrics = metrics }
}

// ReactorMeter sets the tracing meter.
func ReactorMeter(meter injmetrics.Meter) ReactorOption {
return func(conR *Reactor) { conR.meter = meter }
}

// -----------------------------------------------------------------------------

// PeerState contains the known state of a peer, including its connection and
Expand Down
11 changes: 11 additions & 0 deletions internal/consensus/state.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import (
"strconv"
"time"

injmetrics "github.com/InjectiveLabs/metrics/v2"
"github.com/cosmos/gogoproto/proto"

cmtproto "github.com/cometbft/cometbft/api/cometbft/types/v1"
Expand Down Expand Up @@ -134,6 +135,7 @@ type State struct {

// for reporting metrics
metrics *Metrics
meter injmetrics.Meter

// offline state sync height indicating to which height the node synced offline
offlineStateSyncHeight int64
Expand Down Expand Up @@ -171,6 +173,7 @@ func NewState(
evpool: evpool,
evsw: cmtevents.NewEventSwitch(),
metrics: NopMetrics(),
meter: injmetrics.NewNilMeter(),
}
for _, option := range options {
option(cs)
Expand Down Expand Up @@ -220,6 +223,11 @@ func StateMetrics(metrics *Metrics) StateOption {
return func(cs *State) { cs.metrics = metrics }
}

// StateMeter sets the tracing meter.
func StateMeter(meter injmetrics.Meter) StateOption {
return func(cs *State) { cs.meter = meter }
}

// OfflineStateSyncHeight indicates the height at which the node
// statesync offline - before booting sets the metrics.
func OfflineStateSyncHeight(height int64) StateOption {
Expand Down Expand Up @@ -877,6 +885,9 @@ func (cs *State) receiveRoutine(maxSteps int) {

// state transitions on complete-proposal, 2/3-any, 2/3-one.
func (cs *State) handleMsg(mi msgInfo) {
_, stopFn := cs.meter.FuncTimingCtx(context.Background(), "handleMsg", injmetrics.Tag("msg", fmt.Sprintf("%T", mi.Msg)))
defer stopFn()

cs.mtx.Lock()
defer cs.mtx.Unlock()
var (
Expand Down
Loading
Loading