Add laydown-person detection and rescue approach (Unitree Go2) - #2635
Open
shicaih wants to merge 61 commits into
Open
Add laydown-person detection and rescue approach (Unitree Go2)#2635shicaih wants to merge 61 commits into
shicaih wants to merge 61 commits into
Conversation
github-actions
Bot
added
tests
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new “person lying down” safety/rescue behavior for the Unitree Go2 by introducing a VLM-based input detector, new alert/vision sequencing providers, and MPPI autonomy connector behavior to approach, lock-on near, and continue guidance until cleared.
Changes:
- Added
PersonLaydownDetectorRTSP+VLM input with latched alert/debounce behavior and optional debug logging. - Updated MPPI autonomy connector to support gentle path selection, “turn … slightly” recenter moves, one-move-per-vision-verdict gating, and arrival lock-on behavior.
- Added supporting plumbing: alert/vision sequence providers, multi-image VLM describe support, TTS priority/skip-when-busy behavior, and default timeouts.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| plugins/inputs/vlm/laydown_detector.go | New VLM+RTSP sensor that latches ALERT and publishes vision verdicts into the cortex loop. |
| plugins/actions/unitree/go2/autonomy/mppi.go | Adds gentle-turn selection, recenter rotation goals, alert/arrival gating, and per-vision-step motion constraints. |
| plugins/actions/unitree/go2/autonomy/mppi_test.go | Updates unit tests for the new issueGoal signature. |
| plugins/actions/unitree/go2/autonomy/move.go | Extends shared Move action schema with “turn left/right slightly”. |
| plugins/actions/speak/elevenlabs_tts.go | Adds skip_when_busy and uses tts.Priority to bypass silence throttling. |
| internal/runtime/runtime.go | Adds a timeout around cortex LLM calls. |
| internal/providers/vlm/describe.go | Adds multi-image requests and configurable image detail. |
| internal/providers/tts/state.go | Adds a global Priority flag for emergency speech behavior. |
| internal/providers/alert_state.go | New alert/arrival and VisionSeq state for coordinating emergency behavior. |
| internal/httpclient/httpclient.go | Adds a default HTTP client timeout. |
| config/unitree_go2_laydown_rescue.json5 | New config wiring the detector + MPPI + TTS behavior for the rescue mode. |
Comment on lines
+308
to
+321
| if strings.Contains(strings.ToUpper(text), "ALERT") { | ||
| if !s.alerted { | ||
| s.alertedAt = time.Now() | ||
| } | ||
| s.alerted = true | ||
| s.clearCount = 0 | ||
| s.lastAlert = text | ||
| tts.Priority.Store(true) | ||
| providers.SetPersonDownAlert(true) | ||
| if strings.Contains(strings.ToLower(text), "distance: near") { | ||
| providers.SetPersonDownArrived(true) | ||
| } | ||
| return text | ||
| } |
Comment on lines
+377
to
+391
| func (s *laydownDetector) FormattedLatestBuffer() string { | ||
| s.mu.Lock() | ||
| defer s.mu.Unlock() | ||
|
|
||
| if !s.hasLatest { | ||
| return "" | ||
| } | ||
|
|
||
| result := fmt.Sprintf("\n%s: '%s'\n", vlmDescriptor, s.latest.Message) | ||
|
|
||
| ts := time.Unix(0, int64(s.latest.Timestamp*1e9)) | ||
| providers.IO().AddInput(s.name, s.latest.Message, ts) | ||
|
|
||
| return result | ||
| } |
Comment on lines
+175
to
+178
| if providers.PersonDownArrived() { | ||
| c.log.Info("arrived - locked on, holding position", zap.String("action", action)) | ||
| return nil, nil | ||
| } |
Comment on lines
229
to
233
| // issueGoal selects a path (smallest-magnitude heading when gentlest, else random) | ||
| // and publishes a goal toward it. pathAngles is +right but the mppi goal frame is | ||
| // +left (CCW), so the angle is negated. | ||
| func (c *MPPIConnector) issueGoal(options []uint32, label string, gentlest bool) { | ||
| if len(options) == 0 { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a safety behavior to the Go2 patrol robot: detect a person lying on the ground, approach and lock on at a safe distance, and deliver spoken first-aid guidance until the person gets up. Driven by a new VLM-based input plugin plus an mppi-planner movement path. Lands as the
unitree_go2_laydown_rescueconfig.The control loop, per emergency: VLM verdict → cortex/LLM → one move → mppi executes → lock on
near→ hold & speak, leaving emergency mode only when a fresh "No person" verdict clears the alert.Key changes
New input —
PersonLaydownDetector(plugins/inputs/vlm/laydown_detector.go)Location in view(left/center/right, judged by the head in the robot's image frame) andDistance(near/far).ALERTacross brief VLM dropouts; triggers an immediate cortex tick on each fresh verdict.image_log_dir).mppi autonomy connector (
plugins/actions/unitree/go2/autonomy/mppi.go)turn … slightlyre-center rotations (short arc) and gentle (smallest-angle) path selection.nearverdict latches, hold position and ignore all moves for the rest of the emergency — turns only happen while approaching.pathAnglesis +right; the planner goal frame is +left/CCW, so the angle is negated).Providers (
internal/providers/alert_state.go):PersonDownAlert,PersonDownArrived, and aVisionSeqperception-frame counter.Supporting
Describer: multi-image requests + configurabledetail.Priorityflag (emergency speech bypasses the patrol silence-rate) andskip_when_busy.move.go: reverted to main except the two newturn … slightlyenum values + description — that's the shared action schema the mppi connector relies on. Themoveconnector itself is unused by this config.Testing
go build,go vet, and the autonomy package tests pass.🤖 Generated with Claude Code