Parkour bot project

.github/workflows/generate-definitions.yml

@@ -32,7 +32,23 @@ jobs:
         run: |
           git clone https://github.com/commandblock2/LiquidBounce.git lbng
           cd lbng
-          git checkout 9888c42aa793bdac38767839443de934e509e103  
+          git checkout 9888c42aa793bdac38767839443de934e509e103
+      - name: Create Mods Directory
+        run: mkdir -p lbng/run/mods
+
+      - name: Download and Install Baritone
+        uses: dawidd6/action-download-artifact@v11
+        with:
+          github_token: ${{secrets.BARITONE_DOWNLOAD_TOKEN}}
+          name: Artifacts
+          repo: cabaletta/baritone
+          workflow: gradle_build.yml
+          branch: 1.21.4
+          path: lbng/run/mods
+          workflow_conclusion: success
+      - name: Extract Baritone Fabric API JAR
+        run: |
+          find lbng/run/mods -name "baritone-api-fabric-*.jar" -exec mv {} lbng/run/mods/ \;
 
       - name: Download Latest ts-generator
         run: |

.roomodes

@@ -85,3 +85,60 @@ customModes:
       - command
       - mcp
     source: project
+  - slug: lbng-debugger
+    name: LBNG Debugger
+    description: Specialized for LBNG script debugging.
+    roleDefinition: |-
+      You are Roo, a highly skilled software engineer with extensive knowledge in many programming languages, frameworks, design patterns, and best practices.
+
+      You are in a paused breakpoint in Minecraft. Your primary goal is to assist the user in debugging LiquidBounce NextGen scripts by evaluating expressions and inspecting variables.
+
+      Roo will reference `@src/complete.ts` first to understand what Roo is dealing with, many are not required to import, if Roo need to know which are already there, see `node_modules/jvm-types/ambient/ambient.d.ts`.
+
+      Roo **must assume that classes use the yarn mapping on the latest Minecraft protocol and prioritize using native Minecraft classes**. **Avoid using ViaVersion classes unless there is no native Minecraft equivalent or it is explicitly required for a specific task.**
+
+      When Roo are not confident about if an API exists in the current API, Roo may look up the type information at `node_modules/jvm-types/`. For example, `node_modules/jvm-types/types/net/minecraft/client/MinecraftClient.d.ts` contains the definition for `export class MinecraftClient extends ReentrantThreadExecutor<() => void> implements WindowEventHandler, MinecraftClientAccessor {` and `node_modules/jvm-types/types/net/ccbluex/liquidbounce/utils/movement/MovementUtilsKt.d.ts` contains `export class MovementUtilsKt extends Object {`
+
+      Roo will use `Outter$Inner` syntax to access inner classes. For example, if `PlayerMoveC2SPacket` has an inner class `PositionAndOnGround`, Roo would refer to it as `PlayerMoveC2SPacket$PositionAndOnGround`. When importing such classes, Roo will import them directly from their respective `.d.ts` files, e.g., `import { PlayerMoveC2SPacket$PositionAndOnGround } from "jvm-types/net/minecraft/network/packet/c2s/play/PlayerMoveC2SPacket$PositionAndOnGround";`.
+
+      Roo will use `import` instead of `Java.type` for imports from JVM, as the custom `require`/`import` implementation will be redirected to the graaljs's `Java.type`, direct usage of `Java.type` will lose all type information.
+
+      ---
+      **API Usage and Event Handling in LiquidBounce NextGen (Debugging Context):**
+
+      Roo's primary source of truth for LiquidBounce NextGen API definitions is the `node_modules/jvm-types/` directory. Always consult these type definitions, especially when:
+      *   Roo is not confident about the existence or exact signature of an API.
+      *   Roo is working with events or Kotlin extension functions.
+      *   Roo encounter type errors.
+
+      *   **Kotlin Extension Functions:** When migrating Kotlin code to TypeScript, pay special attention to Kotlin extension functions (e.g., `object.extensionFunction(...)` or `object.property`). These often compile to static methods in a utility class (frequently ending in `Kt`, like `RenderShortcutsKt`) where the extended object is passed as the first argument. Before assuming a direct method on the extended object, consult the `node_modules/jvm-types/` definitions for such utility classes or static methods that mimic the extension function's behavior. If a direct translation is not apparent or causes type errors, prioritize finding the corresponding static utility method.
+      *   **`FunctionN` Types and Type Errors (General Context):** For Kotlin `FunctionN` types (e.g., `Function0`, `Function1`, `Function2`) in `jvm-types` definitions, understand that direct TypeScript lambda expressions might not always satisfy these types due to strict type checking. If a TypeScript lambda provides the correct number and types of arguments and the expected return type, but the type system still reports an incompatibility with a `FunctionN` type *in contexts other than `mod.on()`*, use `// @ts-expect-error` to suppress the error. This indicates a potential limitation in the generated TypeScript definitions for these functional interfaces, and Roo should proceed assuming the runtime behavior is correct.
+
+      ---
+
+      **Special Handling for GraalJS Globals**:
+      Some of the node.js standard are not implemented, for example `setTimeout` and `clearTimeout`. Roo will look around the `src` folder to understand how other script accomplish similar functionality.
+
+      Seeing errors like the following may suggest that the generated TypeScript definition has some problems instead of Roo's work does. When seeing those errors, Roo will consider if they are errors made by the definition generator and ignore them if appropriate.
+
+      Conversion of type 'LivingEntity' to type 'Entity' may be a mistake because neither type sufficiently overlaps with the other. If this was intentional, convert the expression to 'unknown' first.
+
+      Types of property 'getBoundingBox' are incompatible.
+      Type '(pose: EntityPose) => Box' is not comparable to type '() => Box'.
+      Target signature provides too few arguments. Expected 1 or more, but got 0.
+
+      If Roo see `@ts-expect-error` around float-related operations, do not change them, as graaljs uses `double` for the type and cannot tolerate implicit precision loss.
+      ---
+
+      When evaluating expressions or assigning variables, you MUST use string substitution with backticks (` `) and the `${}` syntax. For variable assignment, use the format `${globalThis.yourVariable = value}` to ensure the variable is accessible globally and its assignment is reflected in the output.
+
+      Example evaluation: `${mc.player}`
+      Example assignment: `${globalThis.myServer = mc.getServer()}`
+
+      You have full access to the debugger and can evaluate any valid JavaScript/GraalJS expression.
+    whenToUse: >-
+      Use this mode when actively debugging LiquidBounce NextGen scripts, evaluating expressions,
+      or inspecting runtime variables within a paused debugger session.
+    groups:
+      - read
+      - mcp

.vscode/launch.json

@@ -5,9 +5,19 @@
             "name": "Attach",
             "type": "node",
             "request": "attach",
-            "debugServer": 4242,
+            "debugServer": 4243,
             "restart": true,
             "sourceMaps": true      
+        },
+        {
+            "name": "Attach by cdp (not recommended)",
+            "type": "node",
+            "request": "attach",
+            "port": 4242, // Default inspect port
+            "address": "localhost",
+            "restart": true,
+            "sourceMaps": true
         }
+
     ]
 }

packages/deep-learning-bot-utils/package.json

@@ -0,0 +1,7 @@
+{
+  "name": "deep-learning-bot-utils",
+  "version": "1.0.0",
+  "description": "Utils for making a deep learning bot, prioritizing pathfinding and traversability, then expanding to inventory and game context.",
+  "author": "commandblock2",
+  "license": "GPL-3.0-only"
+}

packages/deep-learning-bot-utils/spec-v1/SPEC-DATA.md

@@ -0,0 +1,128 @@
+### Bot Training Data Specification (Information Types)
+
+**Objective:** Define the data types required for iterative training of a Minecraft bot, prioritizing pathfinding and traversability, then expanding to inventory and game context.
+
+**Core Principles:**
+
+1.  **State-Action Pairing:** Data is structured as `(State, Action)` pairs.
+2.  **Modularity:** State components can be included or excluded based on training phase.
+3.  **Actionability:** Inventory and game item data must be represented to enable specific in-game actions.
+4.  **Contextual Relevance:** Data collected should accurately reflect the environment relevant to the bot's current skill focus.
+5.  **Temporal Awareness:** Incorporate historical context for improved decision-making and learning from past experience.
+
+---
+
+#### **Phase 1: Core Pathfinding & Traversability (Information Types)**
+
+**Player State:**
+
+*   **Position:** Absolute 3D world coordinates (`X`, `Y`, `Z`).
+*   **Velocity:** 3D velocity vector (`VX`, `VY`, `VZ`).
+*   **Look Direction:** Spherical coordinates (`Yaw`, `Pitch`).
+*   **Player Pose:** Categorical state (e.g., `STANDING`, `SNEAKING`, `SPRINTING`).
+*   **Ground Proximity:** Boolean indicating if the player is on the ground.
+*   **Predicted Passive Next Tick State:** The player's simulated position and velocity at the next tick, assuming no active input from the bot, but accounting for current game physics (gravity, potion effects, block friction).
+    *   `Predicted_Pos_X`, `Predicted_Pos_Y`, `Predicted_Pos_Z`
+    *   `Predicted_Vel_X`, `Predicted_Vel_Y`, `Predicted_Vel_Z`
+*   **Historical Player States:** A fixed-size list (e.g., last 3-5 ticks) of:
+    *   **Previous Position:** (`X`, `Y`, `Z`) at past tick.
+    *   **Previous Velocity:** (`VX`, `VY`, `VZ`) at past tick.
+    *   **Previous Look Direction:** (`Yaw`, `Pitch`) at past tick.
+    *   **Previous Player Pose:** Categorical state at past tick.
+    *   **Previous Fall Distance:**.
+
+**Local Environment Scan (Collision Box List):**
+
+*   A collection of collision boxes from nearby environmental elements within a defined spatial radius, plus dynamic areas of interest. Each collision box contains:
+    *   **Bounding Box Coordinates:** The precise geometric bounds (`minX`, `minY`, `minZ`, `maxX`, `maxY`, `maxZ`) of the collision box, normalized relative to the player position.
+    *   **Relative Position:** Center point of the collision box relative to the player (`centerX`, `centerY`, `centerZ`).
+    *   **Box Dimensions:** Width, height, and depth of the collision box (`width`, `height`, `depth`).
+    *   **Element Identifier:** A unique string label for the element type that owns this collision box (e.g., `minecraft:stone`, `minecraft:water`, `minecraft:boat`).
+    *   **Traversability Data:** A categorization indicating how the player can interact with this collision box:
+        *   `SOLID_WALKABLE`
+        *   `FLUID`
+        *   `OBSTRUCTION`
+        *   `AIR`
+        *   `LIQUID_PLACEABLE`
+        *   `PLACEABLE_BLOCK`
+        *   `OTHER`
+    *   **Element State Properties:** Specific configuration data for the element (e.g., directionality of stairs, state of a farmland block, door open/closed).
+    *   **Area Source Type:** Categorical indicator of how this collision box was included in the scan:
+        *   `FIXED_RADIUS` - Collision box within fixed scanning radius around player
+        *   `DYNAMIC_INTEREST` - Collision box included due to area of interest output
+    *   **Box Validity:** Boolean indicating if this collision box slot contains valid data (used for padding in fixed-size arrays).
+
+**Simplified Inventory Snapshot:**
+
+*   **Hotbar Slots:** A fixed-size array (e.g., 9 elements) representing the player's hotbar. For each slot:
+    *   **Item Identifier:** String label for the item (e.g., `minecraft:water_bucket`, `null` for empty).
+    *   **Item Quantity:** The count of that item in the slot.
+*   **Key Utility Indicators:** Boolean flags for critical items relevant to traversal:
+    *   `hasWaterBucket`
+    *   `hasPlaceableBlocks`
+
+**Baritone Reference Data (Training Phase Only):**
+
+*   **Current Target Path:** A sequence of waypoints representing Baritone's computed optimal path to the current goal.
+    *   **Path Waypoints:** Array of 3D coordinates (`X`, `Y`, `Z`) representing the sequence of blocks to traverse.
+    *   **Path Length:** Total number of waypoints in the current path.
+    *   **Next Waypoint Index:** Index of the next waypoint the bot should move toward.
+    *   **Estimated Completion Time:** Baritone's estimate of ticks required to complete the path.
+*   **Path Metadata:**
+    *   **Path Computation Tick:** The game tick when this path was computed by Baritone.
+    *   **Goal Coordinates:** The target destination (`GoalX`, `GoalY`, `GoalZ`) for this path.
+    *   **Path Validity:** Boolean indicating if the path is still valid (no environmental changes detected).
+
+---
+
+#### **Phase 2: PvP & Game Objective Awareness (Information Types)**
+
+**Player State (Additions):**
+
+*   **Equipment:** Identifiers and states of items held in main and off-hand.
+
+**Targeting Information:**
+
+*   A collection of nearby entities within a defined radius. For each entity relevant to objectives or combat:
+    *   **Entity Identifier:** Unique ID.
+    *   **Entity Type:** Categorical label (e.g., `PLAYER`, `ITEM`, `MOB`).
+    *   **Entity Position:** Relative 3D coordinates.
+    *   **Entity State:** Key attributes like health, distance.
+    *   **Combat Relevance:** A flag indicating if the entity is considered an "enemy" or target for the current game mode.
+
+---
+
+**Action Types:**
+
+*   **Movement:** `MOVE` (with directional components), `JUMP`, `SNEAK`, `SPRINT`.
+*   **Interaction:** `LOOK` (with direction changes), `USE_ITEM` (specifying item/slot and target), `PLACE_BLOCK` (specifying item/slot, target position, and face).
+*   **Combat:** `ATTACK_ENTITY` (specifying target entity and arm swing), `SWAP_OFFHAND` (to switch items between hands).
+*   **Path of Interest Generation:** `GENERATE_AREA_OF_INTEREST` (specifying a sequence of 3D coordinates that define additional areas for detailed environmental scanning beyond the fixed radius).
+
+---
+
+#### **Phase 3: Network Manipulation & Latency Awareness (Information Types)**
+
+*   **Network State:**
+    *   **Current Client Latency:** Numerical value representing the round-trip time (ping) to the server in milliseconds.
+    *   **Server Tick Delta:** Numerical value indicating the difference between client and server tick counts, useful for gauging desync.
+    *   **Outgoing Packet Queue Size:** Numerical count of packets currently queued for transmission from the client.
+    *   **Incoming Packet Queue Size:** Numerical count of packets currently buffered for processing on the client.
+
+---
+
+
+#### **Phase 4: Inventory Management & Contextual Actions (Information Types)**
+
+
+**Local Environment Scan (Enhancements):**
+
+*   For each environmental element:
+    *   **Element State Properties:** Specific configuration data for the element (e.g., directionality of stairs, state of a farmland block).
+
+**Detailed Inventory Snapshot:**
+
+*   **Hotbar Slots:** (As in Phase 1)
+*   **Main Inventory Counts:** A mapping of `Item Identifier` to `Quantity` for key stackable items NOT on the hotbar.
+*   **Selected Hotbar Slot:** The index of the currently active hotbar slot.
+