addres comments

Author: xzrderek

.env.example

@@ -5,7 +5,6 @@
 # Fireworks AI Credentials (for interacting with the Fireworks platform)
 # These are used by eval protocol to deploy evaluators to Fireworks, preview, etc.
 FIREWORKS_API_KEY="your_fireworks_api_key_here"
-FIREWORKS_ACCOUNT_ID="your_fireworks_account_id_here" # e.g., "fireworks" or your specific account
 
 # Optional: If targeting a non-production Fireworks API endpoint
 # FIREWORKS_API_BASE="https://dev.api.fireworks.ai"

development/CONTRIBUTING.md

@@ -66,12 +66,11 @@ For a streamlined local development experience, especially when managing multipl
     cp .env.example .env.dev
     ```
 2.  **Populate `.env.dev`:**
-    Open `.env.dev` and fill in the necessary environment variables, such as `FIREWORKS_API_KEY`, `FIREWORKS_ACCOUNT_ID`, and any other variables required for your development tasks (e.g., `E2B_API_KEY`).
+    Open `.env.dev` and fill in the necessary environment variables, such as `FIREWORKS_API_KEY` and any other variables required for your development tasks (e.g., `E2B_API_KEY`).
 
     Example content for `.env.dev`:
     ```
     FIREWORKS_API_KEY="your_dev_fireworks_api_key"
-    FIREWORKS_ACCOUNT_ID="abc"
     FIREWORKS_API_BASE="https://api.fireworks.ai"
     E2B_API_KEY="your_e2b_api_key"
     ```
@@ -92,15 +91,12 @@ Set the following environment variables. For development, you might use specific
 
 *   `FIREWORKS_API_KEY`: Your Fireworks AI API key.
     *   For development, you might use a specific dev key: `export FIREWORKS_API_KEY="your_dev_fireworks_api_key"`
-*   `FIREWORKS_ACCOUNT_ID`: Your Fireworks AI Account ID. This is used to scope operations to your account.
-    *   For development against a shared dev environment, this might be a common ID like `pyroworks-dev`: `export FIREWORKS_ACCOUNT_ID="pyroworks-dev"`
 *   `FIREWORKS_API_BASE`: (Optional) If you need to target a non-production Fireworks API endpoint.
     *   For development: `export FIREWORKS_API_BASE="https://dev.api.fireworks.ai"`
 
 Example for a typical development setup:
 ```bash
 export FIREWORKS_API_KEY="your_development_api_key"
-export FIREWORKS_ACCOUNT_ID="pyroworks-dev" # Or your specific dev account ID
 export FIREWORKS_API_BASE="https://dev.api.fireworks.ai" # If targeting dev API
 ```
 
@@ -110,11 +106,10 @@ Eval Protocol does not read `~/.fireworks/auth.ini` (or any firectl profiles). U
 
 **Credential Sourcing Order:**
 Eval Protocol prioritizes credentials as follows:
-1.  Environment Variables (`FIREWORKS_API_KEY`, optional `FIREWORKS_ACCOUNT_ID`)
+1.  Environment Variables (`FIREWORKS_API_KEY`)
 
 **Purpose of Credentials:**
 *   `FIREWORKS_API_KEY`: Authenticates your requests to the Fireworks AI service.
-*   `FIREWORKS_ACCOUNT_ID`: Identifies your account for operations like managing evaluators. Typically this is derived automatically from `FIREWORKS_API_KEY` via the `verifyApiKey` endpoint.
 *   `FIREWORKS_API_BASE`: Allows targeting different API environments (e.g., development, staging).
 
 **Other Environment Variables:**
@@ -490,11 +485,8 @@ This is perfect for development, webhook testing, or making your reward function
 If you encounter authentication issues:
 
 1.  **Check Credential Sources**:
-    *   Verify that `FIREWORKS_API_KEY` and `FIREWORKS_ACCOUNT_ID` are correctly set as environment variables.
-    *   If not using environment variables, ensure `~/.fireworks/auth.ini` exists, is correctly formatted, and contains the right `api_key` and `account_id` under the `[fireworks]` section.
-    *   Remember the priority: environment variables override the `auth.ini` file.
+    *   Verify that `FIREWORKS_API_KEY` is correctly set as an environment variable.
 2.  **Verify API Key Permissions**: Ensure the API key has the necessary permissions for the operations you are attempting.
-3.  **Check Account ID**: Confirm that the `FIREWORKS_ACCOUNT_ID` is correct for the environment you are targeting (e.g., `pyroworks-dev` for the dev API, or your personal account ID).
 4.  **API Base URL**: If using `FIREWORKS_API_BASE`, ensure it points to the correct API endpoint (e.g., `https://dev.api.fireworks.ai` for development).
 
 You can use the following snippet to check what credentials Eval Protocol is resolving:

development/CORE_STRATEGY.md

@@ -52,8 +52,8 @@ This roadmap outlines the development priorities. Tasks within each phase can be
     *   [ ] **Docstring Checking:** Enable and enforce docstring checking.
 *   **P1.5: Authentication System Refactor:**
     *   [ ] **Centralize Logic:** Create `eval_protocol/auth.py`.
-    *   [ ] **Configuration Methods:** Support environment variables and `auth.ini`.
-    *   [ ] **Documentation:** Clearly document `FIREWORKS_ACCOUNT_ID` usage and auth setup.
+    *   [ ] **Configuration Methods:** Support `FIREWORKS_API_KEY` (environment variable) as the single source of truth.
+    *   [ ] **Documentation:** Clearly document API-key-based auth and account id derivation.
     *   [ ] **Codebase Update:** Refactor `eval_protocol/evaluation.py` etc. to use new auth module.
 *   **P1.6: Build, Packaging, and Basic CI:**
     *   [ ] **`setup.py` Review:** Evaluate `openai` pinning, clean `extras_require`, populate `long_description`.

development/record_replay_testing_handoff.md

@@ -101,7 +101,6 @@ PORT=8001 python simulation_server.py --port 8001
 ### Environment Variables
 ```bash
 export FIREWORKS_API_KEY="your_dev_fireworks_api_key"
-export FIREWORKS_ACCOUNT_ID="your_account_id"
 export EP_PLAYBACK_FILE="/path/to/recording.jsonl"  # For playback mode
 ```
 

docs/cli_reference/cli_overview.mdx

@@ -329,7 +329,6 @@ The CLI recognizes the following environment variables:
 
 - `FIREWORKS_API_KEY`: Your Fireworks API key (required for deployment operations)
 - `FIREWORKS_API_BASE`: Base URL for the Fireworks API (defaults to `https://api.fireworks.ai`)
-- `FIREWORKS_ACCOUNT_ID`: Your Fireworks account ID (optional; typically derived automatically from `FIREWORKS_API_KEY`)
 - `MODEL_AGENT`: Default agent model to use (e.g., "openai/gpt-4o-mini")
 - `MODEL_SIM`: Default simulation model to use (e.g., "openai/gpt-3.5-turbo")
 

eval_protocol/auth.py

@@ -26,30 +26,21 @@ def get_fireworks_account_id() -> Optional[str]:
     """
     Retrieves the Fireworks Account ID.
 
-    The Account ID is sourced in the following order:
-    1. FIREWORKS_ACCOUNT_ID environment variable.
-    2. If an API key is available (env), resolve via verifyApiKey.
-
     Returns:
         The Account ID if found, otherwise None.
     """
-    account_id = os.environ.get("FIREWORKS_ACCOUNT_ID")
-    if account_id and account_id.strip():
-        logger.debug("Using FIREWORKS_ACCOUNT_ID from environment variable.")
-        return account_id.strip()
-
-    # Fallback: if API key is present, attempt to resolve via verifyApiKey (env)
+    # Account id is derived from the API key (single source of truth).
     try:
         api_key_for_verify = get_fireworks_api_key()
         if api_key_for_verify:
             resolved = verify_api_key_and_get_account_id(api_key=api_key_for_verify, api_base=get_fireworks_api_base())
             if resolved:
-                logger.debug("Using FIREWORKS_ACCOUNT_ID resolved via verifyApiKey: %s", resolved)
+                logger.debug("Resolved account id via verifyApiKey: %s", resolved)
                 return resolved
     except Exception as e:
-        logger.debug("Failed to resolve FIREWORKS_ACCOUNT_ID via verifyApiKey: %s", e)
+        logger.debug("Failed to resolve account id via verifyApiKey: %s", e)
 
-    logger.debug("Fireworks Account ID not found in environment variables or via verifyApiKey.")
+    logger.debug("Fireworks Account ID not found via verifyApiKey.")
     return None
 
 
@@ -107,7 +98,7 @@ def verify_api_key_and_get_account_id(
         # Header keys could vary in case; requests provides case-insensitive dict
         account_id = resp.headers.get("x-fireworks-account-id") or resp.headers.get("X-Fireworks-Account-Id")
         if account_id and account_id.strip():
-            logger.debug("Resolved FIREWORKS_ACCOUNT_ID via verifyApiKey: %s", account_id)
+            logger.debug("Resolved account id via verifyApiKey: %s", account_id)
             return account_id.strip()
         return None
     except Exception as e:

eval_protocol/cli_commands/create_rft.py

@@ -753,7 +753,7 @@ def create_rft_command(args) -> int:
 
     account_id = _ensure_account_id()
     if not account_id:
-        print("Error: FIREWORKS_ACCOUNT_ID not set and could not be resolved.")
+        print("Error: Could not resolve Fireworks account id from FIREWORKS_API_KEY.")
         return 1
 
     api_base = get_fireworks_api_base()

eval_protocol/cli_commands/utils.py

@@ -396,20 +396,11 @@ def _normalize_evaluator_id(evaluator_id: str) -> str:
 
 
 def _ensure_account_id() -> Optional[str]:
-    """Resolve and cache FIREWORKS_ACCOUNT_ID if possible."""
-    account_id = os.environ.get("FIREWORKS_ACCOUNT_ID")
-    if account_id and account_id.strip():
-        return account_id.strip()
-
+    """Resolve Fireworks account id from FIREWORKS_API_KEY via verifyApiKey."""
     api_key = get_fireworks_api_key()
     if not api_key:
         return None
-
-    resolved = verify_api_key_and_get_account_id(api_key=api_key, api_base=get_fireworks_api_base())
-    if resolved:
-        os.environ["FIREWORKS_ACCOUNT_ID"] = resolved
-        return resolved
-    return None
+    return verify_api_key_and_get_account_id(api_key=api_key, api_base=get_fireworks_api_base())
 
 
 def _extract_terminal_segment(resource_name: str) -> str:

eval_protocol/platform_api.py

@@ -302,23 +302,22 @@ def delete_fireworks_secret(
 
     # These should be set in your .env.dev, .env file (or shell environment) for this test to run
     # FIREWORKS_API_KEY="your_fireworks_api_key"
-    # FIREWORKS_ACCOUNT_ID="your_fireworks_account_id"
     # FIREWORKS_API_BASE="https://api.fireworks.ai" # or your dev/staging endpoint
 
     test_account_id = get_fireworks_account_id()
     test_api_key = get_fireworks_api_key()  # Not passed directly, functions will resolve
     test_api_base = get_fireworks_api_base()
 
     logger.info("Attempting to use the following configuration for testing Fireworks secrets API:")
-    logger.info(f"  Resolved FIREWORKS_ACCOUNT_ID: {test_account_id}")
+    logger.info(f"  Resolved account id (derived from API key): {test_account_id}")
     logger.info(f"  Resolved FIREWORKS_API_BASE: {test_api_base}")
     logger.info(
         f"  Resolved FIREWORKS_API_KEY: {'********' + test_api_key[-4:] if test_api_key and len(test_api_key) > 4 else 'Not set or too short'}"
     )
 
     if not test_account_id or not test_api_key or not test_api_base:
         logger.error(
-            "CRITICAL: FIREWORKS_ACCOUNT_ID, FIREWORKS_API_KEY, and FIREWORKS_API_BASE must be correctly set in environment or .env file to run this test."
+            "CRITICAL: FIREWORKS_API_KEY and FIREWORKS_API_BASE must be correctly set in environment or .env file to run this test."
         )
         import sys  # Make sure sys is imported if using sys.exit
 

examples/metrics/llm_resource_example/README.md

@@ -21,14 +21,13 @@ This example demonstrates how to use an LLM as a judge within the reward functio
 ```bash
 # Set your Fireworks API key
 export FIREWORKS_API_KEY=your_api_key_here
-export FIREWORKS_ACCOUNT_ID=pyroworks
 ```
 
 ## Running the Example
 
 ```bash
 # Run evaluation using the configuration file
-FIREWORKS_ACCOUNT_ID=pyroworks eval-protocol run --config-path conf --config-name simple_llm_judge_eval
+eval-protocol run --config-path conf --config-name simple_llm_judge_eval
 ```
 
 ## Configuration