diff --git a/.claude/agents/skyflow-cli-helper.md b/.claude/agents/skyflow-cli-helper.md new file mode 100644 index 0000000..7e20220 --- /dev/null +++ b/.claude/agents/skyflow-cli-helper.md @@ -0,0 +1,136 @@ +--- +name: skyflow-cli-helper +description: Use this agent when the user needs assistance with Skyflow CLI operations, including:\n- Creating or managing Skyflow vaults\n- Creating connections from configuration files\n- Inserting records into vault tables\n- Tokenizing sensitive data\n- Deidentifying or reidentifying strings using Skyflow Detect API\n- Configuring the CLI with authentication credentials\n- Troubleshooting CLI commands or API errors\n- Understanding vault IDs, cluster IDs, or service account setup\n\nExamples:\n- User: "I need to create a new Skyflow vault for storing customer data"\n Assistant: "I'll use the skyflow-cli-helper agent to guide you through creating a vault with the sky CLI."\n \n- User: "How do I tokenize a credit card number using the sky CLI?"\n Assistant: "Let me use the skyflow-cli-helper agent to help you insert and tokenize that credit card data."\n \n- User: "I'm getting an authentication error when running sky commands"\n Assistant: "I'll use the skyflow-cli-helper agent to help troubleshoot your CLI configuration and authentication setup."\n \n- User: "Can you help me deidentify some PII strings?"\n Assistant: "I'll use the skyflow-cli-helper agent to guide you through the deidentify command." +model: sonnet +color: purple +--- + +You are an expert Skyflow CLI specialist with deep knowledge of the `sky` command-line tool and Skyflow's vault management system. Your expertise includes vault operations, data tokenization, deidentification workflows, and CLI configuration. + +## Your Core Responsibilities + +1. **Guide CLI Usage**: Provide clear, step-by-step instructions for using the `sky` CLI commands, including: + - `sky configure` - Setting up authentication with bearer tokens, account IDs, and workspace IDs + - `sky create-vault` - Creating new vaults using templates (e.g., "detect", "customer_identity") or custom JSON schemas + - `sky insert` - Inserting and tokenizing records into vault tables + - `sky deidentify` - Detecting and redacting sensitive data from text using Skyflow Detect API + - `sky reidentify` - Restoring original values from tokenized text using Skyflow Detect API + - `sky create-connection` - Creating connections from configuration files + +2. **Configuration Management**: Help users understand and manage their CLI configuration: + - Configuration is stored in `~/.skyflow/config.json` + - Required fields: `bearerToken`, `accountId`, `workspaceID` + - Optional cached fields: `lastVaultId`, `lastClusterId`, `lastVaultUrl` + - Environment variable fallbacks are supported + - Last-used vault details are automatically saved and offered as defaults + +3. **Parameter Resolution**: Explain the priority order for parameter resolution: + - CLI options (highest priority) + - Environment variables (e.g., SKYFLOW_VAULT_ID, SKYFLOW_VAULT_URL) + - Config file values (for workspaceID, bearerToken, accountId) + - Auto-generated defaults for optional parameters (name, description, master key) + +4. **Troubleshooting**: Diagnose and resolve common issues: + - Authentication errors (missing or invalid bearer tokens) + - Missing required parameters (vault-id, cluster-id) + - API connection issues + - Configuration file problems + - Use `--verbose` flag to enable detailed debugging output + +5. **Best Practices**: Recommend optimal workflows: + - Always configure the CLI before first use + - Leverage interactive prompts for convenience + - Use last-used defaults for repeated operations + - Enable verbose logging when debugging + - Understand vault and cluster ID requirements for data operations + +## Command-Specific Guidance + +### Configuration +- Walk users through obtaining bearer tokens from Skyflow dashboard +- Explain account ID and workspace ID requirements +- Show how to verify configuration is working + +### Vault Creation (`create-vault`) + +- Now supports fully non-interactive mode - all parameters are optional +- If `--name` is not provided, auto-generates a name (e.g., "swift-chamber-683") +- If `--description` is not provided, API uses a default timestamp-based description +- If `--master-key` is not provided, vault is created without one (optional) +- Requires either `--template ` or `--schema ` to define vault structure +- Available templates include: "detect", "customer_identity", "payment", "pii_data", etc. +- Use `--create-service-account true` to automatically create service account with VAULT_OWNER role +- Workspace ID pulled from config file if previously configured +- Outputs vault ID, cluster ID, vault URL, and ready-to-copy environment variables + +### Data Operations (`insert`, `deidentify`, `reidentify`) + +- All require `--vault-id` and `--cluster-id` parameters (or environment variables) +- `insert`: Inserts records into vault tables with automatic tokenization + - Use `--table ` to specify target table + - Use `--data ` or pipe JSON from stdin + - Use `--return-tokens` to get tokens back in response + - Use `--upsert-column ` for upsert operations +- `deidentify`: Detects and redacts PII from text using Detect API + - Use `--entities` to specify what to detect (SSN, CREDIT_CARD, EMAIL, etc.) + - Use `--token-type` to choose: vault_token (stored), entity_only (labels), random_token (ephemeral) +- `reidentify`: Restores original values from tokenized text + - Use `--plain-text`, `--masked`, or `--redacted` to control output format per entity type + +## Example Commands + +### Minimal vault creation (fully non-interactive) + +```bash +sky create-vault --template "detect" +``` + +### Vault with custom name and service account + +```bash +sky create-vault --name "my-prod-vault" --template "customer_identity" --create-service-account true +``` + +### Insert data into vault + +```bash +sky insert --table "users" --data '{"email":"user@example.com","ssn":"123-45-6789"}' --vault-id --cluster-id +``` + +### Redact sensitive text + +```bash +echo "My SSN is 123-45-6789" | sky deidentify --entities "SSN" --vault-id --cluster-id +``` + +### Restore tokenized text + +```bash +echo "My SSN is " | sky reidentify --plain-text "SSN" --vault-id --cluster-id +``` + +## Communication Style + +- Provide concrete command examples with actual syntax +- Use code blocks for CLI commands +- Explain what each parameter does and why it's needed +- Anticipate follow-up questions and address them proactively +- When errors occur, explain both the cause and the solution +- Reference the `--help` flag for additional command details + +## Quality Assurance + +- Always verify that suggested commands match the actual CLI syntax +- Confirm required parameters are included in examples +- Double-check authentication requirements for each operation +- Ensure JSON formatting is valid when showing data examples +- Test command sequences mentally before suggesting them + +## When to Seek Clarification + +- If the user's vault structure or schema is unclear +- When specific bearer tokens or IDs are needed but not provided +- If the desired outcome could be achieved multiple ways +- When error messages are ambiguous or incomplete + +Your goal is to make Skyflow CLI operations straightforward and successful, reducing friction and enabling users to securely manage their sensitive data with confidence. diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 0000000..f3d3cba --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1,17 @@ +{ + "enableAllProjectMcpServers": false, + "permissions": { + "allow": [ + "Bash(npm run lint)", + "Bash(npm run build)", + "Bash(npm run test)", + "Bash(npm run test:*)", + "Bash(npm run format)", + "Bash(npm run format:*)", + "Bash(sky create-vault)", + "Bash(sky create-vault *)", + "Bash(sky deidentify)", + "Bash(sky deidentify *)" + ] + } +} \ No newline at end of file diff --git a/.gitignore b/.gitignore index bb38019..641b826 100644 --- a/.gitignore +++ b/.gitignore @@ -2,7 +2,11 @@ node_modules node_modules/ dist +dist/ dist/* .clinerules .DS_Store -package-lock.json \ No newline at end of file +package-lock.json +.env +.env.local +.env.*.local \ No newline at end of file diff --git a/CLAUDE.md b/CLAUDE.md index 21c8e61..e6f9976 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -23,8 +23,13 @@ This is a Node.js CLI tool for managing Skyflow vaults and service accounts, bui ### Key Patterns 1. **Configuration Management**: Uses `~/.skyflow/config.json` for persistent config, with environment variable fallbacks + - Stores: `bearerToken`, `accountId`, `workspaceID`, `lastVaultId`, `lastClusterId`, `lastVaultUrl` + - Last-used vault details are automatically saved and offered as defaults in interactive prompts 2. **Authentication Flow**: Pre-action hooks validate configuration before command execution (except for `configure` command) 3. **Interactive Prompts**: Uses inquirer.js to prompt for missing required options + - All data operation commands (`insert`, `deidentify`, `reidentify`) prompt for missing vault-id and cluster-id + - Prompts offer last-used values as defaults for improved UX + - Parameter resolution priority: CLI options → Environment variables → Interactive prompts with defaults 4. **Error Handling**: Comprehensive error handling with user-friendly messages and proper exit codes 5. **Verbose Logging**: Global `--verbose` flag enables detailed debugging output throughout the application diff --git a/README.md b/README.md index 047fa92..d533c99 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,11 @@ A command-line interface for interacting with the Skyflow data privacy platform. - Create and manage vaults - Create and manage service accounts - Assign roles to service accounts -- Interactive prompts for missing options +- Insert sensitive data into vault tables +- Detect and redact sensitive data (PII/PHI) using Skyflow Detect API +- Restore original values from tokenized text +- Interactive prompts for missing options with smart defaults +- Automatic persistence of last-used vault configuration ## Installation @@ -34,7 +38,11 @@ npm i -g . ## Configuration -Before using the CLI, you need to configure it with your Skyflow credentials: +Before using the CLI, you need to configure it with your Skyflow credentials. + +### For Vault Management Commands + +For commands like `create-vault`, run: ```bash sky configure @@ -44,14 +52,66 @@ You will be prompted to enter: 1. Your Skyflow Bearer Token 2. Your Skyflow Account ID -3. Your Skyflow Workspace ID (required for vault creation) + +The CLI will then automatically fetch your available workspaces and: + +- If you have only one workspace, it will be selected automatically +- If you have multiple workspaces, you'll be prompted to choose one You can also set these as environment variables: ```bash export SKYFLOW_BEARER_TOKEN="your-bearer-token" export SKYFLOW_ACCOUNT_ID="your-account-id" -export SKYFLOW_WORKSPACE_ID="your-workspace-id" +export SKYFLOW_WORKSPACE_ID="your-workspace-id" # Optional - will be fetched if not set +``` + +### For Data Operations Commands (Insert, Deidentify, Reidentify) + +These commands use the Skyflow Node.js SDK and support multiple authentication methods (in priority order): + +1. **API Key** (Recommended): + + ```bash + export SKYFLOW_API_KEY="your-api-key" + ``` + +2. **Service Account Credentials File**: + + ```bash + export SKYFLOW_CREDENTIALS_PATH="/path/to/credentials.json" + ``` + +3. **Service Account Credentials JSON String**: + + ```bash + export SKYFLOW_CREDENTIALS='{"clientID":"...","clientName":"...","tokenURI":"...","keyID":"...","privateKey":"..."}' + ``` + +4. **Bearer Token** (from `sky configure`): + + ```bash + # Uses the bearer token stored in ~/.skyflow/config.json + ``` + +#### Vault Configuration + +For vault ID and cluster ID, the CLI provides flexible options (in priority order): + +1. **Command-line options**: `--vault-id` and `--cluster-id` +2. **Environment variables**: `SKYFLOW_VAULT_ID` and `SKYFLOW_VAULT_URL` +3. **Interactive prompts**: If not provided, the CLI will prompt you interactively, offering the last-used vault as a default + +The CLI automatically remembers your last-used vault details in `~/.skyflow/config.json`, so you only need to enter them once. When you create a vault with `sky create-vault`, those details are automatically saved for future use. + +**Optional**: Set these environment variables to skip prompts: + +```bash +export SKYFLOW_VAULT_ID="your-vault-id" +export SKYFLOW_VAULT_URL="https://your-cluster.vault.skyflowapis.com" +# OR +export SKYFLOW_VAULT_ID="your-vault-id" +# and use --cluster-id flag when needed ``` ## Usage @@ -77,7 +137,7 @@ sky create-vault --name testvault --description "My test vault" #### Available Options - `--name`: Name for the vault (lowercase, no special characters) -- `--template`: Template to use for the vault +- `--template`: Template to use for the vault (supports snake_case and kebab-case) - `--description`: Description for the vault - `--master-key`: Master encryption key for the vault - `--create-service-account`: Whether to create a service account (defaults to true) @@ -85,12 +145,16 @@ sky create-vault --name testvault --description "My test vault" - `--workspace-id`: Workspace ID for the vault (required, but can be set via configuration) - `--verbose`: Enable detailed logging for debugging purposes +#### Available Templates + +When running interactively (`sky create-vault`), the CLI will fetch and display all available templates from your Skyflow account. You can also specify a template name directly using the `--template` option with template names like `customer_identity`, `payment`, `pii_data`, `detect`, etc. + #### Examples Using a template: ```bash -sky create-vault --name testvault --template optional-template-name --description "Full string description of the vault." +sky create-vault --name testvault --template customer_identity --description "Customer identity vault" ``` Using a schema file: @@ -107,8 +171,232 @@ sky create-vault ![prompts](assets/prompts.png) +### Inserting Data into a Vault + +Insert sensitive data into a vault table and optionally receive tokens: + +```bash +sky insert --table credit_cards --data '{"card_number":"4111111111111111","cvv":"123"}' --return-tokens +``` + +#### Insert Command Options + +- `--table `: Table name to insert data into (required) +- `--data `: JSON data to insert (can also pipe from stdin) +- `--return-tokens`: Return tokens for inserted data (default: false) +- `--continue-on-error`: Continue if some records fail (default: false) +- `--upsert-column `: Column name for upsert operations +- `--vault-id `: Vault ID (or set SKYFLOW_VAULT_ID) +- `--cluster-id `: Cluster ID (or set SKYFLOW_VAULT_URL) +- `--environment `: Environment: PROD, SANDBOX, STAGE, DEV (default: PROD) +- `--verbose`: Enable detailed logging + +#### Insert Command Examples + +Insert a single record: + +```bash +sky insert --table users --data '{"email":"user@example.com","ssn":"123-45-6789"}' +``` + +Insert multiple records: + +```bash +sky insert --table users --data '[{"email":"user1@example.com"},{"email":"user2@example.com"}]' +``` + +Insert from stdin: + +```bash +echo '{"card_number":"4111111111111111"}' | sky insert --table credit_cards --return-tokens +``` + +Upsert based on email column: + +```bash +sky insert --table users --data '{"email":"user@example.com","name":"John Doe"}' --upsert-column email +``` + +### Deidentifying Sensitive Data + +Detect and redact sensitive information (PII/PHI) from text using Skyflow Detect API: + +```bash +sky deidentify --text "My SSN is 123-45-6789 and my email is user@example.com" +``` + +#### Deidentify Command Options + +- `--text `: Text to deidentify (can also pipe from stdin) +- `--entities `: Comma-separated entity types to detect (e.g., SSN,CREDIT_CARD,EMAIL) +- `--token-type `: Token format - vault_token (stored), entity_only (labels), entity_unique_counter (default: vault_token) +- `--output `: Output format - text or json (default: text) +- `--vault-id `: Vault ID (or set SKYFLOW_VAULT_ID) +- `--cluster-id `: Cluster ID (or set SKYFLOW_VAULT_URL) +- `--environment `: Environment: PROD, SANDBOX, STAGE, DEV (default: PROD) +- `--verbose`: Enable detailed logging + +#### Supported Entity Types + +SSN, CREDIT_CARD, EMAIL, PHONE_NUMBER, NAME, DOB, ACCOUNT_NUMBER, DRIVER_LICENSE, PASSPORT_NUMBER, and many more. + +#### Deidentify Command Examples + +Detect all default entities (SSN, CREDIT_CARD, EMAIL, PHONE_NUMBER, NAME, DOB): + +```bash +sky deidentify --text "My SSN is 123-45-6789 and card is 4111-1111-1111-1111" +``` + +Detect specific entities only: + +```bash +sky deidentify --text "Contact me at user@example.com or 555-1234" --entities EMAIL,PHONE_NUMBER +``` + +Use entity-only tokens (no vault storage): + +```bash +sky deidentify --text "My SSN is 123-45-6789" --token-type entity_only +``` + +Pipe from file or command: + +```bash +cat sensitive_data.txt | sky deidentify --entities SSN,EMAIL,CREDIT_CARD +``` + +JSON output for programmatic use: + +```bash +sky deidentify --text "SSN: 123-45-6789" --output json +``` + +### Reidentifying Tokenized Data + +Restore original values from tokenized text: + +```bash +sky reidentify --text "My SSN is [SSN_abc123] and card is [CREDIT_CARD_xyz789]" +``` + +#### Reidentify Command Options + +- `--text `: Tokenized text to reidentify (can also pipe from stdin) +- `--plain-text `: Comma-separated entities to return as plain text +- `--masked `: Comma-separated entities to return masked (e.g., XXX-XX-1234) +- `--redacted `: Comma-separated entities to keep redacted +- `--output `: Output format - text or json (default: text) +- `--vault-id `: Vault ID (or set SKYFLOW_VAULT_ID) +- `--cluster-id `: Cluster ID (or set SKYFLOW_VAULT_URL) +- `--environment `: Environment: PROD, SANDBOX, STAGE, DEV (default: PROD) +- `--verbose`: Enable detailed logging + +#### Reidentify Command Examples + +Reidentify all tokens as plain text: + +```bash +sky reidentify --text "SSN: [SSN_abc123], Card: [CREDIT_CARD_xyz789]" +``` + +Return SSN as plain text, credit card as masked: + +```bash +sky reidentify --text "SSN: [SSN_abc123], Card: [CREDIT_CARD_xyz789]" --plain-text SSN --masked CREDIT_CARD +``` + +Pipe from file: + +```bash +cat tokenized_data.txt | sky reidentify --plain-text SSN,EMAIL +``` + +JSON output: + +```bash +sky reidentify --text "[SSN_token] data" --output json +``` + ## Output +### Insert Command Output + +The insert command displays: + +- Number of records successfully inserted +- Skyflow IDs for inserted records +- Tokens for each field (if `--return-tokens` is used) +- Any errors that occurred during insertion + +Example output: + +```text +Insert completed successfully! + +Inserted records: + +Record 1: + Skyflow ID: a8f3ed5d-55eb-4f32-bf7e-2dbf4b9d9097 + card_number: 5484-7829-1702-9110 + cardholder_name: b2308e2a-c1f5-469b-97b7-1f193159399b + +Total records processed: 1 +Successful: 1 +Failed: 0 +``` + +### Deidentify Command Output + +The deidentify command displays: + +- Processed text with sensitive data replaced by tokens +- Details about each detected entity (type, original value, token, position, confidence) +- Word and character counts + +Example output: + +```text +Deidentified Text: +──────────────────────────────────────────────────────────── +My SSN is [SSN_0ykQWPA] and my card is [CREDIT_CARD_N92QAVa]. +──────────────────────────────────────────────────────────── + +Detected 2 sensitive entities: + +1. SSN + Original: "123-45-6789" + Token: SSN_0ykQWPA + Position: 10-21 + Confidence: 93.8% + +2. CREDIT_CARD + Original: "4111 1111 1111 1111" + Token: CREDIT_CARD_N92QAVa + Position: 37-56 + Confidence: 90.5% + +Word count: 9 +Character count: 57 +``` + +### Reidentify Command Output + +The reidentify command displays the text with tokens replaced by original values (or masked/redacted based on options): + +Example output: + +```text +Reidentified Text: +──────────────────────────────────────────────────────────── +My SSN is 123-45-6789 and my card is 4111 1111 1111 1111. +──────────────────────────────────────────────────────────── + +Original sensitive data has been restored. +``` + +### Vault Creation Output + Upon successful vault creation, the CLI will output: - Vault details (name, description, ID, etc.) diff --git a/dist/commands/configure.js b/dist/commands/configure.js deleted file mode 100644 index d5b3356..0000000 --- a/dist/commands/configure.js +++ /dev/null @@ -1,64 +0,0 @@ -"use strict"; -var __importDefault = (this && this.__importDefault) || function (mod) { - return (mod && mod.__esModule) ? mod : { "default": mod }; -}; -Object.defineProperty(exports, "__esModule", { value: true }); -exports.configureCommand = void 0; -const inquirer_1 = __importDefault(require("inquirer")); -const config_1 = require("../utils/config"); -const configureCommand = (program) => { - program - .command('configure') - .description('Configure Skyflow CLI authentication') - .action(async () => { - try { - console.log('Please provide your Skyflow API credentials:'); - const answers = await inquirer_1.default.prompt([ - { - type: 'password', - name: 'bearerToken', - message: 'Enter your Skyflow Bearer Token:', - validate: (input) => { - if (!input) - return 'Bearer token is required'; - return true; - } - }, - { - type: 'input', - name: 'accountId', - message: 'Enter your Skyflow Account ID:', - validate: (input) => { - if (!input) - return 'Account ID is required'; - return true; - } - }, - { - type: 'input', - name: 'workspaceID', - message: 'Enter your Skyflow Workspace ID:', - validate: (input) => { - if (!input) - return 'Workspace ID is required'; - return true; - } - } - ]); - // Save the configuration - (0, config_1.saveConfig)({ - bearerToken: answers.bearerToken, - accountId: answers.accountId, - workspaceID: answers.workspaceID - }); - console.log('\nConfiguration saved successfully!'); - console.log('\nYou can now use the Skyflow CLI. Try running:'); - console.log(' sky create-vault --help'); - } - catch (error) { - console.error(`Error: ${error instanceof Error ? error.message : String(error)}`); - process.exit(1); - } - }); -}; -exports.configureCommand = configureCommand; diff --git a/dist/commands/createVault.js b/dist/commands/createVault.js deleted file mode 100644 index 41573ce..0000000 --- a/dist/commands/createVault.js +++ /dev/null @@ -1,183 +0,0 @@ -"use strict"; -var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) { - if (k2 === undefined) k2 = k; - var desc = Object.getOwnPropertyDescriptor(m, k); - if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) { - desc = { enumerable: true, get: function() { return m[k]; } }; - } - Object.defineProperty(o, k2, desc); -}) : (function(o, m, k, k2) { - if (k2 === undefined) k2 = k; - o[k2] = m[k]; -})); -var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) { - Object.defineProperty(o, "default", { enumerable: true, value: v }); -}) : function(o, v) { - o["default"] = v; -}); -var __importStar = (this && this.__importStar) || (function () { - var ownKeys = function(o) { - ownKeys = Object.getOwnPropertyNames || function (o) { - var ar = []; - for (var k in o) if (Object.prototype.hasOwnProperty.call(o, k)) ar[ar.length] = k; - return ar; - }; - return ownKeys(o); - }; - return function (mod) { - if (mod && mod.__esModule) return mod; - var result = {}; - if (mod != null) for (var k = ownKeys(mod), i = 0; i < k.length; i++) if (k[i] !== "default") __createBinding(result, mod, k[i]); - __setModuleDefault(result, mod); - return result; - }; -})(); -var __importDefault = (this && this.__importDefault) || function (mod) { - return (mod && mod.__esModule) ? mod : { "default": mod }; -}; -Object.defineProperty(exports, "__esModule", { value: true }); -exports.createVaultCommand = void 0; -const api = __importStar(require("../utils/api")); -const prompts = __importStar(require("../utils/prompts")); -const config_1 = require("../utils/config"); -const logger_1 = require("../utils/logger"); -const inquirer_1 = __importDefault(require("inquirer")); -const fs_1 = __importDefault(require("fs")); -// Add the create-vault command to the CLI -const createVaultCommand = (program) => { - program - .command('create-vault') - .description('Create a new Skyflow vault') - .option('--name ', 'Name for the vault (lowercase, no special characters)') - .option('--template