Skip to content

README.md

Latest commit

 

History

History
442 lines (332 loc) · 9.71 KB

README.md

File metadata and controls

442 lines (332 loc) · 9.71 KB

Queue Plugin

Event batching and queuing with automatic flushing for efficient data transmission.

Features

  • Dual-trigger auto-flush: Time-based and size-based triggers
  • Manual flush: Flush on demand
  • Optional persistence: Save queue to localStorage
  • Size limiting: FIFO queue with configurable max size
  • Retry tracking: Track retry attempts per item
  • Page unload handling: Auto-flush on beforeunload
  • Lifecycle integration: Automatic cleanup on SDK destroy
  • Event emission: queue:add, queue:flush, queue:error

Installation

import { SDK } from '@prosdevlab/sdk-kit';
import { queuePlugin } from '@prosdevlab/sdk-kit-plugins';

const sdk = new SDK();
sdk.use(queuePlugin);

Configuration

await sdk.init({
  queue: {
    maxSize: 20,           // Auto-flush after 20 items (default: 20)
    flushInterval: 5000,   // Auto-flush every 5s (default: 5000ms)
    persist: false,        // Persist to localStorage (default: false)
    maxRetries: 3,         // Max retry attempts (default: 3)
    maxQueueSize: 1000     // Max queue size before FIFO (default: 1000)
  }
});

Blocking Mode

Disable auto-flush for manual control:

await sdk.init({
  queue: {
    maxSize: null,         // Disable size-based flush
    flushInterval: null    // Disable time-based flush
  }
});

// Add events
sdk.queue.add({ event: 'click' });
sdk.queue.add({ event: 'pageview' });

// Flush manually when ready
sdk.queue.flush();

API

add(data: unknown): string

Add an item to the queue. Returns a unique item ID.

const id = sdk.queue.add({ event: 'click', button: 'signup' });
console.log('Added:', id); // "1234567890-abc123"

remove(id: string): boolean

Remove an item from the queue by ID. Returns true if removed, false if not found.

const removed = sdk.queue.remove(id);
if (removed) {
  console.log('Item removed');
}

flush(): void

Manually flush all queued items. Emits queue:flush event.

sdk.queue.flush();

clear(): void

Clear all items from the queue without flushing.

sdk.queue.clear();

size(): number

Get the current number of items in the queue.

const count = sdk.queue.size();
console.log(`${count} items queued`);

configure(config: QueueConfig): void

Update queue configuration at runtime.

sdk.queue.configure({
  maxSize: 50,
  flushInterval: 10000
});

Logging

The queue plugin integrates with the logging plugin for operational visibility and debugging.

Setup

Load the logging plugin before the queue plugin:

import { SDK } from '@prosdevlab/sdk-kit';
import { loggingPlugin, queuePlugin } from '@prosdevlab/sdk-kit-plugins';

const sdk = new SDK({
  logging: { level: 'debug' },
  queue: { maxSize: 20 }
});

sdk.use(loggingPlugin).use(queuePlugin);
await sdk.init();

What Gets Logged

Event Level Data Logged
Item added debug queueSize
FIFO eviction (queue full) warn maxSize
Manual flush info trigger: 'manual', count
Auto-flush (size) debug trigger: 'size', count, threshold
Auto-flush (interval) debug trigger: 'interval', count
Flush complete debug count, duration (ms)
Flush error error error message, count

Privacy Guarantees

The queue plugin NEVER logs queued item data - only metadata like queue size, counts, and timing information.

// Safe: Only logs queue size, not data
sdk.queue.add({
  password: 'secret123',
  creditCard: '4111111111111111'
});
// Logs: { queueSize: 1 }

// Safe: Only logs count and trigger
sdk.queue.flush();
// Logs: { trigger: 'manual', count: 1 }
// Then: { count: 1, duration: 5 }

// Never logs: password, creditCard, or any queued data

Example: Queue Monitoring

sdk.logging.onLog((entry) => {
  if (entry.namespace === 'queue') {
    if (entry.message === 'Queue full, evicting oldest item') {
      // Alert on data loss
      console.error('Queue overflow! Increase maxQueueSize');
    }

    if (entry.message === 'Queue flushed' && entry.data?.count > 0) {
      // Track successful batches
      analytics.track('batch_sent', { count: entry.data.count });
    }
  }
});

Example: Performance Tracking

sdk.logging.onLog((entry) => {
  if (entry.namespace === 'queue' && entry.message === 'Queue flushed') {
    const { count, duration } = entry.data;
    const itemsPerSec = (count / duration) * 1000;

    console.log(`Flushed ${count} items in ${duration}ms (${itemsPerSec.toFixed(0)} items/sec)`);
  }
});

Load Order

The logging plugin is optional. If not loaded, all log calls are safely ignored:

✅ sdk.use(loggingPlugin).use(queuePlugin); // Logging enabled
✅ sdk.use(queuePlugin);                     // No logging (graceful)
❌ sdk.use(queuePlugin).use(loggingPlugin); // No logs (load order)

Events

queue:add

Emitted when an item is added to the queue.

sdk.on('queue:add', (item) => {
  console.log('Item added:', item);
  // item: { id, data, timestamp, retries }
});

queue:flush

Emitted when the queue is flushed. This is where you send the data.

sdk.on('queue:flush', async ({ items }) => {
  console.log(`Flushing ${items.length} items`);
  
  // Send via Transport plugin
  await sdk.transport.send({
    url: 'https://api.example.com/batch',
    method: 'fetch',
    data: { events: items.map(item => item.data) }
  });
});

queue:error

Emitted when an error occurs (e.g., queue size limit reached).

sdk.on('queue:error', (error) => {
  console.error('Queue error:', error);
});

Usage Examples

Basic Batching

import { SDK } from '@prosdevlab/sdk-kit';
import { queuePlugin, transportPlugin } from '@prosdevlab/sdk-kit-plugins';

const sdk = new SDK();
sdk.use(queuePlugin);
sdk.use(transportPlugin);

await sdk.init({
  queue: {
    maxSize: 20,
    flushInterval: 5000
  }
});

// Listen for flush and send data
sdk.on('queue:flush', async ({ items }) => {
  await sdk.transport.send({
    url: 'https://api.example.com/batch',
    data: { events: items }
  });
});

// Add events (auto-flushes after 20 items or 5s)
sdk.queue.add({ event: 'pageview', page: '/home' });
sdk.queue.add({ event: 'click', button: 'signup' });

With Persistence

import { SDK } from '@prosdevlab/sdk-kit';
import { queuePlugin, storagePlugin, transportPlugin } from '@prosdevlab/sdk-kit-plugins';

const sdk = new SDK();
sdk.use(storagePlugin);  // Required for persistence
sdk.use(queuePlugin);
sdk.use(transportPlugin);

await sdk.init({
  queue: {
    persist: true,  // Save to localStorage
    maxSize: 20,
    flushInterval: 5000
  }
});

// Queue survives page reloads!
sdk.queue.add({ event: 'click' });

Retry Logic

sdk.on('queue:flush', async ({ items }) => {
  try {
    await sdk.transport.send({
      url: 'https://api.example.com/batch',
      data: { events: items }
    });
  } catch (error) {
    // Re-add items with incremented retry count
    items.forEach(item => {
      if (item.retries < 3) {
        sdk.queue.add({
          ...item.data,
          _retries: item.retries + 1
        });
      } else {
        console.error('Max retries reached for item:', item.id);
      }
    });
  }
});

Architecture

Queue Item Structure

interface QueueItem {
  id: string;           // Unique ID (timestamp-random)
  data: unknown;        // User data
  timestamp: number;    // When added (Date.now())
  retries: number;      // Retry count (starts at 0)
}

Auto-Flush Triggers

The queue auto-flushes when:

  1. Size limit reached: items.length >= maxSize
  2. Time interval elapsed: Every flushInterval ms
  3. Page unload: beforeunload event
  4. SDK destroy: sdk:destroy event

FIFO Size Limiting

When the queue reaches maxQueueSize, the oldest item is dropped:

// maxQueueSize: 3
sdk.queue.add({ event: 'click1' });  // Queue: [1]
sdk.queue.add({ event: 'click2' });  // Queue: [1, 2]
sdk.queue.add({ event: 'click3' });  // Queue: [1, 2, 3]
sdk.queue.add({ event: 'click4' });  // Queue: [2, 3, 4] (1 dropped)

Best Practices

1. Choose Appropriate Batch Size

// High-frequency events (clicks, scrolls)
maxSize: 50,
flushInterval: 10000  // 10s

// Low-frequency events (pageviews)
maxSize: 10,
flushInterval: 5000   // 5s

2. Handle Flush Errors

Always handle errors in the queue:flush handler:

sdk.on('queue:flush', async ({ items }) => {
  try {
    await sendToServer(items);
  } catch (error) {
    console.error('Failed to send batch:', error);
    // Implement retry logic here
  }
});

3. Use Persistence for Critical Data

Enable persistence for important events that shouldn't be lost on page reload:

await sdk.init({
  queue: {
    persist: true  // Requires Storage plugin
  }
});

4. Monitor Queue Size

Track queue size to detect issues:

sdk.on('queue:add', () => {
  const size = sdk.queue.size();
  if (size > 100) {
    console.warn('Queue is getting large:', size);
  }
});

Browser Compatibility

  • Modern browsers: Full support (Chrome, Firefox, Safari, Edge)
  • IE11: Not supported (uses modern JavaScript features)
  • Node.js: Supported (no beforeunload handler)

Performance

  • Memory: O(n) where n is queue size
  • Add: O(1)
  • Remove: O(n)
  • Flush: O(n)
  • Size: O(1)

Related Plugins

  • Transport Plugin: Send batched events via HTTP
  • Storage Plugin: Required for persistence
  • Context Plugin: Add context to queued events

License

MIT