Skip to content

docs: add Dalgo integration entity classification (Step 1) #1571

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pokhiii wants to merge 3 commits into
base: dev
Choose a base branch
from
Open

docs: add Dalgo integration entity classification (Step 1) #1571

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a15fae8
docs: add Dalgo integration entity classification (Step 1)
pokhiii Feb 17, 2026
dca07c0
docs: add Dalgo integration action plan
pokhiii Feb 17, 2026
b5e08c3
docs: add Dalgo integration API audit (Step 2)
pokhiii Feb 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 111 additions & 0 deletions docs/dalgo-integration/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# ColoredCow Action Plan: CiviCRM API Audit for Dalgo

## Step 1: Categorize entities by API type

You built these extensions as CiviCRM plugins. So you already know which are native CiviCRM entities and which are Goonj-custom. Classify each:

**Likely native CiviCRM APIs (APIv3/v4 out of the box):**
- Events → `Event` API
- Activities → `Activity` API
- Volunteer Information → `Contact` API (filtered by contact type/group/tag)
- Mailing Information → `Mailing` + `MailingJob` + `MailingEventBounce/Delivered/Opened` APIs

**Goonj-custom entities (built as CiviCRM plugins — need verification):**
- Collection Camp (Individual/Institution)
- Dropping Center (Individual/Institution)
- Dispatch
- Dispatch Acknowledgement
- Logistics
- Outcome
- Feedback
- GCOC

For each custom entity — confirm whether CiviCRM auto-exposes an API (it usually does for entities registered via `hook_civicrm_entityTypes`), or if a custom API wrapper is needed.

## Step 2: For each of the 14 entities, produce this
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Feb 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Fix the entity count mismatch.

Line 25 says “14 entities,” but the lists above total 12. This can confuse the audit scope. Either update the count or expand the list.

✏️ Proposed fix 
-## Step 2: For each of the 14 entities, produce this
+## Step 2: For each of the 12 entities, produce this
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
## Step 2: For each of the 14 entities, produce this
## Step 2: For each of the 12 entities, produce this
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@docs/dalgo-integration/README.md` at line 25, The header "Step 2: For each of
the 14 entities, produce this" is inconsistent with the lists above which total
12; either change the header to "12 entities" or add the two missing entities to
the lists so they total 14; update the text in the "Step 2" header (search for
the exact phrase "Step 2: For each of the 14 entities, produce this") and any
related references in the README to match the corrected entity count, and if
adding entities ensure their names follow the existing list format and ordering.


Hit the Goonj CiviCRM instance and document:

| Check | What to capture |
|-------|----------------|
| **API endpoint** | Exact entity name and action (e.g., `CollectionCamp.get`) |
| **Fields returned** | Full field list from the API response — then cross-check against what the CSV asks for |
| **Missing fields** | Fields the CSV requires but the API doesn't return |
| **Filters available** | Can you filter by `is_active`, date range, geography, contact type? |
| **Incremental sync** | Does the response include `created_date` and `modified_date`? Can you sort by `modified_date`? |
| **Pagination** | Does `limit` + `offset` work? What's the max page size? |
| **PII concern** | For Collection Camp, Dropping Center, Volunteers — the CSV says "avoid personal information like email, phone, address". Confirm these fields can be excluded in the API call or need to be stripped. |
| **Aggregation** | For Contributors (Collection Camp, Dropping Center, Events) — does the API return counts, or do you get raw records that Dalgo will aggregate? |

## Step 3: Test incrementals specifically

Dalgo was explicit — they need incremental sync to work. For each entity:

1. Call the API with `sort=modified_date ASC` and `options.limit=10`
2. Confirm `modified_date` updates when a record is edited
3. Confirm `created_date` exists and is reliable
4. If either is missing on a custom entity — that's a code fix ColoredCow needs to make before handing off

## Step 4: Handle the Contacts/Volunteers problem

The CSV asks for Volunteer info — name, contact ID, status (registered, inducted, active, inactive). But there are 3 lakh contacts.

- Check if volunteers are in a specific CiviCRM **Group** or tagged with a **Tag** that separates them from general contacts
- Test the `Contact.get` API with that group/tag filter + `is_active` filter
- Report the count — how many volunteers come back? If it's manageable (say <50k), Dalgo can sync them. If it's still huge, flag it.

## Step 5: Map the entity relationships

This is the part Goonj didn't document but you already know from building the system. Write down:

```
Collection Camp
├── has Dispatch(es)
│ └── has Dispatch Acknowledgement
├── has Logistics
├── has Outcome
├── has Feedback
├── has Volunteers (contacts)
└── has Contributors (contacts, need count only)

Dropping Center
└── (same sub-entity structure as Collection Camp?)

Event
├── has Participants (registered/attended)
└── has Contributions (monetary)

GCOC
└── (what links to what?)
```
Comment on lines +61 to +80
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Feb 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Add a language to the fenced code block.

Markdownlint (MD040) flags the block without a language; add one for clarity and tooling compatibility.

📝 Proposed fix 
-```
+```text
 Collection Camp
   ├── has Dispatch(es)
   │     └── has Dispatch Acknowledgement
   ├── has Logistics
   ├── has Outcome
   ├── has Feedback
   ├── has Volunteers (contacts)
   └── has Contributors (contacts, need count only)

 Dropping Center
   └── (same sub-entity structure as Collection Camp?)

 Event
   ├── has Participants (registered/attended)
   └── has Contributions (monetary)

 GCOC
   └── (what links to what?)
-```
+```
🧰 Tools
🪛 markdownlint-cli2 (0.20.0)

[warning] 61-61: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@docs/dalgo-integration/README.md` around lines 61 - 80, The fenced code block
containing the entity diagram (the block that starts with triple backticks and
the lines "Collection Camp ... GCOC") lacks a language hint; update the opening
fence to include a language token (e.g., change ``` to ```text) so Markdownlint
MD040 is satisfied and tooling can treat it as plain text while keeping the
block content unchanged.


Document the **foreign key** for each relationship — what field in Dispatch links back to Collection Camp? Is it `case_id`? `activity_id`? A custom field?

This is critical for Dalgo to build joins in the warehouse. You don't need Shivangi for this — you built the system.

## Step 6: Compile the deliverable

One document (or spreadsheet) with:

| Entity | API Endpoint | Fields Available | Fields Missing | Supports Incremental | Pagination | Related Entities & FK | Notes |
|--------|-------------|-----------------|----------------|---------------------|------------|----------------------|-------|
| | | | | | | | |

Plus the entity relationship map from Step 5.

## Step 7: Share with Dalgo

Hand off the document. At this point Dalgo has everything they need to build the Airbyte connector. The only open items should be:
- Any missing fields that need a code fix (with your timeline)
- Any entities where incremental sync needs to be added (with your timeline)

## Timeline

| Day | Work |
|-----|------|
| Day 1 | Steps 1–2: Categorize entities + hit the APIs, document fields |
| Day 2 | Steps 3–4: Test incrementals + figure out the contacts/volunteers filtering |
| Day 3 | Steps 5–6: Map relationships with foreign keys + compile the deliverable |
| Day 4 | Step 7: Share with Dalgo + fix any gaps found |

Four working days. No dependency on Goonj for any of this — you have the CiviCRM instance, you built the plugins, you have the CSV.
191 changes: 191 additions & 0 deletions docs/dalgo-integration/step-1-entity-classification.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,191 @@
# Dalgo Integration - Entity Classification

This document classifies CiviCRM entities in the Goonj CRM as **native** (built-in CiviCRM with APIv3/v4) or **custom** (defined by extensions via the ECK framework), to support integration with Dalgo.

## Architecture Overview

The Goonj CRM uses two mechanisms to model its domain entities:

1. **Native CiviCRM entities** - built-in entities accessed via `\Civi\Api4\{EntityName}` (e.g., `Event`, `Activity`, `Contact`, `Contribution`)
2. **ECK (Entity Construction Kit) entities** - custom entities created via the `de.systopia.eck` extension, accessed via API entity name `Eck_{EntityTypeName}`. All ECK entities are automatically API-exposed.

ECK entities share base fields defined in `CRM/Eck/DAO/Entity.php`: `id`, `title`, `subtype`, `created_id`, `modified_id`, `created_date`, `modified_date`.

## Summary Table

| # | Entity Name | Classification | API Entity Name | `created_date` | `modified_date` |
|---|-------------|---------------|----------------|----------------|-----------------|
| 1 | Collection Camp | Custom (ECK) | `Eck_Collection_Camp` | Yes | Yes |
| 2 | Dropping Center | Custom (ECK) | `Eck_Collection_Camp` | Yes | Yes |
| 3 | Events | Native | `Event` | Yes | No |
| 4 | Activities | Native | `Activity` | Yes | Yes |
| 5 | Volunteer Info | Native (Contact subtype) | `Contact` | Yes | Yes |
| 6 | GCOC | Mixed | `Eck_Institution_Visit`, `Contribution`, `Activity` | Varies | Varies |
| 7 | Mailing Info | Native | `Mailing` | Yes | Yes |
| 8 | Dispatch | Custom (ECK) | `Eck_Collection_Source_Vehicle_Dispatch` | Yes | Yes |
| 9 | Dispatch Ack | Custom field group on Dispatch | `Eck_Collection_Source_Vehicle_Dispatch` | Inherited | Inherited |
| 10 | Logistics | Custom field group on Collection Camp | `Eck_Collection_Camp` | Inherited | Inherited |
| 11 | Outcome | Custom field group on Collection Camp | `Eck_Collection_Camp` | Inherited | Inherited |
| 12 | Feedback | Custom (ECK) | `Eck_Collection_Source_Feedback` | Yes | Yes |

## Detailed Classification

### 1. Collection Camp (Individual/Institution)

- **Type:** Custom (ECK entity)
- **ECK Entity Type:** `Collection_Camp`
- **ECK Subtypes:** `Collection_Camp` (Individual), `Institution_Collection_Camp` (Institution)
- **API call:** `Eck_Collection_Camp.get` with `subtype:name = 'Collection_Camp'` or `'Institution_Collection_Camp'`
- **DB table:** `civicrm_eck_collection_camp`
- **Custom field groups:** `Collection_Camp_Core_Details`, `Collection_Camp_Intent_Details`, `Collection_Camp_QR_Code`, `Logistics_Coordination`, `Camp_Outcome`, `Volunteer_Camp_Feedback`
- **`created_date`:** Yes
- **`modified_date`:** Yes

Both Individual and Institution variants are subtypes of the same ECK entity type `Collection_Camp`.

### 2. Dropping Center (Individual/Institution)

- **Type:** Custom (ECK entity - subtype of `Collection_Camp`)
- **ECK Entity Type:** `Collection_Camp` (same entity type as Collection Camp)
- **ECK Subtypes:** `Dropping_Center` (Individual), `Institution_Dropping_Center` (Institution)
- **API call:** `Eck_Collection_Camp.get` with `subtype:name = 'Dropping_Center'` or `'Institution_Dropping_Center'`
- **DB table:** `civicrm_eck_collection_camp` (same table as Collection Camp)
- **Custom field groups:** `Dropping_Centre`, `Collection_Camp_Core_Details`, `Collection_Camp_QR_Code`
- **Companion entity:** `Eck_Dropping_Center_Meta` for monthly status tracking
- **`created_date`:** Yes
- **`modified_date`:** Yes

### 3. Events

- **Type:** Native CiviCRM
- **API call:** `Event.get`
- **DB table:** `civicrm_event`
- **`created_date`:** Yes
- **`modified_date`:** No (CiviCRM Events lack a native `modified_date`)

Used for Goonj-initiated events (Rural Planned Visits, Book Fairs, Chaupals, etc.) with event type codes defined in `config/eventCode.php`.

### 4. Activities

- **Type:** Native CiviCRM
- **API call:** `Activity.get`
- **DB table:** `civicrm_activity`
- **`created_date`:** Yes
- **`modified_date`:** Yes

Used for Induction, Volunteering, and Material Contribution tracking. Note: there is also a separate custom ECK entity `Eck_Collection_Camp_Activity` for camp-specific activity records.

### 5. Volunteer Information

- **Type:** Native CiviCRM (Contact with subtype)
- **API call:** `Contact.get` with `contact_sub_type:name = 'Volunteer'`
- **DB table:** `civicrm_contact`
- **Custom field group:** `Volunteer_fields`
- **`created_date`:** Yes
- **`modified_date`:** Yes

Volunteer status (registered, inducted, active, inactive) is tracked via custom fields and Induction activity status in `InductionService.php`.

### 6. GCOC (Material, Monetary, Visit tracking)

GCOC is a business concept spanning multiple entities, not a single entity.

| Sub-entity | Type | API call | `created_date` | `modified_date` |
|------------|------|----------|----------------|-----------------|
| Visit tracking | Custom (ECK) | `Eck_Institution_Visit.get` with `subtype:name = 'Urban_Visit'` | Yes | Yes |
| Material contribution | Native | `Activity.get` (with material contribution activity type) | Yes | Yes |
| Monetary contribution | Native | `Contribution.get` | No (`receive_date` instead) | No |

Event code mapping: `'Urban Visit' => 'GCOC'` in `config/eventCode.php`.

### 7. Mailing Information

- **Type:** Native CiviCRM
- **API call:** `Mailing.get`
- **DB table:** `civicrm_mailing`
- **`created_date`:** Yes
- **`modified_date`:** Yes

Note: The codebase primarily uses `\CRM_Utils_Mail::send()` for transactional emails (logistics notifications, feedback reminders), which is separate from the CiviMail `Mailing` entity.

### 8. Dispatch

- **Type:** Custom (ECK entity)
- **ECK Entity Type:** `Collection_Source_Vehicle_Dispatch`
- **API call:** `Eck_Collection_Source_Vehicle_Dispatch.get`
- **DB table:** `civicrm_eck_collection_source_vehicle_dispatch`
- **Custom field group:** `Camp_Vehicle_Dispatch` (Collection Camp, Date/Time, Bags, Weight, Vehicle Category, PU Center, etc.)
- **`created_date`:** Yes
- **`modified_date`:** Yes

### 9. Dispatch Acknowledgement

- **Type:** Not a separate entity - custom field group on the Dispatch entity
- **API call:** `Eck_Collection_Source_Vehicle_Dispatch.get` (same as Dispatch, reading `Acknowledgement_For_Logistics.*` fields)
- **Custom field group:** `Acknowledgement_For_Logistics` (fields: `No_of_bags_received_at_PU_Office`, `Ack_Email_Sent`, `Verified_By`)
- **`created_date`:** Inherited from parent Dispatch entity
- **`modified_date`:** Inherited from parent Dispatch entity

### 10. Logistics

- **Type:** Not a separate entity - custom field group on `Collection_Camp`
- **API call:** `Eck_Collection_Camp.get` (reading `Logistics_Coordination.*` fields)
- **Custom field group:** `Logistics_Coordination` (fields: `Camp_to_be_attended_by`, `Email_Sent`, `Self_Managed_By_Camp_Organiser`, `Feedback_Email_Sent`)
- **`created_date`:** Inherited from parent Collection Camp entity
- **`modified_date`:** Inherited from parent Collection Camp entity

### 11. Outcome

- **Type:** Not a separate entity - custom field group on `Collection_Camp`
- **API call:** `Eck_Collection_Camp.get` (reading `Camp_Outcome.*` fields)
- **Custom field group:** `Camp_Outcome` (fields: `Rate_the_camp`, `Camp_Status_Completion_Date`, `Filled_By`, `Last_Reminder_Sent`)
- **`created_date`:** Inherited from parent Collection Camp entity
- **`modified_date`:** Inherited from parent Collection Camp entity

### 12. Feedback

- **Type:** Custom (ECK entity)
- **ECK Entity Type:** `Collection_Source_Feedback`
- **API call:** `Eck_Collection_Source_Feedback.get`
- **DB table:** `civicrm_eck_collection_source_feedback`
- **Custom field group:** `Collection_Source_Feedback` (camp code, address, ratings, difficulties, images)
- **`created_date`:** Yes
- **`modified_date`:** Yes

## All ECK Entity Types

| ECK Entity Type | API Entity Name | DB Table |
|----------------|----------------|----------|
| `Collection_Camp` | `Eck_Collection_Camp` | `civicrm_eck_collection_camp` |
| `Collection_Camp_Activity` | `Eck_Collection_Camp_Activity` | `civicrm_eck_collection_camp_activity` |
| `Collection_Source_Feedback` | `Eck_Collection_Source_Feedback` | `civicrm_eck_collection_source_feedback` |
| `Collection_Source_Vehicle_Dispatch` | `Eck_Collection_Source_Vehicle_Dispatch` | `civicrm_eck_collection_source_vehicle_dispatch` |
| `Dropping_Center_Meta` | `Eck_Dropping_Center_Meta` | `civicrm_eck_dropping_center_meta` |
| `Institution_Visit` | `Eck_Institution_Visit` | `civicrm_eck_institution_visit` |

## All Collection_Camp Subtypes

| Subtype | Description |
|---------|-------------|
| `Collection_Camp` | Individual Collection Camp |
| `Dropping_Center` | Individual Dropping Center |
| `Institution_Collection_Camp` | Institutional Collection Camp |
| `Institution_Dropping_Center` | Institutional Dropping Center |
| `Goonj_Activities` | Individual Goonj Activities |
| `Institution_Goonj_Activities` | Institutional Goonj Activities |

## Verification Commands

```bash
# List all registered ECK entity types
cv api4 EckEntityType.get +s name

# Confirm API exposure and list all fields for an ECK entity
cv api4 Eck_<EntityType>.getFields

# Example: List Collection Camp fields
cv api4 Eck_Collection_Camp.getFields

# Example: Get Collection Camps with subtype filter
cv api4 Eck_Collection_Camp.get +w subtype:name=Collection_Camp +l 5
```
Loading