feat(telemetry server): create Go Telemetry server and update cli to env #1

fishonamos · 2025-05-30T09:12:22Z

This PR enhances the telemetry server with several key features for improved usability, flexibility, and deployment.

Server Updates (Read/Search Endpoints)

How to Run

# Build the server
go build -o pyrevit-telemetryserver

# Run with environment variables
PYREVT_TELEMETRY_DB_CONNSTRING="mongodb://localhost:27017/atoma-test" \
PYREVT_TELEMETRY_SCRIPTS_TABLE="scripts" \
PYREVT_TELEMETRY_EVENTS_TABLE="events" \
./pyrevit-telemetryserver --port=8080

Added GET endpoints for reading and searching telemetry data:
- /api/v1/scripts/
- /api/v2/scripts/
- /api/v2/events/
These endpoints support pagination and query filtering for easier data retrieval.
Backend support added for both SQL and MongoDB.
Minor code cleanups and fixes (e.g., linter issues, missing imports, and type errors).

Environment Variable Support

CLI flags (e.g., --scripts, --events, --port, --debug, --trace) can now be overridden with environment variables like:
- PYREVT_TELEMETRY_SCRIPTS_TABLE
- PYREVT_TELEMETRY_EVENTS_TABLE
- PYREVT_TELEMETRY_PORT
- PYREVT_TELEMETRY_DB_CONNSTRING
CLI flags still take precedence over env vars if both are set.

Testing

I verified the following endpoints using curl and environment variables:
- GET /api/v1/status
- GET /api/v1/scripts/
- GET /api/v2/scripts/
- GET /api/v2/events/
- POST /api/v1/scripts/
- POST /api/v2/events/
- Server starts and listens on the correct port.
- Requests return expected responses (e.g., JSON payloads, appropriate error messages like "Invalid session ID").

jmcouffin · 2025-05-30T13:00:11Z

pyrevitlabs/pyRevit#2157

fishonamos · 2025-05-30T19:36:02Z

@jmcouffin This first pr is ready for review. I want to break the pr into two parts. The second pr will take care of the docker setup, general refactor, (and Grafana?)

dosymep · 2025-06-02T11:53:28Z

@fishonamos please add github actions ci\cd script, maybe it's worth adding a script for docker with github actions.

example configuration for go build (I recommend checking the script, I wrote it quickly):

name: main

on:
  workflow_dispatch:

  push:
    branches: [ main ]

jobs:
  build:
    runs-on: windows-latest
    steps:
      - uses: actions/checkout@v4

      # Setup go env: https://github.com/actions/setup-go
      - name: Setup Go
        uses: actions/setup-go@v5

      - name: Build
      - run: go build -o bin/pyrevit-telemetryserver
      
      # Upload docs site: https://github.com/marketplace/actions/upload-a-build-artifact
      - name: Upload Artifacts
        uses: actions/upload-artifact@v4
        with:
          name: bin
          path: bin

fishonamos · 2025-06-02T14:14:16Z

@fishonamos please add github actions ci\cd script, maybe it's worth adding a script for docker with github actions.

example configuration for go build (I recommend checking the script, I wrote it quickly):

name: main

on:
  workflow_dispatch:

  push:
    branches: [ main ]

jobs:
  build:
    runs-on: windows-latest
    steps:
      - uses: actions/checkout@v4

      # Setup go env: https://github.com/actions/setup-go
      - name: Setup Go
        uses: actions/setup-go@v5

      - name: Build
      - run: go build -o bin/pyrevit-telemetryserver
      
      # Upload docs site: https://github.com/marketplace/actions/upload-a-build-artifact
      - name: Upload Artifacts
        uses: actions/upload-artifact@v4
        with:
          name: bin
          path: bin

Yes. Let me push the build.

fishonamos · 2025-06-02T14:21:47Z

Updated @dosymep

sanzoghenzo

I realize that I should have moved the original code here before this PR 😅

Code changes seems fine, I threw some suggestions, but they can be done in future refactorings.

One great addition would be turning the curl tests you mentioned into unit tests with httptest; and maybe also tests on the config/envvars/cli flags part.

sanzoghenzo · 2025-06-02T09:21:37Z

cli/args.go

Do you think we could use a library like config to handle config, envs and cli args all at once declaratively?

This should also remove the need of the usage.go module

sanzoghenzo · 2025-06-02T09:29:57Z

cli/logger.go

Perhaps a note for later... could we use slog here?

Yes. We could do that in the next refactor pr. Slog will provide a more structured logging. I will note it in the list of next updates to make. I can do it

we should do that in a separate pr after this is merged? @sanzoghenzo because this already has too many changes.

Yes, let's save it for later.

I'll create new issues from these comments (and move/split the original issue from pyrevit repo)

sanzoghenzo · 2025-06-13T06:30:47Z

One great addition would be turning the curl tests you mentioned into unit tests with httptest; and maybe also tests on the config/envvars/cli flags part.

@fishonamos are you able to do that, so then we can kickstart the project?

fishonamos · 2025-06-13T07:27:03Z

One great addition would be turning the curl tests you mentioned into unit tests with httptest; and maybe also tests on the config/envvars/cli flags part.

@fishonamos are you able to do that, so then we can kickstart the project?

Yes. But we need to merge this first, then i will branch out and start working on it.

sanzoghenzo · 2025-06-13T08:27:36Z

One great addition would be turning the curl tests you mentioned into unit tests with httptest; and maybe also tests on the config/envvars/cli flags part.

@fishonamos are you able to do that, so then we can kickstart the project?

Yes. But we need to merge this first, then i will branch out and start working on it.

Why? I would rather add tests in this pr/branch to ensure everything is working, then start to work on the other features

fishonamos · 2025-06-13T08:36:34Z

One great addition would be turning the curl tests you mentioned into unit tests with httptest; and maybe also tests on the config/envvars/cli flags part.

@fishonamos are you able to do that, so then we can kickstart the project?

Yes. But we need to merge this first, then i will branch out and start working on it.

Why? I would rather add tests in this pr/branch to ensure everything is working, then start to work on the other features

Oh. I think I understand you now. You mean the endpoints I added in the description right? Definitely! they can be added. I will do that and push.

sanzoghenzo · 2025-07-08T14:49:49Z

Hi @fishonamos ! I totally forgot about this PR... is it ready for review?

fishonamos · 2025-07-08T15:05:31Z

Hi @fishonamos ! I totally forgot about this PR... is it ready for review?

Yes. I sent you a dm. I think you missed it.

sanzoghenzo · 2025-07-08T15:50:07Z

Hi @fishonamos ! I totally forgot about this PR... is it ready for review?

Yes. I sent you a dm. I think you missed it.

Oh, sorry for that... where did you sent it? I cannot find anything on discourse, but it might be that I deleted/ignored it because of the many moderation messages I got 😅

Anyway, here it comes my review...

sanzoghenzo

Here you are!

sanzoghenzo · 2025-07-08T14:58:34Z

.github/workflows/main.yml

+  docker:
+    runs-on: ubuntu-latest
+    needs: build
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v3
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Login to DockerHub # TODO: uncomment if pushing
+        # uses: docker/login-action@v3
+        # with:
+        #   username: ${{ secrets.DOCKERHUB_USERNAME }}
+        #   password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Build Docker image
+        run: docker build -t pyrevit-telemetryserver:latest .
+
+      # - name: Push Docker image # Will be uncommented if pushing
+      #   run: docker push pyrevit-telemetryserver:latest


This job should run only on "release" and sync the docker tag with the version.
Since this PR doesn't have a Dockerfile, it is better to remove this job and think about it later.

sanzoghenzo · 2025-07-08T15:03:24Z

.github/workflows/main.yml

+      - name: Run Tests
+        run: go test -v ./...


Test should be run before the build to ensure everything is OK

sanzoghenzo · 2025-07-08T15:17:11Z

server/server_test.go

+func TestStatusEndpoint(t *testing.T) {
+	router := setupTestRouter()
+
+	t.Run("GET /api/v1/status returns 200", func(t *testing.T) {
+		req := httptest.NewRequest("GET", "/api/v1/status", nil)
+		w := httptest.NewRecorder()
+		router.ServeHTTP(w, req)
+
+		if w.Code != http.StatusOK {
+			t.Errorf("expected 200 OK, got %d", w.Code)
+		}
+		if w.Body.Len() == 0 {
+			t.Errorf("expected non-empty body")
+		}
+	})
+}


I know it is overkill for this simple endpoint, but there's also the /api/v2/status endpoint. Since it is identical, a for loop of "v1" and "v2" should cover everything.

sanzoghenzo · 2025-07-08T15:22:04Z

.github/workflows/main.yml

+      - name: Setup Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: "1.22"


go.mod uses: "go 1.23.0" and "toolchain go1.24.2", should this variable reflect that?

sanzoghenzo · 2025-07-08T15:37:20Z

cli/args.go

+	if scriptTable == "" {
+		scriptTable = os.Getenv("PYREVT_TELEMETRY_SCRIPTS_TABLE")
+	}
+	if eventTable == "" {
+		eventTable = os.Getenv("PYREVT_TELEMETRY_EVENTS_TABLE")
+	}


This is another ranting for the future: do we really need to have dynamic table names? Especially now that you're introducing the APIs to get the data out of the DB....

sanzoghenzo · 2025-07-08T15:37:53Z

README.md

+
+## API Endpoints
+
+- `GET /api/v1/status` — Health check


There is also the /api/v2/status endpoint

feat(telemetry server): create Go Telemetry server and update cli to env

c6291f6

fishonamos marked this pull request as ready for review May 30, 2025 09:27

fishonamos marked this pull request as draft May 30, 2025 11:08

jmcouffin added the ODHack14 label May 30, 2025

fishonamos added 4 commits May 30, 2025 14:05

go tidy and update main

3eae1d2

add support os env variables and refac cli

d831892

setup mongo

ae66528

add documentation (readme)

7188b88

fishonamos marked this pull request as ready for review May 30, 2025 19:29

fishonamos mentioned this pull request Jun 2, 2025

Improve Telemetry Server pyrevitlabs/pyRevit#2157

Open

8 tasks

fishonamos added 2 commits June 2, 2025 15:18

add github actions script

68e3db1

fix yml

3a09a37

sanzoghenzo reviewed Jun 2, 2025

View reviewed changes

This was referenced Jun 2, 2025

Use library to handle config, envs and cli args all at once #2

Open

Use slog for logging #3

Open

fishonamos added 6 commits June 24, 2025 10:59

refactor server for testing

92a6663

fa- add server tests

4ae08bf

add go test

d461ee7

update test

8d430cc

update test to units and verbose test actions

bc190f4

workflow -v

3e4e4a7

add go sum

ec9ffda

sanzoghenzo requested changes Jul 8, 2025

View reviewed changes

feat(telemetry server): create Go Telemetry server and update cli to env #1

Are you sure you want to change the base?

feat(telemetry server): create Go Telemetry server and update cli to env #1

Uh oh!

Conversation

fishonamos commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Server Updates (Read/Search Endpoints)

How to Run

Environment Variable Support

Testing

Uh oh!

jmcouffin commented May 30, 2025

Uh oh!

fishonamos commented May 30, 2025

Uh oh!

dosymep commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fishonamos commented Jun 2, 2025

Uh oh!

fishonamos commented Jun 2, 2025

Uh oh!

sanzoghenzo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fishonamos Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sanzoghenzo commented Jun 13, 2025

Uh oh!

fishonamos commented Jun 13, 2025

Uh oh!

sanzoghenzo commented Jun 13, 2025

Uh oh!

fishonamos commented Jun 13, 2025

Uh oh!

sanzoghenzo commented Jul 8, 2025

Uh oh!

fishonamos commented Jul 8, 2025

Uh oh!

sanzoghenzo commented Jul 8, 2025

Uh oh!

sanzoghenzo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fishonamos commented May 30, 2025 •

edited

Loading

dosymep commented Jun 2, 2025 •

edited

Loading

fishonamos Jun 2, 2025 •

edited

Loading