Skip to content

Update processing - #159

Merged
StefanoSperetta merged 73 commits into
mainfrom
update_processing
Apr 30, 2026
Merged

Update processing#159
StefanoSperetta merged 73 commits into
mainfrom
update_processing

Conversation

@StefanoSperetta

@StefanoSperetta StefanoSperetta commented Mar 1, 2024 •
edited
Loading

Copy link
Copy Markdown
Member

Update the processing and displaying of the data:

  • reduce the number of allocation for XTCEParsers: now it happens only once per thread
  • only call the processing thread of satellites that actually have a processor
  • avoid calling processing threads for satellites that have no frame to process
  • reduce the number of connections to influxDB: now it happens twice per raw frame to be processed, maxing ~10 connections per second

@StefanoSperetta
Adding extra help message to initbuckets.py
6a212e4
@StefanoSperetta
Set up the InfluxDB organization for grafana from environment variable
fa9edd5
@StefanoSperetta
Removing extra log messages
977c902
@StefanoSperetta
Setting display dates for Grafana dashboards
68d8d2e
@StefanoSperetta
Reduce the crowdsec log file size
3a6bd47
@StefanoSperetta
Solving the Django 4 deprecation warnings
1602483
@StefanoSperetta
Printing launch date and last received data in the main page
b414422
@StefanoSperetta
Increase the hasked_key size to 150 characters following djangorestfr…
55037bd 
…amework-api-key version 2.2.0
@StefanoSperetta
Store the user name with frames, not the Django ID
55a68c9
@StefanoSperetta
Start processign job imediatly when frames is received
1bd0bd0
@StefanoSperetta
Map zoom adjusted
7a15074
@StefanoSperetta
Cleanup debug messages
c52d490
@StefanoSperetta
Added norad id to satellite location api
3955e46
@StefanoSperetta
Adding satellite icon to map
956a5b1
@StefanoSperetta
Changed homepage main picture
1471898
@StefanoSperetta
Removing debug messages
70f8e50
@StefanoSperetta
Allow staff users to submit frames on behalf of other users
ad783e7
@StefanoSperetta
Added buffer reprocessing task
f067eba
@StefanoSperetta
Skip already processed messages in the query
263d19b
@StefanoSperetta
Added debug messages in scraper
d61339e
@StefanoSperetta
Speeding up buffer processing by ~4x
68d2b89
@StefanoSperetta
Removing debug logger calls
f80cbb7
@StefanoSperetta
Start all tasks once frames are received to allow multi-threaded proc…
a710be3 
…essing
@StefanoSperetta
Processing the raw bucket iteratively to allow multi-threaded processing
9d46229
@StefanoSperetta
Loading XTCETools only when starting the scheduler
a289b60
@StefanoSperetta
Removing debug logger call
581440c
@StefanoSperetta
Processing frames in chunks to be more responsive
02fe72c
@StefanoSperetta
Handle exceptions in the processign threads
d4f74e5
@StefanoSperetta
Removing bookkeeping files, frames to process fully handled via the d…
9cbd5a6 
…atabase
@StefanoSperetta
Removing debug logger messages
eba82d7
@StefanoSperetta
Improving the Delfi-PQ dashboard
27ac4e7
@StefanoSperetta
Removing legend from graphs with a single value
8e7422f
@StefanoSperetta
Added total number of received frames
6ecf3b6
@StefanoSperetta
Restructured grafana dashboards
8085a01
@StefanoSperetta
Addind microseconds to the frames tables
e0a0fbe
@StefanoSperetta
Ignore fields marked as to be ignored
0ec7f81
@StefanoSperetta
Fix pylint warnings
c9d53ef
@StefanoSperetta
Remove debug log call
e991da7
@StefanoSperetta
Merge pull request #165 from iiacoban42/main
80cc534 
Pull from main
@StefanoSperetta
Update crowdsec and pgadmin
3e5e2ce
@StefanoSperetta
Set the user and application as tags in the raw frames table
2efb7f5
@StefanoSperetta
Merge branch 'main' into update_processing
ee6dda2
@StefanoSperetta
Fixex copy-paste error in documentation
56feef1
@StefanoSperetta
Fixed copy-paste error in documentation
aa7f624
@StefanoSperetta
Added missing step to local setup guide
3da3c9e
@StefanoSperetta
Added missing step to local setup guide
f6c9fca
@StefanoSperetta
Merge branch 'main' into update_processing
c16e23c
@StefanoSperetta
Do not create a Delfispace bucket anymore
8570ad9
@StefanoSperetta
Update crowdsec to latest version
5388f9f
@StefanoSperetta
Merge branch 'main' into update_processing
1ed9203
@StefanoSperetta
Merge branch 'main' into update_processing
4a05e1b
@StefanoSperetta
Fixing unit test username / usi swap
fc88604
@StefanoSperetta
Fixing unit test username / usi swap
07114a2
@StefanoSperetta
Merge branch 'main' into update_processing
781b56d
@StefanoSperetta
Merge branch 'main' into update_processing
d7dbd13
@StefanoSperetta
Merge branch 'main' into update_processing
4c89249
@StefanoSperetta
Pushing old commits to branch. To be cleaned up.
6781910
@StefanoSperetta
Merge branch 'main' into update_processing
d8fe201
@StefanoSperetta
StefanoSperetta marked this pull request as ready for review April 30, 2026 12:08
@StefanoSperetta
StefanoSperetta merged commit c80b27e into main Apr 30, 2026
0 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@StefanoSperetta