Codice continuation packets #2569

greglucas · 2026-01-09T15:20:35Z

Change Summary

Overview

This changes the CoDICE packet definition for direct events. Previously, we were reading in many fields for each packet, but this is incorrect, the fields are only in the first packet. This meant that we were extracting things with XTCE, then recombining those fields back into binary and then extracting again. The issue came about when the second packet's binary payload was not long enough to unpack all the individual fields.

This is a significant refactor as well. Rather than working with binary strings, we are working with bytes directly and doing bitshifts on numpy arrays. Thanks to good unit tests, this was able to be refactored with the help of AI.

closes #2568

tech3371 · 2026-01-15T08:16:44Z

imap_processing/codice/codice_l1a_de.py

+    All packets have (SHCOARSE, EVENT_DATA, CHKSUM) fields. To combine
+    the segmented packets, we only concatenate along the EVENT_DATA field
+    into the first packet of the group.


This is for my future self and may be others. I don't want to lose context of these changes if someone looks at this later since those details are in our email. Can you break line where appropriate and make it more clear as needed?

Suggested change

All packets have (SHCOARSE, EVENT_DATA, CHKSUM) fields. To combine

the segmented packets, we only concatenate along the EVENT_DATA field

into the first packet of the group.

Direct event data are segmented into packets. CoDICE assembles these segmented packets slightly differently than the standard. Onboard, CoDICE direct event data are segmented as follows:

Standalone / unsegmented packets:

Data is packed in the order defined in the telemetry definition in Excel. eg.

(SHCOARSE, many metadata fields, EVENT_DATA, CHECKSUM).

First segment:

Data is packed in the order defined in the telemetry definition in Excel. Eg.

(SHCOARSE, many metadata fields, EVENT_DATA, CHECKSUM).

As we see here, the first segment packet contains metadata defined in the telemetry definition. Those metadata are unpacked in later functions/steps because those metadata will always be fixed bit lengths and should exist.

Middle segment:

Data is packed in the following order: SHCOARSE, EVENT_DATA, CHECKSUM.

There can be multiple middle packets for a given packet group.

Last segment:

Data is packed in the following order: SHCOARSE, EVENT_DATA, CHECKSUM.

This last segment of event data can contain padded values.

Because of this behavior, we defined the XTCE to unpack the data using the field order

(SHCOARSE, EVENT_DATA, CHKSUM). This simplifies XTCE-based unpacking and allows

the remaining packet-specific details to be handled in code. In this function,

the segmented event data are combined by concatenating the EVENT_DATA field across

the packet group.

I think this is too verbose. I made this a generic function in the ultra continuation packet PR so that multiple instruments can use it. Over there, it isn't the same packet definition as here, so this wouldn't make sense to add to the generic function.

can we add it somewhere in this codice_l1a_de.py? I don't mind where we put this in that file but would be nice to capture this for future reference.

I added some of this to the codice_l1a_de file now, explaining what fields we have in XTCE and why we are doing it this way now.

imap_processing/codice/codice_l1a_de.py

tech3371 · 2026-01-16T10:46:47Z

Current code looks good to me. I think there are few changes coming from yesterday discussion. let me know when you want me to review this again.

We have had some issues with missing source sequence counters, meaning not all necessary packets were present. This just adds some simple logs that can warn about this case and give some awareness about what is going on.

The acquisition start times dictate what priorities are in a group. i.e. all priorities have the same start time. Since we know how many priorities there are per Hi/Lo configuration, we know that we should have that many exact items with a given acquisition start time. This allows us to filter out items that weren't complete.

greglucas · 2026-01-18T20:09:26Z

OK, I think the last few commits contain the logic that we need for handling missing priority groups and dropping those. I still need to look into whether that is OK, or if we want to fill in some priority values and use FILL for the rest, or drop the entire sequence when that happens. But, this should be ready for review now and I did a cursory check and looks like it is producing valid events for the December test cases now.

lacoak21

OK ive only review part of this! Im going to do the rest first thing tomorrow.

lacoak21 · 2026-01-20T19:51:20Z

imap_processing/codice/codice_l1a_de.py

    ----------
-    packets : xarray.Dataset
-        Dataset containing the packets to group.
+    packets : xr.Dataset


Wont this cause documenation build errors?

xarray.Dataset -> xr.Dataset

🤔 This passed the documentation build... I think you're right though, so I went through and updated this here and elsewhere throughout this file. It looks like there was previously a mix of usage.

tech3371

This looks great to me. Thank you for taking this on! Sorry for delay review. I didn't see the comment in my filtered email.

imap_processing/utils.py

tech3371 · 2026-01-21T11:34:10Z

imap_processing/utils.py

+    """
+    # Check for sequential source sequence counters
+    # CCSDS source sequence counter is a 14-bit field (0-16383)
+    counter_max = 16384


should we make this into a constant above? Otherwise, it looks good

I don't think this is a critical constant and isn't used in other places, so I just kept it outside for now. We can re-evaluate later and move it out if needed/desired.

imap_processing/utils.py

imap_processing/codice/codice_l1a_de.py

lacoak21 · 2026-01-21T15:31:34Z

imap_processing/codice/codice_l1a_de.py

+        pkt_bytes = pkt_bytes.reshape(n_events, 8)[:, ::-1]
+        all_event_bytes[offset : offset + n_events] = pkt_bytes
+
+        # Record destination indices for scattering


what do you mean by scattering here and on line 331?

Great question! My AI friend wrote that comment. But, what it is getting at is that we are "scattering" the data from one layout into another array-based layout in a sense. So an item in array 1 at location (1, 2, 3) will get "scattered" into array2 at location (2, 5, 7) and there is a relationship we are storing for how that mapping is working.

I don't think that was clear, so I tried to reword both locations. It is basically doing array-based index lookups and assignments.

Interesting! Got it. Ive only ever "scattered" data in reference to uncertainty and not something that we can "fix".

FYI: https://en.wikipedia.org/wiki/Gather/scatter_(vector_addressing)

greglucas added this to the January 2026 milestone Jan 9, 2026

greglucas requested review from lacoak21 and tech3371 January 9, 2026 15:20

greglucas self-assigned this Jan 9, 2026

greglucas added Ins: CoDICE Related to the CoDICE instrument Level: L1 Level 1 processing labels Jan 9, 2026

tech3371 reviewed Jan 15, 2026

View reviewed changes

greglucas added 3 commits January 16, 2026 10:39

Handling codice segmented packets

33e73ef

MNT: Add logging for source sequence counters to packet decom

c765c7e

We have had some issues with missing source sequence counters, meaning not all necessary packets were present. This just adds some simple logs that can warn about this case and give some awareness about what is going on.

greglucas force-pushed the codice-continuation-packets branch from d065c55 to d9ca856 Compare January 18, 2026 20:06

lacoak21 reviewed Jan 20, 2026

View reviewed changes

tech3371 approved these changes Jan 21, 2026

View reviewed changes

lacoak21 reviewed Jan 21, 2026

View reviewed changes

DOC: Update documentation on CoDICE segmented packet handling

deada8e

greglucas force-pushed the codice-continuation-packets branch from d9ca856 to deada8e Compare January 21, 2026 18:48

lacoak21 self-requested a review January 21, 2026 18:55

lacoak21 approved these changes Jan 21, 2026

View reviewed changes

greglucas merged commit 0de5674 into IMAP-Science-Operations-Center:dev Jan 21, 2026
14 checks passed

greglucas deleted the codice-continuation-packets branch January 21, 2026 19:12

-    All packets have (SHCOARSE, EVENT_DATA, CHKSUM) fields. To combine
-    the segmented packets, we only concatenate along the EVENT_DATA field
-    into the first packet of the group.
+Direct event data are segmented into packets. CoDICE assembles these segmented packets slightly differently than the standard. Onboard, CoDICE direct event data are segmented as follows:
+Standalone / unsegmented packets:
+Data is packed in the order defined in the telemetry definition in Excel. eg.
+(SHCOARSE, many metadata fields, EVENT_DATA, CHECKSUM).
+First segment:
+Data is packed in the order defined in the telemetry definition in Excel. Eg.
+(SHCOARSE, many metadata fields, EVENT_DATA, CHECKSUM).
+As we see here, the first segment packet contains metadata defined in the telemetry definition. Those metadata are unpacked in later functions/steps because those metadata will always be fixed bit lengths and should exist.
+Middle segment:
+Data is packed in the following order: SHCOARSE, EVENT_DATA, CHECKSUM.
+There can be multiple middle packets for a given packet group.
+Last segment:
+Data is packed in the following order: SHCOARSE, EVENT_DATA, CHECKSUM.
+This last segment of event data can contain padded values.
+Because of this behavior, we defined the XTCE to unpack the data using the field order
+(SHCOARSE, EVENT_DATA, CHKSUM). This simplifies XTCE-based unpacking and allows
+the remaining packet-specific details to be handled in code. In this function,
+the segmented event data are combined by concatenating the EVENT_DATA field across
+the packet group.

Codice continuation packets #2569

Codice continuation packets #2569

Uh oh!

Conversation

greglucas commented Jan 9, 2026

Change Summary

Overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tech3371 Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tech3371 commented Jan 16, 2026

Uh oh!

greglucas commented Jan 18, 2026

Uh oh!

lacoak21 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tech3371 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tech3371 Jan 16, 2026 •

edited

Loading