DataProduct Catalog Runtime Insertion #4083

Willmac16 · 2025-08-27T22:17:58Z


*Related Issue(s)*	N/A
*Has Unit Tests (y/n)*	y
*Documentation Included (y/n)*	y
*Generative AI was used in this contribution (y/n)*	n

Change Description

Modifies DpCatalog to allow for runtime insertion of data products onto the queue through the DpWriter notifier port

Rationale

Long running missions with the need for automated file downlink would benefit from simultaneous catalog transmission and construction.

Testing/Review Recommendations

Progress towards Draft Completion

Refactor file addition into separate function
Call into processFile from addToCatalog port
Allow adding to catalog while Transmiting
Add UT coverage for addToCatalog port
Fix running the 2 recently enabled UTs back to back
Fix/Remove that final DISABLED unit test
Round out coverage

Nice to Haves

Clean all artifacts run to run
Catalog Navigation w/o the stack
- would make auto-balancing easier
- reduces memory req
- probably doable w/ a current node + traversal direction (up/down) since I've added the parent link
Self balancing (prob AVL or Red-Black)
- Seems prudent given how carved up the tree will be with simultaneous additions and removals
Random Sequence order validation @ the addToCat -> fileOut level

AI Usage (see policy)

N/A

Svc/DpCatalog/DpCatalog.cpp

timcanham · 2025-09-01T03:45:29Z

Svc/DpCatalog/DpCatalog.cpp

+        this->log_WARNING_HI_NoDpMemory();
+        return;
+    }
+


We should add a check to see if the catalog has already been loaded. If it hasn't, just drop this call, since the new file will be noticed when the catalog is loaded.

@Willmac16 Where is this checked an dropped?

I believe I added a check at one point for the dpTree pointer being non-null; however, I later removed that check since nodes could be added after tree completion (and pruning) with the remainActive arg. I'm adding a state file initialization check, since that is the primary thing a runtime added file needs from the catalog build.

I am now using a catalogBuilt flag since you can't guarantee the state file that is loaded will have any items.

timcanham · 2025-09-01T03:52:48Z

Looks good so far!

Willmac16 · 2025-09-10T17:04:24Z

Do you have a recommendation for a cmake way to clean up the test artifacts consistently regardless of if the test passes?

…llocated

…d when TRANSMITTED is set * TRANSMITTED will be filtered by process file

Fw/Types/StringUtils.cpp

+    for (FwSizeType ii = 0; ii <= (source_size - sub_size); ii++) {
+        const FwSizeType source_index = (source_size - sub_size) - ii;
+
+        // if the current character matches
+        for (FwSizeType sub_index = 0; sub_index < sub_size; sub_index++) {
+            // Prevent read overrun
+            FW_ASSERT((source_index + sub_index) < source_size);
+            // if there is a mismatch, go to next character
+            if (source_string[source_index + sub_index] != sub_string[sub_index]) {
+                break;
+            } else if (sub_index == (sub_size - 1)) {
+                // if we matched all the way to the end of the substring
+                match_index = source_index;
+
+                // Ensure the result converts properly
+                FW_ASSERT(static_cast<FwSizeType>(static_cast<FwSignedSizeType>(match_index)) == match_index);
+
+                return static_cast<FwSignedSizeType>(match_index);
+            }
+        }
+    }


Svc/DpCatalog/DpCatalog.cpp

+    // Grab the directory string (up until the final slash)
+    // Could be found directly w/ a dirname func or regex
+    FwSignedSizeType loc = Fw::StringUtils::substring_find_last(
+        fullFile.toChar(), fullFile.length(), DIRECTORY_DELIMITER,


Svc/DpCatalog/DpCatalog.cpp

+    Fw::Buffer hdrBuff(dpBuff, sizeof(dpBuff));   // buffer for container header decoding
+    Fw::DpContainer container;                    // container object for extracting header fields
+
+    this->log_ACTIVITY_LO_ProcessingFile(fullFile);


Svc/DpCatalog/DpCatalog.cpp

+
+    // add entry to catalog.
+    DpStateEntry entry;
+    entry.dir = static_cast<FwIndexType>(dir);


Svc/DpCatalog/DpCatalog.cpp

+    }
+
+    Fw::FileNameString addedFileName;
+    addedFileName.format(DP_FILENAME_FORMAT, this->m_directories[dir].toChar(), entry.record.get_id(),


Svc/DpCatalog/DpCatalog.cpp

+
+    // Check the catalog has been built
+    if (not this->m_catalogBuilt) {
+        this->log_ACTIVITY_HI_NotLoaded(fileName);


Svc/DpCatalog/DpCatalog.cpp

+    }
+
+    // Both of these are grabbed from the header
+    (void)priority;


Svc/DpCatalog/DpCatalog.cpp

+
+    // Both of these are grabbed from the header
+    (void)priority;
+    (void)size;


Svc/DpCatalog/DpCatalog.cpp

+
+    // Since this is a runtime addition
+    // Check if file is in one of our directories
+    FwSizeType dir = this->determineDirectory(fileName);


Svc/DpCatalog/DpCatalog.cpp

-void DpCatalog ::START_XMIT_CATALOG_cmdHandler(FwOpcodeType opCode, U32 cmdSeq, Fw::Wait wait) {
+void DpCatalog ::START_XMIT_CATALOG_cmdHandler(FwOpcodeType opCode, U32 cmdSeq, Fw::Wait wait, bool remainActive) {
    Fw::CmdResponse resp = this->doCatalogXmit();
+    this->m_remainActive = remainActive;


Fw/Types/StringUtils.cpp

    // if we make it here, no matches were found
    return -1;
 }
+


Fw/Types/StringUtils.cpp

    return -1;
 }
+
+FwSignedSizeType Fw::StringUtils::substring_find_last(const CHAR* source_string,


Svc/DpCatalog/DpCatalog.cpp

 }

-void DpCatalog ::START_XMIT_CATALOG_cmdHandler(FwOpcodeType opCode, U32 cmdSeq, Fw::Wait wait) {
+void DpCatalog ::START_XMIT_CATALOG_cmdHandler(FwOpcodeType opCode, U32 cmdSeq, Fw::Wait wait, bool remainActive) {


Svc/DpCatalog/DpCatalog.cpp

+      m_xmitCmdSeq(0),
+      m_pendingFiles(0),
+      m_pendingDpBytes(0),
+      m_remainActive(false) {}


Svc/DpCatalog/DpCatalog.cpp

 }  // end fillBinaryTree()

-bool DpCatalog::insertEntry(DpStateEntry& entry) {
+FwSizeType DpCatalog::determineDirectory(Fw::String fullFile) {


Svc/DpCatalog/DpCatalog.cpp

    this->pingOut_out(0, key);
 }

+void DpCatalog ::addToCat_handler(FwIndexType portNum,


Svc/DpCatalog/DpCatalog.cpp

+    return DP_MAX_DIRECTORIES;
+}
+
+int DpCatalog::processFile(Fw::String fullFile, FwSizeType dir) {


Svc/DpCatalog/DpCatalog.cpp

 }

-void DpCatalog::deleteEntry(DpStateEntry& entry) {}
+void DpCatalog::deallocateNode(DpBtreeNode* node) {


Svc/DpCatalog/DpCatalog.cpp

 }

-void DpCatalog::deleteEntry(DpStateEntry& entry) {}
+void DpCatalog::deallocateNode(DpBtreeNode* node) {


Svc/DpCatalog/DpCatalog.cpp

+    return compareEntries(*this, other) < 0;
+}
+
+DpCatalog::DpBtreeNode* DpCatalog::insertEntry(DpStateEntry& entry) {


Svc/DpCatalog/DpCatalog.cpp

+// ----------------------------------------------------------------------
+// DpStateEntry Comparison Ops
+// ----------------------------------------------------------------------
+int DpCatalog::DpStateEntry::compareEntries(const DpStateEntry& left, const DpStateEntry& right) {


Svc/DpCatalog/DpCatalog.cpp

+    }
+
+    // ret > 0 := success
+    int ret = processFile(fileName, dir);


Fw/Types/StringUtils.cpp

    return -1;
 }
+
+FwSignedSizeType Fw::StringUtils::substring_find_last(const CHAR* source_string,


Fw/Types/StringUtils.cpp

+
+FwSignedSizeType Fw::StringUtils::substring_find_last(const CHAR* source_string,
+                                                      FwSizeType source_size,
+                                                      const CHAR* sub_string,


Svc/DpCatalog/DpCatalog.cpp

-            // make sure we haven't exceeded the limit
-            if (this->m_numDpRecords > this->m_numDpSlots) {
-                this->log_WARNING_HI_DpCatalogFull(entry.record);
+            int ret = processFile(fullFile, dir);


Svc/DpCatalog/DpCatalog.cpp

+    return DP_MAX_DIRECTORIES;
+}
+
+int DpCatalog::processFile(Fw::String fullFile, FwSizeType dir) {


Svc/DpCatalog/DpCatalog.cpp

+// ----------------------------------------------------------------------
+// DpStateEntry Comparison Ops
+// ----------------------------------------------------------------------
+int DpCatalog::DpStateEntry::compareEntries(const DpStateEntry& left, const DpStateEntry& right) {


Svc/DpCatalog/DpCatalog.cpp

 }  // end fillBinaryTree()

-bool DpCatalog::insertEntry(DpStateEntry& entry) {
+FwSizeType DpCatalog::determineDirectory(Fw::String fullFile) {


Svc/DpCatalog/DpCatalog.cpp

+    return DP_MAX_DIRECTORIES;
+}
+
+int DpCatalog::processFile(Fw::String fullFile, FwSizeType dir) {


… empty catalogs

Willmac16 · 2025-11-16T00:39:22Z

I am finally ready for re-review!

Willmac16 force-pushed the Willmac16/dp-cat-runtime-insertion branch 2 times, most recently from 189a5a8 to 90958a1 Compare August 27, 2025 22:27

Willmac16 commented Aug 27, 2025

View reviewed changes

Svc/DpCatalog/DpCatalog.cpp Show resolved Hide resolved

timcanham requested changes Sep 1, 2025

View reviewed changes

Willmac16 force-pushed the Willmac16/dp-cat-runtime-insertion branch 2 times, most recently from d15405c to b21bc30 Compare September 9, 2025 19:48

Willmac16 added 22 commits September 10, 2025 18:40

DPCat Docs fix transposed words

1ab161f

Migrate file processing into processFile & Fix/ReEnable most DpCat UTs

5df1d5e

substring_find_last & addToCat handler

1471acf

New Warnings & Style Tweaks

c67494a

Migrate DpCat to new register_fprime_* format

f5af805

Prevent UT State Files from Persisting

8970ee3

Ignore /Testing & *.swp

7a1904b

Test Runtime Addition & Add remainActive parameter

f79d5b0

Only Reset DpCat Traversal Stack if addition is higher priority

e7e17d4

Leave binary tree intact upon insertion error

23e5a41

Implement, but don't yet use deallocateNode

2b84c5f

Deallocate and shift nodes after transmit

f46a0d9

WIP: Stitching behavior to ensure every finished file's node gets dea…

37d44e9

…llocated

WIP: No more assert/UB, but UTs stall

b5ab14f

Log pending counters @ file complete & Fix UTs w/ deallocate logic

11b6579

Make deallocateNode more assertive and ensure things are set

526ccac

Use the same CompareEntries function in code and UT

e5f99c9

Fix CompareEntries & overload comparison operators

32fc47b

partial_sort_copy all output lists & ignore issue on RandomTransmitte…

5967d32

…d when TRANSMITTED is set * TRANSMITTED will be filtered by process file

Use the DpStateEntry comparison when inserting nodes

ed1514d

Transmited -> Transmitted

fac3b8c

Randomized Testing of Data Product addition & transmission

a253af2

Willmac16 added 5 commits November 2, 2025 16:27

DpCatalog docs tweaks

28fab53

Spelling

270477b

Prevent Duplicate Cmd Responses

2237265

Remove numDpRecords & improve comments

a5afc42

Check State file has been opened

fb7599f

Willmac16 force-pushed the Willmac16/dp-cat-runtime-insertion branch from 7e89f01 to 01fd6b1 Compare November 2, 2025 23:54

Use a catalogBuilt flag to prevent runtime insertion until ready

9fd9624

Willmac16 force-pushed the Willmac16/dp-cat-runtime-insertion branch from 01fd6b1 to 9fd9624 Compare November 2, 2025 23:57

Willmac16 added 6 commits November 2, 2025 18:16

Tweaking readDps add and done sequencing

e740f54

Test Catalog Stop, Clear, & Faulty Restart

67e520a

Fixes for deallocateNode w/ left hand branch

d86a108

Test XmitStop warning

7d7fd18

Test StartXmit before config

0064434

Delaying dispatch seems to break random test file counts

cad827f

github-advanced-security bot found potential problems Nov 3, 2025

View reviewed changes

Fw/Types/StringUtils.cpp

// if we make it here, no matches were found

return -1;

}

Check warning

Code scanning / CppCheck

Could not find a newline character at the end of the file. Warning

Could not find a newline character at the end of the file.

github-advanced-security bot found potential problems Nov 3, 2025

View reviewed changes

thomas-bc added the Deferred To Next Release This PR is marked as deferred until after the upcoming release. label Nov 12, 2025

Willmac16 added 9 commits November 15, 2025 15:49

Print the DpCatalog Tree

f38385e

One Hundred Random Dp Testing Iters & Cleanup Random Dps btw runs

cc060f0

Remove Tree Printing

e4b494f

Add rm -rf to tester construction

81c7862

Ensure all test artifacts are in DpTest*

fb416cb

Low Hanging Coverage & Delete Print Functions

c99d949

Test FileDone failure EVR & delayed cmd response and allow built time…

9a50880

… empty catalogs

Consolidate & Test the WAIT response mode

17c7e4e

add unconfigured to spelling

1a097ad

DataProduct Catalog Runtime Insertion #4083

Are you sure you want to change the base?

DataProduct Catalog Runtime Insertion #4083

Uh oh!

Conversation

Willmac16 commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Description

Rationale

Testing/Review Recommendations

Progress towards Draft Completion

Nice to Haves

AI Usage (see policy)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

timcanham Sep 1, 2025

Choose a reason for hiding this comment

Uh oh!

timcanham Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

Willmac16 Nov 2, 2025

Choose a reason for hiding this comment

Uh oh!

Willmac16 Nov 2, 2025

Choose a reason for hiding this comment

Uh oh!

timcanham commented Sep 1, 2025

Uh oh!

Willmac16 commented Sep 10, 2025

Uh oh!

Check warning

Check warning

Check warning

Check warning

Check warning

Uh oh!

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check notice

Check notice

Check notice

Check notice

Check notice

Check notice

Check notice

Check notice

Check notice

Check notice

Check notice

Check notice

Check notice

Check notice

Check notice

Check notice

Check warning

Check warning

Willmac16 commented Nov 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Willmac16 commented Aug 27, 2025 •

edited

Loading