Skip to content

feat(xdp): Enhance XDP backend with improved concurrency and performance metrics #1620

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 3 commits into from
Closed

feat(xdp): Enhance XDP backend with improved concurrency and performance metrics #1620

wants to merge 3 commits into from

Conversation

mvp-2003
Copy link
Collaborator

  • Added new configuration parameters for TX and RX fill and completion ring sizes.
  • Introduced enhanced concurrency options for XDP operations, including batch sizes and adaptive batching.
  • Implemented a performance testing example to demonstrate the benefits of the new XDP backend.
  • Created a batch processing utility for efficient transmission of packets.
  • Developed functional and unit tests to validate backward compatibility and new features.
  • Updated configuration parsing to support new parameters and ensure proper validation.

@mvp-2003
feat(xdp): Enhance XDP backend with improved concurrency and performa…
cd35b46 
…nce metrics

- Added new configuration parameters for TX and RX fill and completion ring sizes.
- Introduced enhanced concurrency options for XDP operations, including batch sizes and adaptive batching.
- Implemented a performance testing example to demonstrate the benefits of the new XDP backend.
- Created a batch processing utility for efficient transmission of packets.
- Developed functional and unit tests to validate backward compatibility and new features.
- Updated configuration parsing to support new parameters and ensure proper validation.
@mvp-2003 mvp-2003 mentioned this pull request Jul 22, 2025
Open
@anandbonde
Copy link
Contributor

@mvp-2003: Currently, we don't support running CI on branches outside of our repo. And PRs cannot be pulled unless the CI passes. The only way to work around this is to create a new branch in our repo and submit a PR that way.

Also, could you please write a small reasoning behind why you think this should be present in the mainstream codebase?

@anandbonde anandbonde self-requested a review July 27, 2025 15:48
anandbonde
anandbonde requested changes Jul 27, 2025
View reviewed changes
Copy link
Contributor

@anandbonde anandbonde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a general comment on this PR. Please take a look.

@mvp-2003
Copy link
Collaborator Author

mvp-2003 commented Jul 27, 2025 •
edited
Loading

@anandbonde Hello sir. Just to be sure, I understood the instructions clearly, you wanted me to create a new branch on the origianal repo and raise the PR from my forked repo to the new branch of the original repo and also give a proper documentation and state the reason for coming up with this approach, right?

@anandbonde
Copy link
Contributor

@anandbonde Hello sir. Just to be sure, I understood the instructions clearly, you wanted me to create a new branch on the origianal repo and raise the PR from my forked repo to the new branch of the original repo and also give a proper documentation and state the reason for coming up with this approach, right?

@mvp-2003: I've added you as a contributor to the repo, now you can create a new branch on the original repo and submit PR(s) from there. This will allow the CI actions to run on that branch — it's a mandatory check without which PRs cannot be checked in.

Re. doc: It would be good to document what drove this change... are you addressing/fixing an existing issue or are you trying to solve a "local" problem that you are facing and want to contribute those changes back to the community?

@mvp-2003
Copy link
Collaborator Author

mvp-2003 commented Jul 28, 2025 •
edited
Loading

@anandbonde Hello sir. While trying to create a new branch from the "dev" branch, I am getting this error: "Create protected branch 'xdp-performance-improvement' check failed: You're not authorized to push to this branch. Visit https://docs.github.com/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches for more information."
Could you please recheck if I was provided with the necessary permissions? Thank you.

@mvp-2003
Copy link
Collaborator Author

Hello @anandbonde
Did you get a chance to look into this matter, sir?

@mvp-2003 mvp-2003 requested a review from anandbonde July 30, 2025 09:39
@anandbonde
Copy link
Contributor

anandbonde commented Jul 31, 2025 •
edited
Loading

@anandbonde Hello sir. While trying to create a new branch from the "dev" branch, I am getting this error: "Create protected branch 'xdp-performance-improvement' check failed: You're not authorized to push to this branch. Visit https://docs.github.com/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches for more information." Could you please recheck if I was provided with the necessary permissions? Thank you.

I’m seeing that you already have permissions. Can you try ssh auth with the GitHub repo if https isn’t working. Let me know if it doesn’t work for you.

@mvp-2003
Copy link
Collaborator Author

mvp-2003 commented Aug 4, 2025

Hello @anandbonde
I tried as you asked. I cloned the repo through SSH and created a new branch and as clicked on push to publish the branch, i get this error:
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: error: GH006: Protected branch update failed for refs/heads/con-xdp-1437.
remote:
remote: - You're not authorized to push to this branch. Visit https://docs.github.com/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches for more information.
To github.com:microsoft/demikernel.git
! [remote rejected] con-xdp-1437 -> con-xdp-1437 (protected branch hook declined)
error: failed to push some refs to 'github.com:microsoft/demikernel.git'

I don't think I am allowed to create branches. It may also be possible that there are some rulesets configures depending on roles. Can you check once again. Thank you.

@anandbonde
Copy link
Contributor

Hello @anandbonde I tried as you asked. I cloned the repo through SSH and created a new branch and as clicked on push to publish the branch, i get this error: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0) remote: error: GH006: Protected branch update failed for refs/heads/con-xdp-1437. remote: remote: - You're not authorized to push to this branch. Visit https://docs.github.com/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches for more information. To github.com:microsoft/demikernel.git ! [remote rejected] con-xdp-1437 -> con-xdp-1437 (protected branch hook declined) error: failed to push some refs to 'github.com:microsoft/demikernel.git'

I don't think I am allowed to create branches. It may also be possible that there are some rulesets configures depending on roles. Can you check once again. Thank you.

@mvp-2003, sorry for the delay, can you try now? I've explicitly added you in the list of contributors as well now and that should allow creating for new branches.

@mvp-2003
Copy link
Collaborator Author

mvp-2003 commented Aug 4, 2025

I'm sorry @anandbonde it looks like I am still blocked. How possible would it be for you to create a branch for me and just tell the branch name here so that I can raise the PR. I think that would be a simpler approach with a guaranteed success.

@mvp-2003
Implement Dynamic Ring Management and Sizing
212f24f 
- Added `DynamicRingManager` to manage dynamic resizing of RX/TX rings based on workload and system metrics.
- Introduced `DynamicSizingConfig` and `DynamicRingSizer` for configuring and calculating optimal ring sizes.
- Implemented metrics collection through `MetricsCollector` to gather system and workload metrics.
- Created `RingResizeCallback` trait for handling resize events with a default logging implementation.
- Added functionality to evaluate and apply sizing recommendations based on collected metrics.
- Enhanced error handling and validation for configuration parameters.
- Established a background task for periodic evaluation of ring sizes.
- Included detailed logging for resizing operations and metrics collection.
@mvp-2003
Copy link
Collaborator Author

mvp-2003 commented Aug 6, 2025

@anandbonde sir? Any updates?

@anandbonde
Copy link
Contributor

anandbonde commented Aug 6, 2025 •
edited
Loading

I'm sorry @anandbonde it looks like I am still blocked. How possible would it be for you to create a branch for me and just tell the branch name here so that I can raise the PR. I think that would be a simpler approach with a guaranteed success.

@mvp-2003 That will not work because you are not authorized to "push". I'll take a look.

[update]: I've now re-enabled the permissions for your account. I'm confident that this should work. Please retry and let me know if it doesn't work.

@mvp-2003 mvp-2003 closed this Aug 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@iyzhang iyzhang Awaiting requested review from iyzhang

@kyleholohan kyleholohan Awaiting requested review from kyleholohan

@anandbonde anandbonde Awaiting requested review from anandbonde

Assignees

@mvp-2003 mvp-2003

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mvp-2003 @anandbonde