comprehensive Unix-like shell features for console system (IDFGH-16704)

[jimmyw]

Description

This pull request introduces a comprehensive Unix-like shell component to ESP-IDF's console system, along with a VFS pipe implementation.

Background: at my company we have built a home battery energy storage system. It have been growing a lot, and we use the esp_console features extensively, even remotely over mqtt, in a local webserver interface, and for CI testing. Some commands generate a lot of data, so there was a long time dream of me, to be able to use tools like grep or less to be more efficient.

Major Features Added

1. Thread-Safe VFS Pipe Implementation (vfs_pipe.c)
   This is a self containing feature for POSIX Compatibility: Full pipe() syscall implementation with
   standard semantics
   Advanced I/O Support: Blocking/non-blocking operations with select() integration
   Thread Safety: FreeRTOS-based synchronization preventing race conditions
   Resource Management: Proper cleanup and error handling for production use

2. Enhanced Console API (esp_console.h)
   Task-Based Execution: New esp_console_run_on_task() for asynchronous command execution
   Process Management: esp_console_wait_task() and esp_console_task_free() for task lifecycle
   Status Monitoring: esp_console_task_is_running() for non-blocking status checks
   Command Access: esp_console_get_command_list() for programmatic command introspection
   This optional feature can enable to use smaller stack for the console task, and adjustable stack
   sizes for commands that run in their own stack only when active.

3. Shell Component (shell.c)
   Pipeline Support: Full implementation of command pipelines using | operator
   Command Chaining: Support for ; (continue), && (break on fail), || (break on success) operators
   File Redirection: Output redirection with > (write) and >> (append) operators
   Interactive Features: Ctrl+D detection for EOF handling, proper stdin/stdout/stderr management
   Async Execution: Task-based command execution with proper lifecycle management.
   This is a work in progress as a lot of more features can be added, like run tasks in the background.

4. Updated advanced console example (console_example_main.c)
   Now supports using the new shell extension leveraging support for shell component as well as adding commands like cat, echo, true, false, tee and grep

Testing

There is no tests implementation written yet, as i'm not sure of any interest in upstreaming these changes. If there seems to be interest, i will definitely add that before merge.

Examle

esp32s3> help  | grep nvs_set
nvs_set  <key> <type> -v <value>
  nvs_set VarName i32 -v 123 
  nvs_set VarName str -v YourString 
  nvs_set VarName blob -v 0123456789abcdef 

esp32s3> true && echo "Success" || echo "Failed"
Success

esp32s3> help > /data/commands.txt

esp32s3> cat /data/commands.txt
   all the output...

esp32s3> cat /data/commands.txt | grep nvs_set
nvs_set  <key> <type> -v <value>
  nvs_set VarName i32 -v 123 
  nvs_set VarName str -v YourString 
  nvs_set VarName blob -v 0123456789abcdef 
esp32s3>

esp32s3> grep nvs_namespace < /data/commands.txt
nvs_namespace  <namespace>

esp32s3> grep jimmy
Hello my name is jimmy
Ctrl+D detected, sending EOF to command input
esp32s3>

---

Checklist

Before submitting a Pull Request, please ensure the following:

• 🚨 This PR does not introduce breaking changes.
• All CI checks (GH Actions) pass.
• Documentation is updated as needed.
• Tests are updated or added as necessary.
• Code is well-commented, especially in complex areas.
• Git history is clean — commits are squashed to the minimum necessary.

[CLAassistant]

All committers have signed the CLA.

[github-actions]

	Warnings
⚠️	Please consider squashing your 6 commits (simplifying branch history).

	Messages
📖	This PR seems to be quite large (total lines of code: 2006), you might consider splitting it into smaller PRs

👋 Hello jimmyw, we appreciate your contribution to this project!

---

📘 Please review the project's Contributions Guide for key guidelines on code, documentation, testing, and more.

---

🖊️ Please also make sure you have read and signed the Contributor License Agreement for this project.

---

Click to see more instructions ...

This automated output is generated by the PR linter DangerJS, which checks if your Pull Request meets the project's requirements and helps you fix potential issues.

DangerJS is triggered with each push event to a Pull Request and modify the contents of this comment.

Please consider the following:
- Danger mainly focuses on the PR structure and formatting and can't understand the meaning behind your code or changes.
- Danger is not a substitute for human code reviews; it's still important to request a code review from your colleagues.
- Resolve all warnings (⚠️ ) before requesting a review from human reviewers - they will appreciate it.
- Addressing info messages (📖) is strongly recommended; they're less critical but valuable.
- To manually retry these Danger checks, please navigate to the Actions tab and re-run last Danger workflow.

Review and merge process you can expect ...

We do welcome contributions in the form of bug reports, feature requests and pull requests via this public GitHub repository.

This GitHub project is public mirror of our internal git repository

1. An internal issue has been created for the PR, we assign it to the relevant engineer.
2. They review the PR and either approve it or ask you for changes or clarifications.
3. Once the GitHub PR is approved, we synchronize it into our internal git repository.
4. In the internal git repository we do the final review, collect approvals from core owners and make sure all the automated tests are passing.
- At this point we may do some adjustments to the proposed change, or extend it by adding tests or documentation.
5. If the change is approved and passes the tests it is merged into the default branch.
5. On next sync from the internal git repository merged change will appear in this public GitHub repository.

Generated by 🚫 dangerJS against f4857cc

[SoucheSouche]

Hello @jimmyw,

Thanks a lot for this contribution! At first glance, it looks like a really nice addition to the console component.
The PR you submitted is quite big and it will take time to review as is.

The first thing I would suggest is to break it into several smaller PRs to facilitate the merging process.
For instance, I think the pipe addition to the vfs component could represent a PR on its own.

Secondly, it seems to me that you are creating this PR against an older version of esp-idf. I went through the changes briefly and noticed in vfs_pipe.c:

esp_err_t esp_vfs_pipe_register(const esp_vfs_pipe_config_t *config)
{
    esp_vfs_t vfs = {
        .flags        = ESP_VFS_FLAG_DEFAULT,
        .write        = &pipe_write,
        .close        = &pipe_close,
        .read         = &pipe_read,
        .fcntl        = &pipe_fcntl,
#ifdef CONFIG_VFS_SUPPORT_SELECT
        .start_select = &pipe_start_select,
        .end_select   = &pipe_end_select,
#endif
    };
    return esp_vfs_register_with_id(&vfs, NULL, &s_pipe_vfs_id);
}

The way pipe_start_select and pipe_end_select aew outdated which led me to believe that this changes were not based against the master branch.
I think that to be receivable (given the size of the addition), it would be best that your PR(s) are created against the latest commits of the master branch.

I have been working on a refactor of the console component of esp-idf for the past few months.
In the scope of this refactor, I have split the console into several components that are (or will be) available in the component registry.
The idea being that the console as a component of esp-idf will be "deprecated" in the future and users will be encouraged to use the newly created components instead.

Which brings me to my last point. I think it might be more interesting to merge the console related changes of this PR directly in those newly created components. This way, the "new" console will implement this shell feature. And since I will replace the implementation of the esp-idf console with the new components, this feature will - by extension - also be available in the esp-idf console.

Here is the links to the components:

• esp_linenoise : A new version of linenoise supporting multi-instances.
• esp_cli_commands : Component handling command registration and execution. The successor of commands.c, allowing for static and dynamic command registration → were your changes will be merged most likely
• esp_cli : The successor to the REPL logic in the esp-idf console component. Puts esp_linenoise, esp_cli_commands and the REPL logic together and supports multi-instances.

[SoucheSouche]

I will try to take some time this week to continue and review in more details the submitted feature.
If you agree to split this PR and submit the shell related changes in the new components I created, we can further discuss the how.

[jimmyw]

Hi, thanks for feedback.

You are right, I wrote this feature on 5.4 that we currently use in production, and ported it forward just for this attempt to upstream it.

This PR was mostly a request for feedback (DRAFT), to see if there is any interest in this at all, not really meant for merging as is. That's also why i did not break it up into separate commits, just so people could see how things are combined.

I think your new modular approach is a really good idea, and are happy to contribute with your guidance.

Any suggestion where to start?

[ammaree]

+1 +1 +1 +1 for this please!

…

On Tuesday, 11 November 2025 at 11:35, Jimmy Wennlund ***@***.***> wrote: jimmyw left a comment [(espressif/esp-idf#17793)](#17793 (comment)) Hi, thanks for feedback. You are right, I wrote this feature on 5.4 that we currently use in production, and ported it forward just for this attempt to upstream it. This PR was mostly a request for feedback (DRAFT), to see if there is any interest in this at all, not really meant for merging as is. That's also why i did not break it up into separate commits, just so people could see how things are combined. I think your new modular approach is a really good idea, and are happy to contribute with your guidance. Any suggestion where to start? — Reply to this email directly, [view it on GitHub](#17793 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAYT7GBKCA6WP4NMVGKSRH334GUUXAVCNFSM6AAAAACKUVOVVKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTKMJVHA3TCMZYGQ). You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[SoucheSouche]

Hi, thanks for feedback.

You are right, I wrote this feature on 5.4 that we currently use in production, and ported it forward just for this attempt to upstream it.

This PR was mostly a request for feedback (DRAFT), to see if there is any interest in this at all, not really meant for merging as is. That's also why i did not break it up into separate commits, just so people could see how things are combined.

I think your new modular approach is a really good idea, and are happy to contribute with your guidance.

Any suggestion where to start?

I will take a deeper look into your changes this week. I'll reach back after that and we will see then :)
I expect esp_cli and esp_cli_commands to be merged into idf-extra-components by the end of the week. After that it will be easier for you to open PRs in idf-extra-components since the components will exist on master.

But I guess starting by isolating the changes to VFS (pipe addition) into a separate PR might be a good start.
The rest of the changes (apart from the examples) will most likely go into those standalone components I created.
When all those new components are merged and available from the registry, I will update the esp-idf console component to use those new components internally. In this MR I can incorporate the additions you made to the examples.

[SoucheSouche]

@jimmyw, just a little update:

• I merged all the new components I mentioned in one of the messages above. esp_cli (that internally depends on esp_linenoise and esp_cli_commands can now be used as an alternative to the existing esp-idf console component.

• I had a look at your PR a little bit more:

  • apart from the example updates and the pipe addition that will have to be merged in esp-idf, it seems that everything else can either be added in the new component esp_cli_commands or constitute a new component.
    • at first glance, it seems to me that esp_shell could become its own component (provided that pipes are available in esp-idf VFS component and that esp_cli_commands offers esp_console_run_on_task. If I understand it properly, it constitutes an overlayer on top of esp_cli_commands since it provides the same "run" API and rely on esp_cli_commands in its logic.
  • esp_cli_commands provides esp_cli_commands_execute (the equivalent of the old esp_console_run. I hid all the command specific context under the parameter cmd_args (of type esp_cli_commands_exec_arg_t) which will make our life easier to integrate esp_console_run_on_task in the component since we can extend the content of esp_cli_commands_exec_arg_t to contain the parameters of your added function. So I think it will be possible to keep the API of esp_cli_commands unchanged and just slightly modify esp_cli_commands_execute to check the content of esp_cli_commands_exec_arg_t to understand whether it should perform the classic logic or call your "on_task" logic. If we manage this, you will be able to call esp_cli_commands_execute with the proper context in your esp_shell functions.

The only thing remaining to solve in order to completely integrate this nice addition is how to integrate esp_shell in a way that it can be used by the new component esp_cli directly. Solving this would allow users to pull esp_cli in theirs projects together with esp_shell and pass esp_shell_run while instantiating esp_cli to force it to use this function instead of esp_cli_commands_execute. Without this option, you basically have to re-implement esp_cli (exactly like it is done in the examples/system/console/advanced that you updated).

I think I could update esp_cli to offer the option to provide custom "execute command" functions. That way, esp_shell would integrate nicely within the new console design and it would prevent users from having to re-implement the logic of esp_cli in order to use esp_shell.

I am sure that we will find more problems as we go but this approach seems like a good way to go. Let me know what you think.

I you want to take a look at the components that are now replacing the old idf console, you can check:

https://github.com/espressif/idf-extra-components/tree/master/esp_cli
https://github.com/espressif/idf-extra-components/tree/master/esp_cli_commands
https://github.com/espressif/idf-extra-components/tree/master/esp_linenoise