Skip to content

Clickhouse keeper entrypoint chown problem #794

@nenych

Description

@nenych

I checked the Altinity Stable Builds lifecycle table, and the Altinity Stable Build version I'm using is still supported.

Type of problem

Choose one of the following items, then delete the others:

Unexpected behavior - something surprising happened, but it wasn't the good kind of surprise

Describe the situation

In the entrypoint.sh (clickhouse keeper) you have functionality that changes owner of files from root to clickhouse:

    if [ "$DO_CHOWN" = "1" ]; then
        # ensure proper directories permissions
        # but skip it for if directory already has proper premissions, cause recursive chown may be slow
        if [ "$(stat -c %u "$dir")" != "$USER" ] || [ "$(stat -c %g "$dir")" != "$GROUP" ]; then
            chown -R "$USER:$GROUP" "$dir"
        fi
    elif ! $gosu test -d "$dir" -a -w "$dir" -a -r "$dir"; then
        echo "Necessary directory '$dir' isn't accessible by user with id '$USER'"
        exit 1
    fi

Our pod had restarted before the startup was finished, after restart we got this issue:

2025.04.29 09:30:00.965175 [ 1 ] {} <Error> Application: std::exception. Code: 1001, type: std::__1::__fs::filesystem::filesystem_error, e.what() = filesystem error: in remove_all: Permission denied ["/var/lib/clickhouse-keeper/rocksdb"], Stack trace
(when copying this message, always include the lines below):

0. std::system_error::system_error(std::error_code, String const&) @ 0x0000000012572b57
1. std::__fs::filesystem::filesystem_error::filesystem_error[abi:ne180100](String const&, std::__fs::filesystem::path const&, std::error_code) @ 0x000000000b100f7f
2. void std::__fs::filesystem::__throw_filesystem_error[abi:ne180100]<String&, std::__fs::filesystem::path const&, std::error_code const&>(String&, std::__fs::filesystem::path const&, std::error_code const&) @ 0x000000001253814d
3. std::__fs::filesystem::detail::ErrorHandler<unsigned long>::report(std::error_code const&) const @ 0x000000001253cb49
4. DB::KeeperContext::initializeDisks(Poco::Util::AbstractConfiguration const&) @ 0x000000000d624ca8
5. DB::KeeperContext::initialize(Poco::Util::AbstractConfiguration const&, DB::KeeperDispatcher*) @ 0x000000000d6223f5
6. DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::shared_ptr<DB::Macros const> const&) @ 0x000000000d634675
7. DB::Context::initializeKeeperDispatcher(bool) const @ 0x000000000bd1df92
8. DB::Keeper::main(std::vector<String, std::allocator<String>> const&) @ 0x0000000005ec75f8
9. Poco::Util::Application::run() @ 0x000000001192b8c6
10. DB::Keeper::run() @ 0x0000000005ec4370
11. mainEntryClickHouseKeeper(int, char**) @ 0x0000000005ec2fc2
12. main @ 0x0000000005ec1b73
13. ? @ 0x0000785c695f7d90
14. ? @ 0x0000785c695f7e40
15. _start @ 0x000000000501c02e

It was fixed by changing the owner.

How to reproduce the behavior

  • Which Altinity Stable Build version to use:
    Any latest build

Stop the pod during recursive chowning

Expected behavior

Chown any time or more checks to mitigate this behavior.

Logs, error messages, stacktraces, screenshots...

Provided above

Additional context

Add any other context about the issue here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions