Unable to print msgpack to stdout using out_lib.c

[Dim-P]

Bug Report

Describe the bug
I am trying to use the out_lib.c example. The printf to stdout works fine when using the JSON format, however when I am trying the commented out code to print to stdout using the msgpack format, I am getting the following result:
[my_stdout_msgpack]#<UNKNOWN 1644222354 7883960872767849142>

To Reproduce
Use the out_lib.c example example and replace the default output that uses JSON with the commented out msgpack format output.

Additional context
I am trying to embed Fluent Bit into my application using the provided C API. While I can read my logs in using the tail plugin, I can parse them fine and I can also print them to stdout in JSON format, I cannot print them using the msgpack format (I need this functionality for debugging purposes but also to understand how msgpack works, so that I can extract the required data - unless there is a better way to extract the parsed log values using their keys?).

[patrick-stephens]

Msgpack is a binary format so watch out for that, if you print it raw then you'll get some strange results depending on your data.

This is why in the example I pushed it through a python library: https://docs.fluentbit.io/manual/pipeline/outputs/tcp-and-tls#msgpack-format

I'm not sure if the library function handles that for you but it might be what's happening.

[Dim-P]

Thanks for your answer @patrick-stephens . I managed to print to stdout using something along these lines (borrowed from somewhere within the Fluent Bit library):

int my_stdout_msgpack(void *record, size_t size, void *data){

    msgpack_object obj = *(msgpack_object*) record;

    printf("[%s]",__FUNCTION__);

    msgpack_unpacked result;
    size_t off = 0, cnt = 0;
    struct flb_time tmp_time;
    msgpack_object *x;

    msgpack_unpacked_init(&result);

    while (msgpack_unpack_next(&result, record, size, &off) == MSGPACK_UNPACK_SUCCESS) {
        printf("[%zd]: [", cnt++);
        flb_time_pop_from_msgpack(&tmp_time, &result, &x);
        printf("%"PRIu32".%09lu, ", (uint32_t)tmp_time.tm.tv_sec, tmp_time.tm.tv_nsec);

        // msgpack_object_print(stdout, *x);

        if(x->type == MSGPACK_OBJECT_MAP){
            fprintf(stdout, "{");
            if(x->via.map.size != 0) {
                msgpack_object_kv* p = x->via.map.ptr;
                msgpack_object_kv* const pend = x->via.map.ptr + x->via.map.size;
                msgpack_object_print(stdout, p->key);
                fprintf(stdout, "=>");
                msgpack_object_print(stdout, p->val);
                ++p;
                for(; p < pend; ++p) {
                    if (p->key.type != MSGPACK_OBJECT_STR || p->val.type != MSGPACK_OBJECT_STR) {
                        break;
                    }
                    fprintf(stdout, ", ");
                    msgpack_object_print(stdout, p->key);
                    fprintf(stdout, "=>");
                    msgpack_object_print(stdout, p->val);
                }
            }
            fprintf(stdout, "}");
        } // else ??

        printf("]\n");
    }
    msgpack_unpacked_destroy(&result);

    flb_lib_free(record);
    return 0;
}

Shall I make a PR to update the out_lib.c example? I think other new users could benefit from that.

[nokute78]

@Dim-P Could you sent a PR if you can?
I think this is a bug.

Fluent-bit supports two timestamp formats. (Unix epoch and Event time)
https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1#entry

examples/out_lib/out_lib.c expects the format is Unix epoch.
It should support Event time like your above code.

[patrick-stephens]

Agreed, this example also shows how to unpack fields which is useful too. I'd leave it included too rather than commented out like the current stdout example so we can verify examples build.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[Dim-P]

Comment to keep issue alive.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[Dim-P]

Keep alive. Will fix.