Concatenate various fields #113
Description
Problem
Is it possible to concatenate more than one field ? For the key value ?
And is it possible to choose the order of concatenation. For example I have line1 with result field INPROGRESS and line2 with esult field SUCCESS. I would like to put line1 StartUserName in line 2 . With my actual configuration It took line1 as a reference and add line2 field in.
...
here is my config
<source>
@type http
@id input_http
port 8888
</source>
<filter jks-build.**>
@type concat
key startedUserName
separator "|"
n_lines 2
use_first_timestamp true
</filter>
<match jks-build.**>
@type stdout
</match>
Expected Behavior
Line 1
{"ciUrl":"url","jobName":"Test-job","fullJobName":"Test-job","number":18,"slaveInfo":{"slaveName":"master","executor":"1","label":"master"},"startTime":1645523967664,"endTime":1645523967665,"startedUserId":"name","startedUserName":"name","result":"INPROGRESS","duration":0,"parameters":{},"scmInfo":{"url":"","branch":"","commit":""},"queueTime":2,"buildUrl":"job/Test-job/18/","contextId":258453088,"buildCause":"Started by user name","buildFailureCauses":[]}
Line 2
{"ciUrl":"url","jobName":"Test-job","fullJobName":"Test-job","number":18,"slaveInfo":{"slaveName":"","executor":"","label":""},"startTime":1645523967696,"endTime":1645523967696,"startedUserId":"","startedUserName":"","result":"SUCCESS","duration":29,"parameters":{},"scmInfo":{"url":"","branch":"","commit":""},"queueTime":0,"buildUrl":"job/Test-job/18/","contextId":0,"buildCause":"","buildFailureCauses":[]}
Output
{"ciUrl":"url":"Test-job","fullJobName":"Test-job","number":18,"slaveInfo":{"slaveName":"master","executor":"0","label":"master"},"startTime":1645701396629,"endTime":1645701396633,"startedUserId":"name","startedUserName":"name|","result":"INPROGRESS","duration":0,"parameters":{},"scmInfo":{"url":"","branch":"","commit":""},"queueTime":1,"buildUrl":"job/Test-job/41/","contextId":82069267,"buildCause":"Started by user name","buildFailureCauses":[]}
The ouput I want
{"ciUrl":"url","jobName":"Test-job","fullJobName":"Test-job","number":18,"slaveInfo":{"slaveName":"","executor":"","label":""},"startTime":1645523967696,"endTime":1645523967696,"startedUserId":"","startedUserName"name"","result":"SUCCESS","duration":29,"parameters":{},"scmInfo":{"url":"","branch":"","commit":""},"queueTime":0,"buildUrl":"job/Test-job/18/","contextId":0,"buildCause":"","buildFailureCauses":[]}
...
Your environment
- OS version
- td-agent 4.3.0 fluentd 1.14.3 (438a82aead488a86180cd484bbc4e7e344a9032b)
- plugin version
- paste boot log of fluentd or td-agent
- addressable (2.8.0)
async (1.30.1)
async-http (0.56.5)
async-io (1.32.2)
async-pool (0.3.9)
aws-eventstream (1.2.0)
aws-partitions (1.534.0)
aws-sdk-core (3.123.0)
aws-sdk-kms (1.51.0)
aws-sdk-s3 (1.107.0)
aws-sdk-sqs (1.47.0)
aws-sigv4 (1.4.0)
benchmark (default: 0.1.0)
bigdecimal (default: 2.0.0)
bindata (2.4.10)
bundler (2.2.32, default: 2.1.4)
cgi (default: 0.1.0.1)
concurrent-ruby (1.1.9)
console (1.14.0)
cool.io (1.7.1)
csv (default: 3.1.2)
date (default: 3.0.3)
delegate (default: 0.1.0)
did_you_mean (default: 1.4.0)
digest-crc (0.6.4)
digest-murmurhash (1.1.1)
elasticsearch (7.15.0)
elasticsearch-api (7.15.0)
elasticsearch-transport (7.15.0)
etc (default: 1.1.0)
excon (0.88.0)
faraday (1.8.0)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-httpclient (1.0.1)
faraday-net_http (1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
fcntl (default: 1.0.0)
ffi (1.15.4)
fiber-local (1.0.0)
fiddle (default: 1.0.0)
fileutils (1.6.0, default: 1.4.1)
fluent-config-regexp-type (1.0.0)
fluent-diagtool (1.0.1)
fluent-logger (0.9.0)
fluent-plugin-concat (2.5.0)
fluent-plugin-elasticsearch (5.1.4)
fluent-plugin-flowcounter-simple (0.1.0)
fluent-plugin-kafka (0.17.3)
fluent-plugin-prometheus (2.0.2)
fluent-plugin-prometheus_pushgateway (0.1.0)
fluent-plugin-record-modifier (2.1.0)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-s3 (1.6.1)
fluent-plugin-sd-dns (0.1.0)
fluent-plugin-systemd (1.0.5)
fluent-plugin-td (1.1.0)
fluent-plugin-utmpx (0.5.0)
fluent-plugin-webhdfs (1.5.0)
fluentd (1.14.3)
forwardable (default: 1.3.1)
getoptlong (default: 0.1.0)
hirb (0.7.3)
http_parser.rb (0.8.0)
httpclient (2.8.3)
io-console (default: 0.5.6)
ipaddr (default: 1.2.2)
irb (default: 1.2.6)
jmespath (1.4.0)
json (2.6.1, default: 2.3.0)
linux-utmpx (0.3.0)
logger (default: 1.4.2)
ltsv (0.1.2)
matrix (default: 0.2.0)
mini_portile2 (2.6.1)
minitest (5.13.0)
msgpack (1.4.2)
multi_json (1.15.0)
multipart-post (2.1.1)
mutex_m (default: 0.1.0)
net-pop (default: 0.1.0)
net-smtp (default: 0.1.0)
net-telnet (0.2.0)
nio4r (2.5.8)
nokogiri (1.12.5 x86_64-linux)
observer (default: 0.1.0)
oj (3.13.9)
open3 (default: 0.1.0)
openssl (default: 2.1.3)
ostruct (default: 0.2.0)
parallel (1.21.0)
power_assert (1.1.7)
prime (default: 0.1.1)
prometheus-client (2.1.0)
protocol-hpack (1.4.2)
protocol-http (0.22.5)
protocol-http1 (0.14.2)
protocol-http2 (0.14.2)
pstore (default: 0.1.0)
psych (default: 3.1.0)
public_suffix (4.0.6)
racc (1.6.0, default: 1.4.16)
rake (13.0.6, 13.0.1)
rdkafka (0.11.1)
rdoc (default: 6.2.1.1)
readline (default: 0.0.2)
readline-ext (default: 0.1.0)
reline (default: 0.1.5)
rexml (default: 3.2.3.1)
rss (default: 0.2.8)
ruby-kafka (1.4.0)
ruby-progressbar (1.11.0)
ruby2_keywords (0.0.5)
rubyzip (1.3.0)
sdbm (default: 1.0.0)
serverengine (2.2.4)
sigdump (0.2.4)
singleton (default: 0.1.0)
stringio (default: 0.1.0)
strptime (0.2.5)
strscan (default: 1.0.3)
systemd-journal (1.4.2)
td (0.16.9)
td-client (1.0.8)
td-logger (0.3.27)
test-unit (3.3.4)
timeout (default: 0.1.0)
timers (4.3.3)
tracer (default: 0.1.0)
tzinfo (2.0.4)
tzinfo-data (1.2021.5)
uri (default: 0.10.0)
webhdfs (0.10.2)
webrick (1.7.0, default: 1.6.1)
xmlrpc (0.3.0)
yajl-ruby (1.4.1)
yaml (default: 0.1.0)
zip-zip (0.3)
zlib (default: 1.1.0)