Skip to content

Issue with Logstash Processing Updated CSV File #91

Description

@xhayang04

I am using Logstash to process logs from a CSV file and send them to Elasticsearch. Here is my Logstash configuration:
input {
file {
path => "E:/AuditTrail/AuditTrail0.csv"
start_position => "beginning"
sincedb_path => "NUL"
codec => plain { charset => "UTF-8" }
mode => "tail" # Use tail mode to read only new data
}
}

filter {
csv {
separator => ","
columns => ["RecordID", "TimeStamp", "DeltaToUTC", "UserID", "ObjectID", "Description", "Comment", "Checksum"]
skip_empty_columns => true
skip_header => true
}

date {
match => ["TimeStamp", "MM/dd/yyyy hh:mm:ss a"]
target => "@timestamp"
}
}

output {
elasticsearch {
hosts => ["http://10.6.1.114:9200"]
index => "audit_trail_logs"
document_type => "_doc"
}
stdout {
codec => rubydebug
}
}

The sample output looks like this:
{
"message" => "80,"7/8/2024 2:02:54 PM","-7:00","System","User administration","User logged off.",,ikay85\r",
"@Version" => "1",
"ObjectID" => "User administration",
"host" => {
"name" => "DESKTOP-KE828UP"
},
"@timestamp" => 2024-07-08T07:02:54.000Z,
"RecordID" => "80",
"Checksum" => "ikay85",
"DeltaToUTC" => "-7:00",
"log" => {
"file" => {
"path" => "E:/AuditTrail/AuditTrail0.csv"
}
},
"UserID" => "System",
"event" => {
"original" => "80,"7/8/2024 2:02:54 PM","-7:00","System","User administration","User logged off.",,ikay85\r"
},
"Description" => "User logged off.",
"TimeStamp" => "7/8/2024 2:02:54 PM"
}

However, when I update the CSV file and add new lines, the resulting output contains parsing errors:
{
"message" => "ystem","User administration","User 'Admin' has failed to log on successfully 2 times.",,DtOX3A\r",
"log" => {
"file" => {
"path" => "E:/AuditTrail/AuditTrail0.csv"
}
},
"event" => {
"original" => "ystem","User administration","User 'Admin' has failed to log on successfully 2 times.",,DtOX3A\r"
},
"tags" => [
"_csvparsefailure"
],
"host" => {
"name" => "DESKTOP-KE828UP"
},
"@Version" => "1",
"@timestamp" => 2024-07-08T09:28:48.433441600Z
}

What could be causing this issue and how should I fix it?

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions