org.jruby.exceptions.RaiseException: (GeneratorError) source sequence is illegal/malformed utf-8

JIRA | Lucas Bremgartner | 4 years ago
tip
Click on the to mark the solution that helps you, Samebug will learn from it.
As a community member, you’ll be rewarded for you help.
  1. 0

    In my setup I use logstash to handle syslog input. One of my network appliances (mail-relay) sends logmessages with "strange" or may be wrong encoding, which leads to crashing logstash. I think, the above message is not validly encoded. In my opinion, if logstash is used in mission-critical environments, it should be tollerant to such "wrongly encoded" events. I checked with the file output and the redis output. Error Message (Redis): {noformat} Exception in thread "LogStash::Runner" org.jruby.exceptions.RaiseException: (GeneratorError) source sequence is illegal/malformed utf-8 at json.ext.GeneratorMethods$RbHash.to_json(json/ext/GeneratorMethods.java:71) at LogStash::Event.to_json(file:/opt/logstash/logstash-1.2.1-flatjar.jar!/logstash/event.rb:169) at LogStash::Outputs::Redis.receive(file:/opt/logstash/logstash-1.2.1-flatjar.jar!/logstash/outputs/redis.rb:150) at RUBY.initialize((eval):17) at org.jruby.RubyProc.call(org/jruby/RubyProc.java:255) at LogStash::Pipeline.output(file:/opt/logstash/logstash-1.2.1-flatjar.jar!/logstash/pipeline.rb:247) at RUBY.outputworker(file:/opt/logstash/logstash-1.2.1-flatjar.jar!/logstash/pipeline.rb:212) at RUBY.start_outputs(file:/opt/logstash/logstash-1.2.1-flatjar.jar!/logstash/pipeline.rb:140) {noformat} Error Message (File): {noformat} Exception in thread "LogStash::Runner" org.jruby.exceptions.RaiseException: (GeneratorError) source sequence is illegal/malformed utf-8 at json.ext.GeneratorMethods$RbHash.to_json(json/ext/GeneratorMethods.java:71) at LogStash::Event.to_json(file:/opt/logstash/logstash-1.2.1-flatjar.jar!/logstash/event.rb:169) at LogStash::Outputs::File.receive(file:/opt/logstash/logstash-1.2.1-flatjar.jar!/logstash/outputs/file.rb:68) at RUBY.initialize((eval):28) at org.jruby.RubyProc.call(org/jruby/RubyProc.java:255) at LogStash::Pipeline.output(file:/opt/logstash/logstash-1.2.1-flatjar.jar!/logstash/pipeline.rb:247) at RUBY.outputworker(file:/opt/logstash/logstash-1.2.1-flatjar.jar!/logstash/pipeline.rb:212) at RUBY.start_outputs(file:/opt/logstash/logstash-1.2.1-flatjar.jar!/logstash/pipeline.rb:140) {noformat} I recorded the corrupted messages with tcpdump and was able to reproduce the exception. Steps to reproduce: 1. Start logstash with following config-file: {noformat} input { udp { port => 514 type => syslog } } output { file { path => "/tmp/out.log" flush_interval => 0 } } {noformat} 2. Send correct message to syslog {noformat} echo -en "syslog correct" | nc -u localhost 514 {noformat} (Check /tmp/out.log for the message 3. Send corrupted message to syslog {noformat} echo -en "syslog corrupted: \xfc" | nc -u localhost 514 {noformat} (Nothing is written to /tmp/out.log, logstash crashes with above error-message) I double-checked with an alternative logstash configuration with charset "ASCII-8BIT". In this case the logline wasn't written to /tmp/out.log but logstash at least kept working normaly. Alternative logstash config: {noformat} input { udp { port => 515 type => syslog codec => plain { charset => "ASCII-8BIT" } } } output { file { path => "/tmp/fortimail-log/out.log" flush_interval => 0 } } {noformat}

    JIRA | 4 years ago | Lucas Bremgartner
    org.jruby.exceptions.RaiseException: (GeneratorError) source sequence is illegal/malformed utf-8

    Root Cause Analysis

    1. org.jruby.exceptions.RaiseException

      (GeneratorError) source sequence is illegal/malformed utf-8

      at json.ext.GeneratorMethods$RbHash.to_json()
    2. json.ext
      GeneratorMethods$RbHash.to_json
      1. json.ext.GeneratorMethods$RbHash.to_json(json/ext/GeneratorMethods.java:71)
      1 frame