Don't be all things to all people. Focus only on Heroku log format

Author: rwdaigle

README.md

@@ -1,8 +1,8 @@
 heroku-log-parser
 =======
 
-A multi-provider [syslog (rfc5424)](http://tools.ietf.org/html/rfc5424#section-6) parser written in Ruby and specifically
-targeting Heroku's log drain format.
+A [syslog (rfc5424)](http://tools.ietf.org/html/rfc5424#section-6) parser written in Ruby and specifically
+targeting Heroku's [http log drain](https://devcenter.heroku.com/articles/labs-https-drains).
 
 ## Install
 
@@ -20,35 +20,20 @@ $ bundle install
 
 ## Usage
 
-heroku-log-parser is built on the concept of syslog message flavors. In my brief experience with syslog streams,
-everybody seems to do it differently. So it is necessary to handle these inconsistencies on a per-provider
-basis.
-
-Currently the only flavor available is the flavor for Heroku's log-stream.
-
-Create a parser based on your desired flavor:
-
 ```ruby
-log_parser = heroku-log-parser.parser(:heroku).new
-```
-
-A parser is a stateless, regex-based object that accepts a string of data holding one or more syslog messages
-and emits a hash containing the individual parts of a syslog message. For those unwilling to read the spec, the
-list of syslog tokens is as follows (and is stored in the `heroku-log-parser::SYSLOG_KEYS` array):
+msg_str = "156 <40>1 2012-11-30T06:45:26+00:00 heroku web.3 d.73ea7440-270a-435a-a0ea-adf50b4e5f5a - Starting process with command `bundle exec rackup config.ru -p 24405`"
 
-```ruby
-heroku-log-parser::SYSLOG_KEYS
-#=> [:priority, :syslog_version, :emitted_at, :hostname, :appname, :proc_id, :msg_id, :structured_data, :message]
+HerokuLogParser.parse(msg_str)
+#=> [{:priority=>40, :syslog_version=>1, :emitted_at=>2012-11-30 06:45:26 UTC, :hostname=>"heroku", :appname=>nil, :proc_id=>"web.3", :msg_id=>"d.73ea7440-270a-435a-a0ea-adf50b4e5f5a", :structured_data=>nil, :message=>"Starting process with command `bundle exec rackup config.ru -p 24405`"}]
 ```
 
-To parse a message packet, invoke the `events` method.
+`HerokuLogParser` is a stateless, regex-based parser that accepts a string of data holding one or more syslog messages
+and returns an array of syslog message properties for each message. For those unwilling to read the spec, the
+list of syslog tokens is as follows (and is stored in the `HerokuLogParser::SYSLOG_KEYS` array):
 
 ```ruby
-msg_str = "156 <40>1 2012-11-30T06:45:26+00:00 heroku web.3 d.73ea7440-270a-435a-a0ea-adf50b4e5f5a - Starting process with command `bundle exec rackup config.ru -p 24405`"
-log_parser.events(msg_str) do |event|
-  event.inspect
-  #=> {:priority=>40, :syslog_version=>1, :emitted_at=>2012-11-30 06:45:26 UTC, :hostname=>"heroku", :appname=>nil, :proc_id=>"web.3", :msg_id=>"d.73ea7440-270a-435a-a0ea-adf50b4e5f5a", :structured_data=>nil, :message=>"Starting process with command `bundle exec rackup config.ru -p 24405`"}
-end
+HerokuLogParser::SYSLOG_KEYS
+#=> [:priority, :syslog_version, :emitted_at, :hostname, :appname, :proc_id, :msg_id, :structured_data, :message]
 ```
 
 ## Contributions
@@ -59,12 +44,9 @@ end
 
 * TESTS!!!!
 * 2nd order parsing. For instance, for parsing a structured message body into key=value pairs (including the structured_data message part)
-* Docs for creating diff flavors
-* Pure Syslog compliant default parser
-* Less hacky flavor dynamic loading
 
 ## Issues
 
 Please submit all issues to the project's Github issues.
 
--[@rwdaigle](https://twitter.com/rwdaigle)
+-- [@rwdaigle](https://twitter.com/rwdaigle)

heroku-log-parser.gemspec

@@ -3,7 +3,7 @@ require 'heroku-log-parser/version'
 
 Gem::Specification.new do |s|
   s.name              = "heroku-log-parser"
-  s.version           = heroku-log-parser::VERSION
+  s.version           = HerokuLogParser::VERSION
   s.platform          = Gem::Platform::RUBY
   s.author            = "Ryan Daigle"
   s.email             = ["[email protected]"]

lib/heroku-log-parser.rb

@@ -1,74 +1,77 @@
-module HerokuLogParser
+class HerokuLogParser
 
   SYSLOG_KEYS = :priority, :syslog_version, :emitted_at, :hostname, :appname, :proc_id, :msg_id, :structured_data, :message
 
-  # Never done this before, doesn't feel very good
-  def self.parser(flavor = :heroku)
-    constantize("Parsley::Flavors::#{flavor.to_s.capitalize}")
-  end
-
-  # Shouldn't be referenced directly. Always request a flavor of a parser
-  # Parsley.parser(:heroku).new(syslog_str)
-  class Parser
+  class << self
 
-    def events(data_str, &block)
+    def parse(data_str)
+      events = []
       lines(data_str) do |line|
         if(matching = line.match(line_regex))
-          yield event_data(matching)
+          events << event_data(matching)
         end
       end
+      events
     end
 
     protected
 
-    # Since the Heroku format is the only one I've tested, it's the default until broader
-    # flavor support is added
+    # http://tools.ietf.org/html/rfc5424#page-8
+    # frame <prority>version time hostname <appname-missing> procid msgid [no structured data = '-'] msg
+    # 120 <40>1 2012-11-30T06:45:29+00:00 heroku web.3 d.73ea7440-270a-435a-a0ea-adf50b4e5f5a - State changed from starting to up
     def line_regex
-      @line_regex ||= /\<(\d+)\>(1) (\d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d\+00:00) ([a-z0-9-]+) ([a-z0-9\-\_\.]+) ([a-z0-9\-\_\.]+) \- (.*)$/
+      @line_regex ||= /\<(\d+)\>(1) (\d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d\+00:00) ([a-z0-9-]+) ([a-z0-9\-\_\.]+) ([a-z0-9\-\_\.]+) (\-) (.*)$/
     end
 
-    # Break a given packet into individual syslog messages. Default assumes one message per packet
+    # Heroku's http log drains (https://devcenter.heroku.com/articles/labs-https-drains)
+    # utilize octet counting framing (http://tools.ietf.org/html/draft-gerhards-syslog-plain-tcp-12#section-3.4.1)
+    # for transmission of syslog messages over TCP. Properly parse and delimit
+    # individual syslog messages, many of which may be contained in a single packet.
+    #
+    # I am still uncertain if this is the place for transport layer protocol handling. I suspect not.
+    #
     def lines(data_str, &block)
-      yield data_str
+      d = data_str
+      while d && d.length > 0
+        if matching = d.match(/^(\d+) /) # if have a counting frame, use it
+          num_bytes = matching[1].to_i
+          frame_offset = matching[0].length
+          line_end = frame_offset + num_bytes
+          msg = d[frame_offset..line_end]
+          yield msg
+          d = d[line_end..d.length]
+        elsif matching = d.match(/\n/) # Newlines = explicit message delimiter
+          d = matching.post_match
+        else
+          STDERR.puts("Unable to parse: #{d}")
+          return
+        end
+      end
     end
 
+    # Heroku is missing the appname token, otherwise can treat as standard syslog format
+    def event_data(matching)
+      event = {}
+      event[:priority] = matching[1].to_i
+      event[:syslog_version] = matching[2].to_i
+      event[:emitted_at] = nil?(matching[3]) ? nil : Time.parse(matching[3]).utc
+      event[:hostname] = interpret_nil(matching[4])
+      event[:appname] = nil
+      event[:proc_id] = interpret_nil(matching[5])
+      event[:msg_id] = interpret_nil(matching[6])
+      event[:structured_data] = interpret_nil(matching[7])
+      event[:message] = interpret_nil(matching[8])
+      event
+    end
+
+    private
+
     def interpret_nil(val)
       nil?(val) ? nil : val
     end
 
     def nil?(val)
       val == "-"
     end
-
-    # Default is to assume simple sequential matching
-    # Comment out until it's actually used
-    # def event_data(matching)
-    #   event = {}
-    #   event[:priority] = matching[1].to_i
-    #   event[:syslog_version] = matching[2].to_i
-    #   event[:emitted_at] = nil?(matching[3]) ? nil : Time.parse(matching[3]).utc
-    #   event[:hostname] = interpret_nil(matching[4])
-    #   event[:appname] = interpret_nil(matching[5])
-    #   event[:proc_id] = interpret_nil(matching[6])
-    #   event[:msg_id] = interpret_nil(matching[7])
-    #   event[:structured_data] = interpret_nil(matching[8])
-    #   event[:message] = interpret_nil(matching[9])
-    #   event
-    # end
-  end
-
-  # Taken from ActiveSupport::Inflector: http://apidock.com/rails/v3.2.8/ActiveSupport/Inflector/constantize
-  # Hate that this is here - how do others do dynamic loading?
-  def self.constantize(camel_cased_word)
-    names = camel_cased_word.split('::')
-    names.shift if names.empty? || names.first.empty?
-
-    constant = Object
-    names.each do |name|
-      constant = constant.const_defined?(name) ? constant.const_get(name) : constant.const_missing(name)
-    end
-    constant
   end
-end
-
-require "parsley/flavors"
+end