{"id":2818,"date":"2015-12-23T00:48:44","date_gmt":"2015-12-23T00:48:44","guid":{"rendered":"http:\/\/www.garysieling.com\/blog\/?p=2818"},"modified":"2015-12-23T00:48:44","modified_gmt":"2015-12-23T00:48:44","slug":"monitor-appharbor-with-the-elk-stack","status":"publish","type":"post","link":"https:\/\/www.garysieling.com\/blog\/monitor-appharbor-with-the-elk-stack\/","title":{"rendered":"Monitor AppHarbor with the ELK Stack"},"content":{"rendered":"

If you followed my previous post on setting up the ELK stack on Azure<\/a>, you can configure AppHarbor to use it fairly easily.<\/p>\n

You need to open a port on Azure (I’ve chosen 9000). You can set the inbound IP range to AppHarbors IPs1<\/a><\/sup>.
\n\"image1\"<\/p>\n

Once you do this, make a new file in \/etc\/logstash\/conf.d called AppHarbor.conf, and enter the following into it’s contents:<\/p>\n

\ninput {\n  tcp {\n    port => 9000\n    type => syslog\n  }\n  udp {\n    port => 9000\n    type => syslog\n  }\n}\nfilter {\n   grok {\n        match => [ \"message\",  \"%{GREEDYDATA:syslog_message}\" ]\n   }\n   mutate { replace => { type => \"appharbor\" } }\n}\noutput {\n  elasticsearch {\n    hosts => \"localhost\"\n  }\n}\n<\/pre>\n
While the AppHarbor logs supposedly conform to “syslog” format, I did not find this to be true – as of this time, I haven’t established the best log format yet, but this is easier to control than using “syslog” as an input type.<\/p>\n
If you don’t do this, you will get a lot of messages tagged with “grokparsefailure_sysloginput”.<\/p>\n
Adding this to AppHarbor is super-easy, start by selecting a logging configuration:
\n\"image2\"<\/p>\n
Select the option to add a new log drain:
\n\"image3\"<\/p>\n
Then, add the URL. This url needs to be in the format of “syslog:\/\/test.domain.com:9000”. If you use HTTP instead of “syslog”, you will get errors that say “(output buffer overflow)”, and lose all the useful data2<\/a><\/sup>
\n\"image4\"<\/p>\n
Once you finish, this is what you’ll see:
\n\"image4\"<\/p>\n
And you should be all set.<\/p>\n
If you want to get the contents of these log messages into specific fields, I recommend the grok debugger<\/a>.<\/p>\n
  1. https:\/\/support.appharbor.com\/kb\/tips-and-tricks\/application-server-ips-ec2-configuration [↩<\/a>]<\/span><\/li>
  2. http:\/\/stackoverflow.com\/questions\/17532337\/error-l10-output-buffer-overflow-when-writing-to-splunk-drain [↩<\/a>]<\/span><\/li><\/ol>","protected":false},"excerpt":{"rendered":"
    If you followed my previous post on setting up the ELK stack on Azure, you can configure AppHarbor to use it fairly easily. You need to open a port on Azure (I’ve chosen 9000). You can set the inbound IP range to AppHarbors IPs1. Once you do this, make a new file in \/etc\/logstash\/conf.d called … <\/p>\n
    Continue reading “Monitor AppHarbor with the ELK Stack”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[9,22,25],"tags":[56,74,169,330,344,347,370,522],"aioseo_notices":[],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts\/2818"}],"collection":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/comments?post=2818"}],"version-history":[{"count":0,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts\/2818\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/media?parent=2818"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/categories?post=2818"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/tags?post=2818"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}