Jump to content


Log parsing for ELK stack


  • Please log in to reply
5 replies to this topic

#1 sammex

sammex

    Newbie

  • Members
  • Pip
  • 6 posts

Posted 12 April 2018 - 02:20 PM

We are looking over or log management and want to integrate Perforce in our ELK stack using logstash or something similar that would fit in.

Do anyone have a grok pattern that would fit perforce server logs?

Are anyone else using ELK for perforce logs?

Thanks,
Sammex

#2 Guest_dwhitfield_*

Guest_dwhitfield_*
  • Guests

Posted 12 April 2018 - 03:28 PM

This came up in late February on an internal list and as far as the list conversation went, this is not something anyone has a lot of familiarity with here. If you were willing to take a look at Nagios Log Server (NLS), then you could get help with the grok filter at https://support.nagi...forum.php?f=37. There are pluses and minuses to NLS, but if you're already familiar with the ELK stack, then the biggest minus to NLS is going to be that it's not a pure ELK stack. If NLS is an option, post the link to the Nagios support thread and I'll keep an eye on that too.

You can get the trial of NLS at https://www.nagios.c...ios-log-server/ . Perforce doesn't have a relationship with Nagios, but I previously worked for Nagios, so I know a bit about their solution. Unfortunately, as I did not work on NLS, and have never worked on ELK, a grok filter is beyond me. However, if you told us what info you wanted to be pulled into ELK, we could tell you where that information if located and what log levels you would need.

In theory, you could build the grok filter in NLS and then move it to a pure ELK solution, but I do not know what versions of NLS/ELK are compatible. Nagios should be able to tell you that information for the purposes of modules and such.


EDIT: I left out an "if" in the second paragraph, last sentence.

Edited by dwhitfield, 13 April 2018 - 01:30 PM.


#3 Matt Janulewicz

Matt Janulewicz

    Advanced Member

  • Members
  • PipPipPip
  • 134 posts
  • LocationSan Francisco, CA

Posted 12 April 2018 - 05:59 PM

I'm not familiar wit ELK, we use Splunk and sometimes Graylog. But generally, I'd suggest running appropriate structured logs if you plan on ingesting them into a log parser. The standard p4 log is a bear to parse consistently.
-Matt Janulewicz
Staff SCM Engineer, Perforce Administrator
Dolby Laboratories, Inc.
1275 Market St.
San Francisco, CA 94103, USA
majanu@dolby.com

#4 phopkins

phopkins

    Advanced Member

  • Members
  • PipPipPip
  • 97 posts

Posted 12 April 2018 - 10:32 PM

Yeah we use ELK for other logging items but haven't thought about using it for perforce. I think Matts existing method is good, would be interested to see if / how you did end up using logstash.

#5 Guest_dwhitfield_*

Guest_dwhitfield_*
  • Guests

Posted 13 April 2018 - 07:19 PM

I just happened to run across P4Splunk: https://swarm.worksh...-allen-p4splunk

It's obviously not ELK, but if you have flexibility there, might be useful.

#6 sammex

sammex

    Newbie

  • Members
  • Pip
  • 6 posts

Posted 19 April 2018 - 12:50 PM

Thanks for all your response. I think we're going with a ELK stack using structured logs actually. I can report back when we have everything set up.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users