Jump to content


Help understanding how Continuous Import works

insights continuous import

  • Please log in to reply
3 replies to this topic

#1 jspang

jspang

    Member

  • Members
  • PipPip
  • 26 posts

Posted 05 September 2014 - 12:02 AM

I installed the Insights OVA on a VM on my workstation, then uploaded+imported a Checkpoint from our Perforce server. Everything worked ok that far. Now, I need to pull in some newer data, but I don't understand how the Continuous Import tool works. Whenever I run it, I get this error:
2014-09-04 20:53:12,507: [ERROR] [service.DatabaseImportService]: Failed to start the continuous import, reason [java.lang.NullPointerException], stacktrace;
java.lang.NullPointerException
at com.perforce.journal.parser.NoteJournalEntry.toString(NoteJournalEntry.java:201)
at java.lang.String.valueOf(String.java:2838)
at java.lang.StringBuilder.append(StringBuilder.java:132)
at com.perforce.journal.reader.FileJournalReader$2.run(FileJournalReader.java:195)
at java.lang.Thread.run(Thread.java:701)

It seems that Insights is looking for a journal file but failing to find it. Where is it looking for one? The documentation says this:

Quote

The streaming import uses "p4 export" to read changes out of the Perforce
Server as they happen. If it needs to catch up, then changes are read out
of the journals. It is important that any rotated journals are uncompressed,
otherwise this data cannot be read. Consult the Perforce administration
documentation for information on how to manage checkpoints and journals,
but in brief, the -Z (rather than -z) flag will compress checkpoints but
leave rotated journals uncompressed.

Unless specified with the server configurable 'journalPrefix', journals are
expected to be in the P4ROOT. If journals are being moved elsewhere after
being rotated, or if a prefix is being specified on the command line, then
the streaming process will not be able to find them. If journals are being
stored outside of the P4ROOT (which is recommended), then it is important to
use the 'journalPrefix' server configurable to control this.
So... does that mean Insights is running "p4 export" to determine what journal file it needs, and then looking for that Journal file somewhere? Do I need to upload it like I did with the checkpoint? That bit about "journals are expected to be in the P4ROOT" confuses me. There doesn't appear to be a configuration in Insights for 'P4ROOT'. Does it mean the P4ROOT of my Perforce server?

Any clarification (or assistance in fixing the error I'm getting) would be greatly appreciated.

#2 P4Martin

P4Martin

    Member

  • Staff
  • 21 posts
  • LocationUK

Posted 05 September 2014 - 09:26 AM

Hi there,  I'll see if I can clarify a bit for you.

Insights connects to your perforce server and uses the "p4 export" command to get the perforce server itself to transmit the journal data.
In order for the perforce server to give journal data to Insights, it needs to know where its own journal files are.

Depending on how your system admin rotates the journals, there are several places they may be, and it's possible that the perforce server doesn't actually know this location.  If the files are moved after the journal is rotated (e.g. by a backup script), or if a parameter is passed on the command line to rotate the journals, then there is no configuration inside the perforce sevrer to tell it where to look for the files.  This means that the perforce server can't find it's own old journals, and so can't pass them to Insights.

Do check out the more detailed documentation below.  Step 4: "Enable continuous import"
http://www.perforce....tion_k53_1ny_km


If you need more help getting this up and going, please contact support where we can provide you with more direct help.

Martin.

#3 jspang

jspang

    Member

  • Members
  • PipPip
  • 26 posts

Posted 06 September 2014 - 12:38 AM

Thanks Martin. That definitely helps my understanding of how this works, but I don't think I'm any closer to resolving this error.

When I examine my Perforce server log, I see Insights running this command:
2014/09/05 16:01:44 pid 3084 jspang@noclient 10.10.10.10 [Insights/2014.1/825802] 'user-export -j1363/1523759 -r'

However, when I log into the Insights VM, and manually run the same command, p4 outputs a bunch of data (so presumably it's working correctly). How can I troubleshoot why Insights is getting this npe?

EDIT: Here are two details which may or may not help solve this. I'm just throwing them out there in case they're relevant:
  • Our Perforce server is Windows
  • We don't store Job information in the //spec depot (but it's still accessible with 'p4 jobs').


#4 P4Martin

P4Martin

    Member

  • Staff
  • 21 posts
  • LocationUK

Posted 08 September 2014 - 08:11 AM

Please give Support a shout - they will be able to give you detailed help to get you going!

http://www.perforce....contact-support

Martin.





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users