Jump to content

Having difficulty with virtual streams and Jenkins continuous integration server

  • Please log in to reply
3 replies to this topic

#1 tmcsweeney



  • Members
  • PipPip
  • 12 posts

Posted 27 January 2020 - 10:12 PM

We have an infrastructure set up that heavily uses streams for development, and also uses a Jenkins build server to run continuous integration across a subset of those streams.   We use Jenkin's multi-branch scanner to automatically scans all of the streams looking for the existence of a jenkins config file in the stream, and if it exists scheduling a CI build for that stream.

This has been working fine for a number of years now. but we recently started using Virtual streams as well, which has thrown up some issues with the Jenkins Server.

The first problem is that Jenkins check's for the existence of its config file inside a stream using the p4 files command. We've been getting away with this up until now because the stream name has always matched the actual depot path, but with virtual streams the depot path is always different p4 files reports that there are no files at //depot/virtual-stream-name/... and so Jenkins skips building that stream.   (In theory this could happen with a non-virtual stream as well but the default behavior of the location being the same as the stream name has saved us).

The second problem is similar except that it is with the p4 changes command that Jenkins runs in order to decide if anything has changed within the stream and needs rebuilding, again Jenkins tries to use the //depot/virtual-stream-name/... filespec to search for changes but fails to find anything because that path doesn't actually exist, it is only virtual.

I understand why this is happening, but I'd appreciate some help on what to do about it.  

From playing around on the command line I think we can resolve the second issue by changing the order of operations, having Jenkins construct (or re-use) the workspace for the virtual stream and then issues the command as: p4 -c jenkins-virtualstream-workspace changes rather than p4 changes //depot/virtual-stream-name/... so that p4 can correctly map the paths through the clientspec defined by the virtual stream.

However I'm at somewhat of a loss as how to fix the p4 files bit inside the branch scanner.  I know I could in theory do a similar trick and specify something like: p4 -c jenkins-virtualstream-workspace files //jenkins-virtualstream-workspace/... but we don't want to create a workspace for every stream, (The whole point of this system is that only a subset of the streams get noticed by the CIBuild).  Is there some other way we can check for the existence of the file inside a stream, virtual or otherwise?

Thanks in advance

#2 asmith



  • Members
  • Pip
  • 2 posts

Posted 30 January 2020 - 10:28 PM

I'm confused -- are you trying to run CI on the virtual stream, but not its parent (the actual development stream?) If so: why?

We use a similar workflow, with a multi-branch Jenkins job over a stream depot. Our mainline streams include a source art directory that is, in no uncertain terms, huge. I don't want my Jenkins agents checking that out, so we have a standard virtual stream that gets created for every development stream like //depot/streamname_jenkins/...

Jenkins then finds //depot/streamname/path/to/Jenkinsfile and schedules it for build. We use a lightweight checkout and explicitly check out and submit to the "${BRANCH}_jenkins" stream instead.

I realize this forces you to make a virtual stream for every development stream you want CI running for. That was a palatable cost for our development team, but YMMV.

#3 Sambwise


    Advanced Member

  • Members
  • PipPipPip
  • 1065 posts

Posted 02 February 2020 - 06:28 AM

Creating a temp workspace is the only way, unfortunately.  

It’s bugged me for a long time that there’s no such thing as a “stream syntax” to handle this use case, but the fact that stream names correspond to their associated depot paths (instead of being a “domain” like clients/labels/etc) makes that difficult to do even in theory.

#4 umar96



  • Members
  • PipPip
  • 12 posts

Posted 19 May 2020 - 01:41 PM

I have basically the same problem. I work on a rather large mono-repo and have a development stream that aggregates several Maven modules that my team is responsible for. We also have several development files unique to our team, but not applicable to the mono-repo as a whole. I don't want those files to ever get copied up to mainline. We've solved this with streams by dumping those files in another stream, and disabling copy up to parent. We then include them in our development stream with import. This seems to work well: it ensures those files are still available in our task streams (child streams of our development stream), and it prevents us from accidentally copying up to mainline.

The problem I'm having is with the multi-branch pipeline I have based on my development stream: if I put my Jenkinsfile in the other stream and import it in to my development stream, then no pipelines are ever created because Jenkins doesn't find the Jenkinsfile. Even if I fix that problem by isolating the Jenkinfile in my development stream (making it invisible to task streams), I still have the aggregator POM and other files that aren't being found. It would be great if the P4 Jenkins plugin actually respected stream specifications instead of treating everything like as a branch (from what I can tell, basically just a directory structure).

Some of you mentioned creating a workspace, but I don't quite understand. What exactly do you mean by that, and how did you go about doing that? Is it something you changed in the multibranch configuration? Something in the Jenkinsfile?

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users