We have at least 3 sites around the world, working on the same code&asset base. We are investigating the idea of using P4 Commit-Edge architecture, with a Commit server in our main site in EU, and Edge servers in each other location.
For that purpose, we set up a basic commit-edge configuration from scratch, following the steps exposed here: https://www.perforce...stributed.setup
We first tried a EU / North America configuration, but observed a huge latency when submitting an important amount of files.
Suspecting the network, we tried to run both commit and edge server on the same local machine (on 2 different ports), but the issue still remain.
Here what is observed with P4V, from a computer on the same local network:
- We submit 16 000 new binary files of 525B each to the edge server from workspace A
- The progress bar complete within a few seconds
- The progress bar hangs at 100% for about 30 min
During this time:
- A client connected to the commit server see everything as if the changelist has been perfectly submitted, he can get latest and get those files.
- A client connected to the edge server won't see the submitted files before the progress bar finish hanging.
If we cancel the submit when it hang at 100%:
- Files still appear as submitted and are available from "get latest" on the commit server
- On workspace A:
* the submitted changelist has a "shelve" icon, but we do not see any shelved files, or do not contains any files anymore.
* we can't remove the changelist, because "it contains some shelved files"
* the added files are not "marked for add" anymore.
- about 30 minutes latter, workspace A becomes normal, same state as commit server, as if every thing were submitted successfully.
We tried with less files: it's faster, but still very slow:
30 files: 2 seconds
100 files: 6 seconds
500 files: 34 seconds
However, it took only 115 ms to submit those 500 files directly on the commit server. But the edge server still takes minutes to show the files.
That make us think the pulling is the bottleneck here. "p4 pull" show the following during the "un-consistent" state (output of "pull -l" varies with time):
perforce@perforceproxy:~$ p4 pull -lj Current replica journal state is: Journal 0, Sequence 3983733. Current master journal state is: Journal 0, Sequence 8309721. The statefile was last modified at: 2015/11/18 15:40:01. The replica server time is currently: 2015/11/18 15:41:23 +0100 CET perforce@perforceproxy:~$ p4 pull -l //depot/test4000/1a5d33d5-1a43-467a-8985-9e2309ee6d3e.bin 1.1 binary active add 8C8B1FBB2FD5F825C1014A3907962CDA 525 1 2015/11/18 15:40:01 2015/11/16 23:13:18 4296 2015/11/18 15:41:30 1 0 //depot/test4000/1a69015b-0be9-4446-a795-3eebd5b0feda.bin 1.1 binary new add 4CF1EFF5A5E1A96EC3FB3C2ECFE07422 525 1 2015/11/18 15:40:01 2015/11/16 23:13:18 4295 2015/11/18 15:41:29 1 0 //depot/test4000/1a6c905f-1b08-47c5-8e68-5d492b40c739.bin 1.1 binary new add BBD5DAA2B714526B4684B2D9BD3FA9FF 524 1 2015/11/18 15:40:01 2015/11/16 23:13:18 4295 2015/11/18 15:41:29 1 0 //depot/test4000/1a6daaa1-f677-43c8-af15-594b05af581f.bin 1.1 binary new add 02BA9C6DB142974782222F600A7D1CD5 524 1 2015/11/18 15:40:01 2015/11/16 23:13:18 4295 2015/11/18 15:41:30 1 0 //depot/test4000/1a70188c-a4e0-43eb-96ad-ad2bcb886d6c.bin 1.1 binary new add FB8F53634F4AB013944508ED0DC8E90F 524 1 2015/11/18 15:40:01 2015/11/16 23:13:18 4295 2015/11/18 15:41:30 1 0 //depot/test4000/1a70fad0-0592-4308-ac7f-d806ecc5759b.bin 1.1 binary new add 0AFB84C2407BA75A398B5268E8BFE48A 525 1 2015/11/18 15:40:01 2015/11/16 23:13:18 4295 2015/11/18 15:41:30 1 0 //depot/test4000/1a710263-ea15-4a3e-b5ee-921ee87d8f3b.bin 1.1 binary new add 3B35FF33A7A70B6F2E8EF1BC57918F58 523 1 2015/11/18 15:40:01 2015/11/16 23:13:18 4295 2015/11/18 15:41:30 1 0 //depot/test4000/1a729830-8098-46ec-84a8-c3f4e3116db1.bin 1.1 binary new add F01C2341E63E64A353B24BDDA24129DF 524 1 2015/11/18 15:40:01 2015/11/16 23:13:18 4295 2015/11/18 15:41:30 1 0 //depot/test4000/1a76c6e8-187f-461c-bf3e-a14b6ecf890a.bin 1.1 binary new add AB8AD36DA09992DF06644F2B6BF19747 524 1 2015/11/18 15:40:01 2015/11/16 23:13:18 4295 2015/11/18 15:41:30 1 0
- p4 server is P4D/LINUX26X86_64/2015.2/1252060 (2015/10/22)
- server is a debian linux
- both edge and commit refer to each other with localhost:1667 and localhost:1668
What we tried:
* with and without unicode
* with and without db.peeking=2
* with pull -i 1 and pull -i 0 (0 seems faster)
* with or without exclusive checkout
What we didn't try (yet):
* text file
* testing it on an isolated vm, outside the network
* probably 1 000 thinks we didn't even know about
* why would the edge server need to pull everything when it is the one that submitted the file?
* if pulling is the issue, why is it so fast to transfer the file one way, and so slow the other way?
* we are about to test a simple forwarding replicate. But I guess everything tends to say that we'll have the same issue with pulling?
* most of all: any tips/idea/guess on what could be wrong here?
* In the tutorial linked above, it seems that step 4 of "Create and start the edge server" has a wrong ticket path, should be "/tokyo/p4root/.p4tickets" instead of "/chicago/p4root/.p4tickets", since we create the edge ticket. Am I right?
* Also, could you explain the double "pull -u -i 1" in the edge configuration?
* We were very surprised with the "same timezone" condition. What is the reason behind that? Do you plan to allow different timezone in the future?
Thanks for reading me