Jump to content


Converting commit server to replica without checkpoint?

replica

  • Please log in to reply
5 replies to this topic

#1 Sean Houghton

Sean Houghton

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 23 February 2017 - 10:32 PM

I'm setting up commit and replica servers and I would like to swap their roles automatically on a schedule. Stopping the old commit server and restarting the replica as the new commit server works, but when I try to start the old commit server as a replica I get errors about it still being a commit-server. I'm assuming this is because the "p4 server" definition still includes "Services commit-server" locally (the services have been swapped in the new commit server's database).

I know I could take a synchronous checkpoint on the new commit server to seed the new replica, as mentioned here, but that could take hours or days. Is there any way to convert the old commit server into a functional replica without causing a huge service outage? I could introduce a third replica and take an "offline" checkpoint there, but that just makes things more complicated to orchestrate.


Notes:
I'm using P4NAME to control replication configuration which has been pre-configured for each server to function as either commit or replica servers.


Perforce server warning:
server_1  |     This server is configured as a type of commit-server. Database and librarian replication flags have been ignored.
server_1  | Perforce Server starting...
server_1  | Perforce server error:
server_1  |     2017/02/23 19:59:10 pid 45 replicasvc@1666 background 'pull -i 1'
server_1  |     Pull only allowed on replica servers.

#2 Mailman Sync

Mailman Sync

    Advanced Member

  • Maillist Aggregator
  • 2495 posts

Posted 23 February 2017 - 10:50 PM

Originally posted to the perforce-user mailing list by: Michael Mirman


We swap a commit server with a standby replica on a monthly basis (controlled fail-over).
When we start any server, we use the -In option, which specifies the server id.
(We also write P4ROOT/serverid file, but it's more for an assertion check.)

So, when we fail over, we have to swap those id's. Say, if your commit server is called Master and replica is Replica, you shut down both, and then restart with the swapped id's in the -In option.
Does it make sense?

--
Michael Mirman
MathWorks, Inc.
3 Apple Hill Drive, Natick, MA 01760
508-647-7555

Quote

-----Original Message-----
From: perforce-user [mailto:perforce-user-bounces@perforce.com] On
Behalf Of Sean Houghton
Sent: Thursday, February 23, 2017 5:35 PM
To: perforce-user@perforce.com
Subject: [p4] Converting commit server to replica without checkpoint?

Posted on behalf of forum user 'Sean Houghton'.

I'm setting up commit and replica servers and I would like to swap their
roles automatically on a schedule. Stopping the old commit server and
restarting
the replica as the new commit server works, but when I try to start the old
commit server as a replica I get errors about it still being a commit-server.
I'm assuming this is because the "p4 server" definition still
includes "Services commit-server" locally (the services have been
swapped in the new commit server's database).

I know I could take a synchronous checkpoint on the new commit server to
seed
the new replica, as mentioned   here
[http://answers.perfo...ticles/KB/2495] , but that could take hours
or
days. Is there any way to convert the old commit server into a functional
replica without causing a huge service outage? I could introduce a third
replica
and take an "offline" checkpoint there, but that just makes things
more complicated to orchestrate.


Notes:
I'm using P4NAME to control replication configuration which has been
pre-configured for each server to function as either commit or replica
servers.


Perforce server warning:
server_1��|���� This server is configured as a
type of commit-server. Database and librarian replication flags have been
ignored.
server_1��| Perforce Server starting...
server_1��| Perforce server error:
server_1��|���� 2017/02/23 19:59:10 pid 45
replicasvc@1666 background 'pull -i 1'
server_1��|���� Pull only allowed on replica
servers.



--
Please click here to see the post in its original format:
  http://forums.perfor...verting-commit-
server-to-replica-without-checkpoint
_______________________________________________
perforce-user mailing list  -  perforce-user@perforce.com
http://maillist.perf...o/perforce-user



#3 Matt Janulewicz

Matt Janulewicz

    Advanced Member

  • Members
  • PipPipPip
  • 223 posts
  • LocationSan Diego, CA

Posted 23 February 2017 - 10:53 PM

Are you sure the old commit is coming up correctly using the replica server spec? If you do 'p4 info' on it when it's in the wrong state, is ServerID what you expect?

We use the SDP, not P4NAME, so for us it's as simple as editing the server.id file in P4ROOT, but this is the type of thing I remember seeing when I forget to change that. I'd be suspicious that P4NAME isn't working properly, or was not changed to reflect the ServerID of the replica.

In any case, this should be possible. We manually swap our read-only and commit server once in a while. It's just a matter of shutting both down, changing server.id (or P4NAME in your case), bringing back up the new commit then the new replica. Then also not forgetting that P4TARGET on any replicas pointing at the new commit are likely in need of an update. Last time we did this we were down for less than 5 minutes.

It does seem odd, though, that the error message(s) seem to indicate it's starting as a commit, but then trying to start pull threads. That sounds like a mixed message to me, like the configurables are off. In our scenario, we only ever have to change the P4TARGET configurable, everything else stays the same. In this instance, are you changing server specs, names, configurables or anything like that?
-Matt Janulewicz
Currently unemployed, looking for work in Boise, ID!

#4 Sean Houghton

Sean Houghton

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 24 February 2017 - 05:55 PM

I'm only changing the "-In" argument when I restart pd4 which changes P4NAME but leaves the server id intact. My expectation was that you could change the "role" of a server while still keeping its identity. It sounds like the common practice is to actually swap the identity of two servers rather than just changing the configuration with P4NAME.

The documentation says

The recommended technique for configuring servers in a multi-server installation is to give each server its own serverid, and specify the server configuration for that serverid; specifying a separates P4NAME for the server is generally not necessary,

"give each server its own serverid" is misleading given that you need to play musical chairs with serverids as part of a controlled failover.



So this is the process?
  • stop both servers

  • swap server ID on both servers

  • Do validation pass on the new commit server
  • start up the new commit server

  • change  Address value for both server IDs with "p4 server" on the new commit server

  • change P4TARGET for the replica with "p4 configure" on the new commit server

  • change P4TARGET for the replica with "p4d -cset" on the old commit server

  • start up the old commit server server


#5 Mailman Sync

Mailman Sync

    Advanced Member

  • Maillist Aggregator
  • 2495 posts

Posted 24 February 2017 - 06:20 PM

Originally posted to the perforce-user mailing list by: Michael Mirman


Our procedure is shorter. We don’t have to change P4TARGET on disk. We have -t option on the command line.

The command line to start p4d actually does not change. What changes is *where* we run it.
The commit server is always started by something like
p4d -d -p 1666 -In Master -u service -r $dbroot -J $jnlroot/journal
and its standby replica is always started by
p4d -d -p 1666 -In Standby -r $dbroot -J off -t $master:1666

In these commands $dbroot and $jnlroot actually don’t change.
What changes is $master - depending on which host it is today, and *where* these commands run (again, they flip).

Back to the list of actions:
- make sure the replica is completely caught up with the commit server
-  stop both servers
-  swap server ID on both servers - meaning, rewrite P4ROOT/serverid on both hosts
-  start up the new commit server (no -t option, and -In Master)
-  start up the old commit server server (-t option, and -In Standby)
-  change Address and ExternalAddress values for both server IDs with "p4 server" on the new commit server

We do other things, but they are probably irrelevant for you.
For example, we keep the archive mounted read-only on the replicas and writable on the actual servers (commit and edge).
But that's to make sure that replicas don't interfere in any way with the archive files.

--
Michael Mirman
MathWorks, Inc.
3 Apple Hill Drive, Natick, MA 01760
508-647-7555

Quote

-----Original Message-----
From: perforce-user [mailto:perforce-user-bounces@perforce.com] On
Behalf Of Sean Houghton
Sent: Friday, February 24, 2017 1:00 PM
To: perforce-user@perforce.com
Subject: Re: [p4] Converting commit server to replica without checkpoint?

Posted on behalf of forum user 'Sean Houghton'.

I'm only changing the "-In" argument when I restart pd4 which
changes P4NAME but leaves the server id intact. My expectation was that
you
could change the "role" of a server while still keeping its identity.
It sounds like the common practice is to actually swap the identity of two
servers rather than just changing the configuration with P4NAME.

The documentation says

The recommended technique for configuring servers in a multi-server
installation
is to give each server its own serverid, and specify the server configuration
for that serverid; specifying a separates   P4NAME
[https://www.perforce...ref/P4NAME.html
]  for the
server is generally not necessary,

" give each server its own serverid " is misleading given that you
need to play musical chairs with serverids as part of a controlled failover.



So this is the process?


-  stop both servers

-  swap server ID on both servers

-  start up the new commit server

-  change��Address value for both server IDs with "p4
  server" on the new commit server

-  change P4TARGET for the replica with "p4 configure" on the new
  commit server

-  change P4TARGET for the replica with "p4d -cset" on the old commit
  server

-  start up the old commit server server




--
Please click here to see the post in its original format:
  http://forums.perfor...verting-commit-
server-to-replica-without-checkpoint
_______________________________________________
perforce-user mailing list  -  perforce-user@perforce.com
http://maillist.perf...o/perforce-user



#6 Robert Cowham

Robert Cowham

    Advanced Member

  • PCP
  • 271 posts
  • LocationLondon, UK

Posted 24 February 2017 - 06:40 PM

Be careful - there is at least one gotcha. If you have a P4NAME var in your environment, then it can override any –In or server.id file – so server starts up with wrong id. Been caught by this myself.

I would avoid using P4NAME at all, and just use –In flag to p4d. Alternatively, make sure they change in lockstep.

I would personally like to:
- consign P4NAME to the eternal flames
- use either –In flag or server.id, but complain and refuse to start if they are both present and different

Regards
Robert

On 23/02/2017, 22:35, "perforce-user on behalf of Sean Houghton" <perforce-user-bounces@perforce.com on behalf of perforce-user-forum@forums.perforce.com> wrote:

    Posted on behalf of forum user 'Sean Houghton'.

    I'm setting up commit and replica servers and I would like to swap their
    roles automatically on a schedule. Stopping the old commit server and restarting
    the replica as the new commit server works, but when I try to start the old
    commit server as a replica I get errors about it still being a commit-server.
    I'm assuming this is because the "p4 server" definition still
    includes "Services commit-server" locally (the services have been
    swapped in the new commit server's database).

    I know I could take a synchronous checkpoint on the new commit server to seed
    the new replica, as mentioned   here
    [http://answers.perfo...ticles/KB/2495] , but that could take hours or
    days. Is there any way to convert the old commit server into a functional
    replica without causing a huge service outage? I could introduce a third replica
    and take an "offline" checkpoint there, but that just makes things
    more complicated to orchestrate.


    Notes:
    I'm using P4NAME to control replication configuration which has been
    pre-configured for each server to function as either commit or replica servers.


    Perforce server warning:
    server_1��|���� This server is configured as a
    type of commit-server. Database and librarian replication flags have been
    ignored.
    server_1��| Perforce Server starting...
    server_1��| Perforce server error:
    server_1��|���� 2017/02/23 19:59:10 pid 45
    replicasvc@1666 background 'pull -i 1'
    server_1��|���� Pull only allowed on replica
    servers.



    --
    Please click here to see the post in its original format:
      http://forums.perfor...hout-checkpoint
    _______________________________________________
    perforce-user mailing list  -  perforce-user@perforce.com
    http://maillist.perf...o/perforce-user



_______________________________________________
perforce-user mailing list  -  perforce-user@perforce.com
http://maillist.perf...o/perforce-user

Co-Author of "Learning Perforce SCM", PACKT Publishing, 25 September 2013, ISBN 9781849687645

"It's wonderful to see a new book about Perforce, especially one written by Robert Cowham and Neal Firth. No one can teach Perforce better than these seasoned subject matter experts"
  • Laura Wingerd, author of Practical Perforce, former VP of Product Technology at Perforce





Also tagged with one or more of these keywords: replica

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users