804357 - symbols pushes should retry and/or be able to select among multiple destinations

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

Currently, if the symbol server is down, symbol uploads will fail, full stop.

It would be great to:
 * retry this a few times on failure (can retry.py do this?)
 * select among a configured set of destination hosts, either sequentially or randomly, so that if one goes down the others will pick up its load

For example, right now we're using symbols1.dmz.phx1.mozilla.com which is a single host in phx1.  With some client-side resiliency and retries, we could add symbols2, so that a failure of either host would not cause build failures.

Comment 1

[[e-mail] (:bhearsum)]

I think this is a dupe of bug 799654, which went live today.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

The first part is, and that's great!  What do you think about adding the second part?  The other option is to switch to using the Zeus VIP, so Zeus can do the failover for you.

Comment 3

[Chris AtLee [:catlee]]

The symbol server host is set as an environment variable that 'make uploadsymbols' uses, so it would be a bit tricky to add more than one host in there without changing the build system. I think client-side retry + load balancer is cleanest.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

So for the moment that means using symbolpush.mozilla.org.

symbolpush.mozilla.org is an alias for symbols1.pub.phx1.mozilla.com.
symbols1.pub.phx1.mozilla.com has address 63.245.216.250

Don't do that yet, though - I'd first like to move that to Zeus (it's NAT now), if possible.  I'll file a bug blocking this one.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Bug 804995 is to the point where symbolpush.mozilla.org is the appropriate name/IP to be using to upload symbols.  At the moment, all access methods work and are equivalent:

 symbols1.pub.phx1.mozilla.com / 63.245.216.250
 symbols1.dmz.phx1.mozilla.com / 10.8.74.48
 symbolpush.mozilla.org / symbolpush.zlb.phx.mozilla.net / 63.245.217.193

So switching which one is used should not cause any problems or require any coordination.

The last is the one I would like to standardize on.  Once that's complete, we can put multiple hosts behind the zlb with failover, and eliminate an SPOF.

The new VIP is accessible from at least some of the build network:

[cltbld@dev-stage01 ~]$ nc -vz 63.245.217.193 22
Connection to 63.245.217.193 22 port [tcp/ssh] succeeded!

but please verify that from each affected VLAN before enabling.

So the action here is:
 * verify all slaves that push symbols can reach symbolpush.mozilla.org
 * replace 'symbols1.dmz.phx1.mozilla.com' in the buildbot configs with 'symbolpush.mozilla.org'.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Any chance this can get fixed up?  It should be nice and simple, and will eliminate an SPOF.

Comment 7

[Kim Moir [:kmoir] ET]

Attachment: patch for buildbot configs

I verified all the slaves types could ssh to the host.  The old host name is also referenced in mozharness scripts.

Comment 8

[Kim Moir [:kmoir] ET]

Attachment: mozharness patch

Comment 9

[Kim Moir [:kmoir] ET]

Attachment: puppet patch

I think a patch for puppet is needed too, not sure how to deploy this patch in a non-breaking fashion.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 721485 [details] [diff] [review]
puppet patch

dustin@cerf ~ $ host symbolpush.mozilla.org
symbolpush.mozilla.org is an alias for symbolpush.zlb.phx.mozilla.net.
symbolpush.zlb.phx.mozilla.net has address 63.245.217.193

so you should be able to add the new one with the correct external IP.  Then there's no conflict (aside from listing the same key twice, which -- I think -- isn't a problem)

Comment 11

[Kim Moir [:kmoir] ET]

Attachment: puppet patch

Comment 12

[Kim Moir [:kmoir] ET]

Comment on attachment 721679 [details]
puppet patch

Not sure how we can test this in staging since there is a different symbols server in staging.

Comment 13

[Chris AtLee [:catlee]]

You can log onto production machines of various OSes that need to push symbols here and make sure they can login ok.

Does windows need a known_hosts update too?

Comment 14

[Kim Moir [:kmoir] ET]

Yes windows needs a host update too. Good catch.  I forgot about the machines that aren't managed by Puppet.

Comment 15

[Kim Moir [:kmoir] ET]

Comment on attachment 721679 [details]
puppet patch

I also updated the known_hosts on all the production windows build machines (mw32-ix-slave and w64-ix-slave).  I don't think the try build machines need this change so I left them alone.

Comment 16

[Aki Sasaki (not active)]

Comment on attachment 721344 [details] [diff] [review]
mozharness patch

This is merged to production.

Comment 17

[Aki Sasaki (not active)]

This is in production.

Comment 18

[Kim Moir [:kmoir] ET]

I verified that this is working on recent builds in tbpl.

Comment 19

[Chris AtLee [:catlee]]

We forgot to update /N/production/darwin11-x86_64/build/Users/cltbld/.ssh/known_hosts for the bld-lion-r5 slaves. I've added this on scl3-production-puppet. Kim, can you make sure the other puppet masters and staging copies of this file are updated as well?

Probably a good idea to update any other known_hosts files on /N too.

Comment 20

[Kim Moir [:kmoir] ET]

I updated them.

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

Should really have waited until bug 849942 was resolved before landing the buildbot-configs patch. Some of the windows machines are having issues uploading symbols.

Comment 22

[Kim Moir [:kmoir] ET]

Sorry nthmomas, I didn't think it would take long to resolve fixing the image in bug 849942.

Comment 23

[Nick Thomas [:nthomas] (UTC+12)]

My mistake actually, I missed comment #15 here.