the back burner

I’ve decided to bring this forward a bit. This is somewhat motivated by the need to get spam[34] up and running so that Adam can start to move his build scripts to target the MBS deployment infrastructure.

There was a tentative plan to SAN-attach the MBS. I’d much rather supply its disk off NFS, which is must faster to find a replacement host for. My plan is to rebuild the MBS setup on hereford (currently a farm host on PRE) and punt defiant back into the farm.

We’ve seen some issues in keeping a pair of MBS DB instances in sync; I’m looking forward to using some of my upcoming copious free time to do some fairly hefty development work on the MBS side (which is long overdue). In order to make this work smoothly and to help with patch management, I’m planning on making a change to the way the MBS is laid out, as follows:

• all MBS checkouts will go directly onto NFS-supplied disk. We’ll also be supplying home dirs from the NFS cluster so that we aren’t in any danger of running out of elbow-room.
• the mbs/src, mbs/zfs, mbs/apache and other “common infrastructure” parts will live under one repo, laid out as follows:

  mbs.src/trunk HEAD, development happens here (or on dev branches)
  mbs.src/branches/mbs.pre HEAD pushed out to here
  mbs.src/branches/mbs.prod …and here
  mbs.src/branches/... …and so on.

  The mbs.src repo will not only contain the code for deployment methods; it’ll also have a skeletal mbs/db that contains the core artifacts common to all database instances: the platform-deployment, bootloader, and os images. These can then be maintained from a single spot.

  Note: in fact I think I might further split up mbs.src and mbs.db into two repos, and put the large OS images under the latter, out of the way of the deployment method implementations.

• all MBS database instances will then be built as separate repos. Each repo will contain svn:externals pointers to the mbs.src:src/, mbs.db:db/image, etc, subdirectories that should be centrally maintained. I may well punt software images (apache, tomcat, etc) into this too.
• This shouldn’t make the blindest bit of difference to any of the scripts, with the exception that the apache users.xml and groups.xml will need to be moved out of those particular subdirectories.
• Checkouts: finally - I’m strongly tempted to run each MBS instance under its own “zone” using OpenVZ. There have been some issues with the PXE platform-deployment instances getting in the way of each other; this’ll let me split them up cleanly.

Schedule: Initially, I said that once the NFS service was bedded down I’d begin this work, andexpect it to run over the summer. At some point there’d be a cutover as the new MBS comes into service.

Actually, I’m going to start prepping this now. This’ll mean cutting a new pair of repos for mbs.src and migrating the existing history into it.

Once that it done I’ll put a freeze on the src/ and other non-db trees whilst I rejig the existing repositories. I expect that the tidy-up work there will take a day or so to sort out, some time in the next couple of weeks.

I’ll also be rolling out a third (count ‘em!) MBS DB instance for the mail system stuff. Initially this’ll be for the spam boxes, although we might look at managing the webmail through this too, since it looks eminently amenable to that treatment.

Tags: appinf, configuration management, deployment, mail

This entry was posted on Thursday, April 24th, 2008 at 1:39 pm and is filed under haste. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.