This is exactly not what I’m after. Apparently this is fixable by adding

__allow_access_to_unprotected_subobjects__ = 1

to the class definition. It’s irritating to have to modify external methods in order to make them Zope-ready; at least the fix is (apparently) a one-liner.

Mind you, there is also a bunch of lack in the Oracle calendar SOAP interface: no WSDL, no querying of delegate calendars, a limited SOAP implementation that appears to have been written in terms of SAX events and that’s particularly sensitive to namespace declarations, and so on.

Anyway: I got enough of this working that I can query a user’s calendar in a fairly Pythonic fashion - https://svn.cse.bris.ac.uk/svn/jan/trunk/calendar/ for details. It’ll suffice to reimplement the guts of Shirley, the ILRT’s automatic receptionist.

This also marked the first time I’ve tried using new-style classes to augment the Python str class with a secondary attribute.

Having tested this tonight, we can indeed pull out a list of ilrt-visitor meetings, together with their organiser’s details (mailto and cn).

our discipline has a tried and tested way of going faster. Cut corners, cut quality. That way you can produce more crap.

So, how does this relate to the DFS?

Well, I’m coming under pressure to declare that the DFS is production-ready. This isn’t a technical thing, this is purely one of PR. Where does this pressure come from? From “on high” - ie, from someone who is one hop away from seeing anything other than that we have a working cluster, what’s holding things up? (Bob’s actually pretty reasonable about this - he’s stuck between a rock and an opinionated git.)

There have been unavoidable supplier and technical delays involved in getting this far. The trouble is that dates have been randomly selected on no basis whatsoever (actually, on the basis of having a meeting and me saying, “it will take n days of uninterrupted work by all involved with nothing else getting in the way, assuming no impediments, no unforseen hitches, and the continued availability of the emotional energy required to sustain that velocity by all involved” - and then n being added to the date of that meeting); and then those dates have been missed because, for example, we require additional FC ports in order to plug in our development array and the vendor arbitrarily cancelled our order and didn’t tell us about it. That kind of thing. I work hard on it, at a rate that I consider to be sustainable - I’ve already managed to get to the point where I looked like a corpse and couldn’t focus my eyes whilst fixing the mess that the previous kit had put us in. It’s about expectation management. So stuff slips.

So, I’ve repeatedly resisted that pressure. Why?

First, I should point out that there is no difference in what we do now compared to what we would do with a system that is production - providing nothing goes wrong. The difference happens when something does go wrong.

What would happen now is that we would run ourselves ragged making stuff up on the fly, recovering the system as quickly as possible, but basically choosing our path on the basis of our best expectation (which would be reasonable except that empirically, we’ve come to understand that Windows clustering seldom meets our best expectations).

In a production system, we would have already simulated that problem, developed and practised the recovery process, and have it documented and understood by at least two people.

That’s the difference.

Now, in a recent meeting, I was challenged with this:

If we had held any of our current production systems up to the same standards of delivery, they wouldn’t be in production.

That may be true, but it isn’t a reason to cut corners and produce crap. It’s a comment on the other services that leak into production, not the state of the DFS.

Why is this the case?

• A lack of project management. We lack well-defined milestones. We’re not in the habit of setting them. Instead, our teams tend to operate in silos, interrupt-driven, incrementally doing development work when the stoking of production systems isn’t in the way. We don’t run clean iterations with clean, measurable, achievable milestones.
• A lack of project teams. We’re not geared up to fix the first problem because of the way that our department is organised. People live under fixed organisational structures. It means that sorting out the logistics (find a DBA; find a sysadmin; find some kit; find a project manager; etc) is difficult because you are naturally trying to squeeze attention out of a small number of vital people who don’t directly live in the same organisational branch that you do.
• A lack of capability to fix the above. The kind of short-term, well-defined, project-related work needs effort from middle-management. It needs a willingness to devote people to well-identified pieces of work for a fixed period, to let them get on. It needs up-front planning and project-management skills. It needs a bit of vision and a bit of courage.

So that’s where we are and what I think is wrong. We’re departmentally in a rut. It’s the role of the departmental directors to sort this out. I’m not sure it’s perceived as a problem; but it’s certainly the case that we could use people better, in more varied ways, and identify the key resource shortages if we were to stop spreading those key people so thinly. We should be giving everyone a more varied, more rewarding experience at work.

And if, when it comes to resourcing a project milestone, 60 people are left in the room because the key people have already been earmarked for working on a particular thing that month: well, then that’s a result, not a failure. Better to identify that problem than to settle for a working practice that permits you to ignore it.

The good news is that entropy is letting me fill in addition sections in my list of “what to do when part X craps out” anyway: we had a fairly hard switch failure the last week and that went swimmingly well; although obviously the nodes that lost paths to the array behind it needed a reboot to find them again afterwards.

To set up the initial mirror of the source/root ZFS to the dest/root ZFS:

# zfs snapshot -r source/root@0001
# zfs send source/root@0001 |
zfs receive -d dest

This’ll create a dest/root and a dest/root@0001 snapshot.

To update the mirror:

# zfs snapshot -r source/root@0002
# zfs send -i 0001 source/root@0002 |
zfs receive -Fd dest

This sends an incremental set of changes and applies them to the dest/root snapshot. The -F first rolls back dest/root to the latest snapshot - resetting any local changes (including atimes). The result of this is dest/root with two snapshots - the latest, 0002, and the first, 0001.

We can remove the earlier snapshots once the mirror is complete:

# zfs destroy -r dest/root@0001
# zfs destroy -r source/root@0001

This leaves just the latest mirror snapshot available at both ends, permitting the next incremental to run using the same pattern.

Note: there is no recursive option for zfs send; where there are multiple dependent filesystems under source/root, we need to send them one after the other. Assuming the initial filesystem transfer has been done, that looks like this:

# zfs snapshot -r source/root@0003
# zfs send -i 0002 source/root@0003 |
zfs receive -Fd dest
# zfs send -i 0002 source/root/x@0003 |
zfs receive -Fd dest
# zfs send -i 0002 source/root/y@0003 |
zfs receive -Fd dest

The recursive flag on zfs destroy takes care of all the dependent snapshots in one go.

I just imported the bb-archive.isys zpool onto the first host, then rebooted it. After it had shut down, I forcibly imported the zpool onto the second host. Now waiting for the first to come back up. It should, perhaps, complain that the zpool is owned by someone else, but should not do a forcible reimport…

And alas, that’s not what happens. So:

I’m going to use altroots for all zpools. Unfortunately, this forces the subdirs to appear only mounted under the altroot. Not quite the combination I was after.

Solution: specify the service IP address explicitly together with a bunch of paths under the networker configuration.

zpool status etc. will show the MPxIO devices that comprise the particular zpool in question - but what about the other seven LUNs? And what about when it comes to stiching LUNs together, and so on? This is going to turn into a cross-referencing nightmare!

OK, so the SataBeasts have fairly well-known IDs (their WWNs) and we can start there. Each LUN exposed by a SataBeast gets a randomly-allocated volume ID; both of these parts are visible within the derived device name that MPxio presents.

However, we typically plan and work using LUNs, rather than those volume IDs. So it’d be nice to at least get a way to dump that information.

That information is exposed in a number of places - prtconf will expose it, for instance - but it’s probably simpler to use fcinfo, as follows:

$ which hbas
hbas () {
        fcinfo hba-port | awk ‘/^HBA Port WWN:/ {print $4}’
}

And to list the LUNs, we can use this:

$ which luns
luns() {
        printf ‘HBA\t\t\trPort\t\t\tLUN\tPath\n’
        while [[ $# != 0 ]]
        do
                fcinfo remote-port -p “$1″ -s |
                awk ‘
                        BEGIN { hba = “‘”$1″‘” }
                        /^Remote Port WWN:/ { rport = $4 }
                        /LUN:/ { lun = $2 }
                        /OS Device Name:/ {
                                print hba “\t” rport “\t” lun “\t” $4
                        }
                ‘
                shift
        done
}

Then:

# luns $( hbas )

provides what we’re after in a tabular form.

… and so on :-)

Order went into Redstor last night at 4pm. They were here this morning by 9am.

Spot the difference.

There was a tentative plan to SAN-attach the MBS. I’d much rather supply its disk off NFS, which is must faster to find a replacement host for. My plan is to rebuild the MBS setup on hereford (currently a farm host on PRE) and punt defiant back into the farm.

We’ve seen some issues in keeping a pair of MBS DB instances in sync; I’m looking forward to using some of my upcoming copious free time to do some fairly hefty development work on the MBS side (which is long overdue). In order to make this work smoothly and to help with patch management, I’m planning on making a change to the way the MBS is laid out, as follows:

• all MBS checkouts will go directly onto NFS-supplied disk. We’ll also be supplying home dirs from the NFS cluster so that we aren’t in any danger of running out of elbow-room.
• the mbs/src, mbs/zfs, mbs/apache and other “common infrastructure” parts will live under one repo, laid out as follows:

  mbs.src/trunk HEAD, development happens here (or on dev branches)
  mbs.src/branches/mbs.pre HEAD pushed out to here
  mbs.src/branches/mbs.prod …and here
  mbs.src/branches/... …and so on.

  The mbs.src repo will not only contain the code for deployment methods; it’ll also have a skeletal mbs/db that contains the core artifacts common to all database instances: the platform-deployment, bootloader, and os images. These can then be maintained from a single spot.

  Note: in fact I think I might further split up mbs.src and mbs.db into two repos, and put the large OS images under the latter, out of the way of the deployment method implementations.

• all MBS database instances will then be built as separate repos. Each repo will contain svn:externals pointers to the mbs.src:src/, mbs.db:db/image, etc, subdirectories that should be centrally maintained. I may well punt software images (apache, tomcat, etc) into this too.
• This shouldn’t make the blindest bit of difference to any of the scripts, with the exception that the apache users.xml and groups.xml will need to be moved out of those particular subdirectories.
• Checkouts: finally - I’m strongly tempted to run each MBS instance under its own “zone” using OpenVZ. There have been some issues with the PXE platform-deployment instances getting in the way of each other; this’ll let me split them up cleanly.

Schedule: Initially, I said that once the NFS service was bedded down I’d begin this work, andexpect it to run over the summer. At some point there’d be a cutover as the new MBS comes into service.

Actually, I’m going to start prepping this now. This’ll mean cutting a new pair of repos for mbs.src and migrating the existing history into it.

Once that it done I’ll put a freeze on the src/ and other non-db trees whilst I rejig the existing repositories. I expect that the tidy-up work there will take a day or so to sort out, some time in the next couple of weeks.

I’ll also be rolling out a third (count ‘em!) MBS DB instance for the mail system stuff. Initially this’ll be for the spam boxes, although we might look at managing the webmail through this too, since it looks eminently amenable to that treatment.