https://svn.cse.bris.ac.uk/svn/jan/trunk/calendar/

This represents a bit of a rejigging of the oracle_calendar.py to give Events the ability to turn themselves into iCalendar-style records. They also pick up vAlarms.

The Event output isn’t (yet) complete; most importantly, attendees aren’t re-serialized. However, it suffices to be able to provide a one-way output to an unjailbroken iPhone – or anything else that can consume the .ics format.

This is coupled with a little flup-based AJP service. I originally used the flup fcgi WSGIServer instead; however, I was having trouble configuring it with Apache 2.2 and mod_fastcgi (I had multiple FastCGI services, only some of which I wanted to pass the HTTP_AUTHORIZATION through to. Looks like mod_fcgid manages all this a little better) so I wound up using the AJP connector instead – without a hitch.

The service requires HTTP-Basic authentication, which it passes through sight unseen to the Oracle Calendar “soap” service.

I’ve probably complained about that before now. It’s not really a SOAP service. It’s a mostly-incomplete access point that will let a user see only their own calendar (no proxy or delegate authentication options, no way to choose another visible calendar) that accepts a precisely-formatted sublanguage of valid SOAP requests; mostly by dint of being half-implemented with a hand-written parser, as far as I can tell. The parser, amongst other things, is extremely sensitive to having namespace declarations in the right place with the right prefixes, and so on. Oh, and if you see an empty calendar (and your server’s throwing expat exceptions) that’s because Oracle calendar won’t always return well-formed XML. Mmm, taste the quality.

Anyway, this is enough to get simple calendar details slurpable through to an external calendar on the iPhone. It’s set up as follows:

<Proxy balancer://fcgi>
    BalancerMember ajp://cmjg.localhost:8642
</Proxy>

and then

    ProxyPass /some/url/path balancer://fcgi/some/url/path
    <Location /some/url/path>
       Allow from all
    </Location>

The calendar.fcgi is launched externally to Apache.

I’ve run one of these up for testing and it suffices for an aide memoire; the iPhone turns vAlarms (of any kind, the ACTION:EMAIL ones that Oracle produces in particular) into bleeps and dings.

Two nodes, alike in dignity. We preconfigured MPxIO on them prior to installing the cluster software. We also allocated a small LUN that was visible to both nodes prior to the installation, and made sure they could see it: this was intended to act as a quorum device.

That just worked. Did the install on the first node (moose); it allocated a DID to the multipathed device, no problems. Bring in the second node (moron) and scinstall wound up identifying the to-be-quorum device as the same DID.

Finished the setup using scsetup to drop out of installmode and we were done. Next steps.

We zoned up the two nodes to be able to see our SATABeast pair that the first data LUNs would be coming off. (Note: I’d prefer that the quorum device actually hold some data too, but we were getting started first.)

We allocated a single LUN and presented that to moose only. On that host:

1. Use fcinfo to scan for the new LUNs and get their multipathed device nodes created; /nfs/bin/luns is a handy script that does this for you.
2. Use cldev refresh to allocate a new DID.

Fine so far. At that point (we were going a step at a time to check this) we LUN masked moron to be able to see the new LUN too, and repeated the process there. The existing DID now showed its availability on both nodes.

This worked so smoothly that in future I think we could simply mask all nodes to see the LUN, run /nfs/bin/luns on each node, then use cldev refresh all in one go.

Zope (modern zope) comes with a configured object that lets your proxy supply alternative server host and port settings. The following sufficies in /etc/httpd/conf.d/zenoss-proxy.conf:

Order deny,allow
Deny from all
Allow from 137.222.12.
Allow from 127.0.0.1
ProxyPass http://127.0.0.1:8080/VirtualHostBase/http/hita.cse.bris.ac.uk:80/VirtualHostRoot/

# /sbin/lspci

04:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
04:00.1 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
05:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
05:00.1 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)

Two dual-port cards, effectively.

Kernel module for these:

# /sbin/lsmod | grep ql
qla2xxx 1079969 0
scsi_transport_fc 73673 1 qla2xxx
scsi_mod 188665 7
sg,qla2xxx,scsi_transport_fc,mptsas,mptscsih,scsi_transport_sas,sd_mod

qla2xxx, already loaded.

To find WWNs, on RHEL 5.x (centos 5.x),

WWN information and other FC stuff is under
/sys/class/scsi_host/hostN/device/fchost:hostN/port_name
for values of N.

# ls /sys/class/scsi_host/host*/device/fc_host*/port_name
/sys/class/scsi_host/host1/device/fc_host:host1/port_name
/sys/class/scsi_host/host2/device/fc_host:host2/port_name
/sys/class/scsi_host/host3/device/fc_host:host3/port_name
/sys/class/scsi_host/host4/device/fc_host:host4/port_name

(there’s a host0 on this box which is the onboard SAS controller)

# cat /sys/class/scsi_host/host*/device/fc_host*/port_name
0x2100001bxxxxxxxx
0x2101001bxxxxxxxx
0x2100001dxxxxxxxx
0x2101001dxxxxxxxx

So these are the addresses that need zoning.

Scanning for LUNs:

# echo "- - -" > /sys/class/scsi_host/host1/scan
[wait a bit]
# more /proc/scsi/scsi
[new paths to luns show up, mpaths appear under /dev/mapper if that's configured]

After scanning for luns (so they show up in /proc/scsi/scsi)…

- /sbin/chkconfig multipathd on
- edit /etc/multipath.conf to look like the one on bastet. I made the following changes:

blacklist {
devnode "*"
}
blacklist_exceptions {
devnode "^sd[b-z].*"
}

(since sda was always the local SAS root device) and modified the defaults {} section to use “failover” rather than “multipath”, “multibus” etc, which appears to be fine.

After you’ve done this you should be able to do the following:

/sbin/multipath -v2 -d

-d is for dry-run (make no changes). It’ll tell you that it’ll make mpath0, mpath1, mpathxxx, tell you the ID of that volume (a long hex string) and what paths are available to it, what SCSI vectors those use, etc.

You can then add a multipath{} section to /etc/multipath.conf which lists that WWID and gives it an alias (ie, doesn’t use “mpath0″ etc) – we used “sb2cc-lun5″ as an example alias.

Start multipathd if it’s not running: /etc/init.d/multipathd start and (with a possible /sbin/multipath -v2 to search for new paths and make them available or to rescan multipath.conf) you’ll find /dev/mapper/sb2cc-lun5, etc, as new block devices.

We labelled these with e2label, made an ext3 filesystem (takes several minutes for a multi-TB filesystem) on there and put entries into /etc/fstab as follows:

/dev/mapper/sb2cc-lun5 /sb2cc-lun5 ext3 defaults 1 3

You probably don’t want to use ext3 for a large filesystem.

These come back happily AND IN A STABLE FASHION on a reboot, which is just as well because the raw “path” devices, sdb..sdi were juggled on the reboot – this is just down to how fast the various HBA scans come back.

- Listing current multipath settings: multipath -v2 -l
- Listing what would be changed (no changes made, dry run): multipath -v2 -d
- Making those changes live: multipath -v2

Getting rind of unwanted LUNs:

# /sbin/multipath -v3 -d
[nothing changes, need to rescan]
# echo "- - -" > /sys/class/scsi_host/host1/scan
[wait a bit]
# more /proc/scsi/scsi
[new paths to luns show up, mpaths appear under /dev/mapper

At that point I noticed that the backup host group on sb3cc has access to some stuff it shouldn't; turned that off. Now need to flush the paths to the LUNs that've disappeared:

# /sbin/multipath -F
# /sbin/multipath -v2 -l

only two paths show, scan for the LUNs on the other controller...
# echo "- - -" > /sys/class/scsi_host/host2/scan

Now you can create the multipath devices.
# /sbin/multipath -v2
# /sbin/multipath -v2 -l

two large LUNs show up...

mpath6 (36000402001fc475761ee919c00000000) dm-5 NEXSAN,SATABeast
[size=6.4T][features=0][hwhandler=0]
^^^ this is sb3mvb (...4757 in WWID)

mpath5 (36000402001fc46db60ef903200000000) dm-2 NEXSAN,SATABeast
[size=6.4T][features=0][hwhandler=0]
^^^ this is SB3cc (....46db in WWID)

At that point you I edit /etc/multipath.conf to give these useful device names, and run /sbin/multipath -v2.

This is exactly not what I’m after. Apparently this is fixable by adding

__allow_access_to_unprotected_subobjects__ = 1

to the class definition. It’s irritating to have to modify external methods in order to make them Zope-ready; at least the fix is (apparently) a one-liner.

Mind you, there is also a bunch of lack in the Oracle calendar SOAP interface: no WSDL, no querying of delegate calendars, a limited SOAP implementation that appears to have been written in terms of SAX events and that’s particularly sensitive to namespace declarations, and so on.

Anyway: I got enough of this working that I can query a user’s calendar in a fairly Pythonic fashion – https://svn.cse.bris.ac.uk/svn/jan/trunk/calendar/ for details. It’ll suffice to reimplement the guts of Shirley, the ILRT’s automatic receptionist.

This also marked the first time I’ve tried using new-style classes to augment the Python str class with a secondary attribute.

Having tested this tonight, we can indeed pull out a list of ilrt-visitor meetings, together with their organiser’s details (mailto and cn).

our discipline has a tried and tested way of going faster. Cut corners, cut quality. That way you can produce more crap.

So, how does this relate to the DFS?

Well, I’m coming under pressure to declare that the DFS is production-ready. This isn’t a technical thing, this is purely one of PR. Where does this pressure come from? From “on high” – ie, from someone who is one hop away from seeing anything other than that we have a working cluster, what’s holding things up? (Bob’s actually pretty reasonable about this – he’s stuck between a rock and an opinionated git.)

There have been unavoidable supplier and technical delays involved in getting this far. The trouble is that dates have been randomly selected on no basis whatsoever (actually, on the basis of having a meeting and me saying, “it will take n days of uninterrupted work by all involved with nothing else getting in the way, assuming no impediments, no unforseen hitches, and the continued availability of the emotional energy required to sustain that velocity by all involved” – and then n being added to the date of that meeting); and then those dates have been missed because, for example, we require additional FC ports in order to plug in our development array and the vendor arbitrarily cancelled our order and didn’t tell us about it. That kind of thing. I work hard on it, at a rate that I consider to be sustainable – I’ve already managed to get to the point where I looked like a corpse and couldn’t focus my eyes whilst fixing the mess that the previous kit had put us in. It’s about expectation management. So stuff slips.

So, I’ve repeatedly resisted that pressure. Why?

First, I should point out that there is no difference in what we do now compared to what we would do with a system that is production – providing nothing goes wrong. The difference happens when something does go wrong.

What would happen now is that we would run ourselves ragged making stuff up on the fly, recovering the system as quickly as possible, but basically choosing our path on the basis of our best expectation (which would be reasonable except that empirically, we’ve come to understand that Windows clustering seldom meets our best expectations).

In a production system, we would have already simulated that problem, developed and practised the recovery process, and have it documented and understood by at least two people.

That’s the difference.

Now, in a recent meeting, I was challenged with this:

If we had held any of our current production systems up to the same standards of delivery, they wouldn’t be in production.

That may be true, but it isn’t a reason to cut corners and produce crap. It’s a comment on the other services that leak into production, not the state of the DFS.

Why is this the case?

• A lack of project management. We lack well-defined milestones. We’re not in the habit of setting them. Instead, our teams tend to operate in silos, interrupt-driven, incrementally doing development work when the stoking of production systems isn’t in the way. We don’t run clean iterations with clean, measurable, achievable milestones.
• A lack of project teams. We’re not geared up to fix the first problem because of the way that our department is organised. People live under fixed organisational structures. It means that sorting out the logistics (find a DBA; find a sysadmin; find some kit; find a project manager; etc) is difficult because you are naturally trying to squeeze attention out of a small number of vital people who don’t directly live in the same organisational branch that you do.
• A lack of capability to fix the above. The kind of short-term, well-defined, project-related work needs effort from middle-management. It needs a willingness to devote people to well-identified pieces of work for a fixed period, to let them get on. It needs up-front planning and project-management skills. It needs a bit of vision and a bit of courage.

So that’s where we are and what I think is wrong. We’re departmentally in a rut. It’s the role of the departmental directors to sort this out. I’m not sure it’s perceived as a problem; but it’s certainly the case that we could use people better, in more varied ways, and identify the key resource shortages if we were to stop spreading those key people so thinly. We should be giving everyone a more varied, more rewarding experience at work.

And if, when it comes to resourcing a project milestone, 60 people are left in the room because the key people have already been earmarked for working on a particular thing that month: well, then that’s a result, not a failure. Better to identify that problem than to settle for a working practice that permits you to ignore it.

The good news is that entropy is letting me fill in addition sections in my list of “what to do when part X craps out” anyway: we had a fairly hard switch failure the last week and that went swimmingly well; although obviously the nodes that lost paths to the array behind it needed a reboot to find them again afterwards.

To set up the initial mirror of the source/root ZFS to the dest/root ZFS:

# zfs snapshot -r source/root@0001
# zfs send source/root@0001 |
zfs receive -d dest

This’ll create a dest/root and a dest/root@0001 snapshot.

To update the mirror:

# zfs snapshot -r source/root@0002
# zfs send -i 0001 source/root@0002 |
zfs receive -Fd dest

This sends an incremental set of changes and applies them to the dest/root snapshot. The -F first rolls back dest/root to the latest snapshot – resetting any local changes (including atimes). The result of this is dest/root with two snapshots – the latest, 0002, and the first, 0001.

We can remove the earlier snapshots once the mirror is complete:

# zfs destroy -r dest/root@0001
# zfs destroy -r source/root@0001

This leaves just the latest mirror snapshot available at both ends, permitting the next incremental to run using the same pattern.

Note: there is no recursive option for zfs send; where there are multiple dependent filesystems under source/root, we need to send them one after the other. Assuming the initial filesystem transfer has been done, that looks like this:

# zfs snapshot -r source/root@0003
# zfs send -i 0002 source/root@0003 |
zfs receive -Fd dest
# zfs send -i 0002 source/root/x@0003 |
zfs receive -Fd dest
# zfs send -i 0002 source/root/y@0003 |
zfs receive -Fd dest

The recursive flag on zfs destroy takes care of all the dependent snapshots in one go.

I just imported the bb-archive.isys zpool onto the first host, then rebooted it. After it had shut down, I forcibly imported the zpool onto the second host. Now waiting for the first to come back up. It should, perhaps, complain that the zpool is owned by someone else, but should not do a forcible reimport…

And alas, that’s not what happens. So:

I’m going to use altroots for all zpools. Unfortunately, this forces the subdirs to appear only mounted under the altroot. Not quite the combination I was after.