Site license! – there are more users for this than you expect. (Note, I’m not ruling out open source here.)
Email from “Everybody” – no preregistration. Eg, HPC, bb-help.
Configuration and integration: extensible API, no proprietary little languages (where are my interface libraries to talk to MQ, etc, in the little language, if needed?)
Clever UI widgets.
BOOKMARKABLE URLs!

I use RT as the concrete example of a ticketing system, but if you are (rationally or irrationally) turned off it for some reason, replace it wih ANY lightweight ticketing tool that supports a rich set of inter-ticket relationships.

I’ll start by listing some of the problems the department has (if we didn’t have problems, there would be no need for an organisational rethink), then describe how one might utilise a ticketing system to help, and finally outline the benefits.

Problems (as outlined to Tim yesterday)

Ok, so I’ve general complaints as follows (collected from talking to various people and summarising some of our issues):

• PRINCE 2 gives us large, monolithic projects. Those projects have deliverables, some of which are concrete software objects (although policies might be another output, and documentation, and so on: the important thing is to consider a deliverable as a concrete object).

  There is no simple way of really telling what a project’s up to, and in particular, if Project A deliverable X depends critically on something that might be coming out of Project C, that’s not made formally visible.

• There are also lighter-weight things: stuff individuals or teams are trying to achieve, development work, long-range planning, and so on. Again, we can’t really tell at all what’s going on. We talk about the SPR being an opportunity to remove the duplication of effort going on out in departments, but (for a number of reasons that will derail this post if I go into them, that’s for another rant) we have a – let’s say, a nonzero degree – of wheel reinvention within our department.
• Management meetings are opaque and not particularly collegiate in their organisation. It’s not clear, perhaps, what the purpose of those meetings is and I’ve certainly had people (attendees) snort in derision when I suggested that it was an appropriate avenue to raise things: for passing upstream recommendations and stuff that needed expediting – that needed attention and some general “yes, this is good” go-aheads.

  If the attendees of the management meetings themselves don’t know why they’re there, there’s something wrong! Note, I don’t think that cutting down on the number of people in attendance is what’s needed. What’s needed are focussed, collegiate (minuted!) meetings where problems can be raised and short-, mid- and long-term planning done effectively.

• Inter-team working happens, but typically it’s by direct collaboration (pestering) of useful technical folk by each other. Individual schedules are hard to work out and it’s never actually clear when anyone will be available to get work done, or to figure out what dates are likely to be for constituent dependencies of a piece of work.
• Our communication, both internally and externally, could stand a great deal of improvement. We should work in the open. This would have positive effects on morale as much as anything else.

And so on. Right, that’s the “motivating” preamble; doom and gloom over.

Using RT (or similar) to effect a measurable approach

Let’s say we adopt a tool like RT. (Not necessarily that one, I name it because it can pretty directly be used to do this and we’ve one in place if you want me to show you what I’m talking about.)

For this to work, all projects, PRINCE 2 or otherwise, need turning into (or registering as) a series of RT tickets (not just one ticket). RT has some complex notions of ticket interrelationships: child of, dependency, blocking on, etc.

In the case of a large prince 2 project, or a JISC project (assuming JISC is still here in 3 months’ time) the project itself might turn into a ticket; the project strands into child tickets; the milestones and deliverables into individual tickets. Each subordinate ticket would become a child of its parent (milestones and deliverables, strands, projects).

Example: we know that any sane IdM project is going to produce some kind of group management facility. That turns into a ticket, the closure of which would represent the delivery of that object.

Why is this useful?

Well, for a start, when someone is working on something, it might not be completely planned out in advance. That’s actually pretty typical. But they may discover (or realise in advance) that they need, say, a group management facility. Some namespace mapping tools. A directory service. A schema in a directory service. And so on.

If they are planning on producing those items themselves, they can register them within RT. Turning them into tickets, effectively is making a series of statements that these things need doing or will be done. They can mark dependencies between those items.

More importantly, some of those items might later pick up ownership, and be turned into projects. Or be specific deliverables of a current, long-running project, that are still 18 months out at least. And so on.

It doesn’t even matter if someone looks for, but fails to find, a ticket that already stands as the promise of a particular object, because RT permits the post-hoc merging of tickets into one, and dependency links are updated appropriately. So if I’m doing something that needs an X, and I can’t find any information about that, I might create a ticket describing X briefly and what I need it to do. When someone else comes along and reviews that ticket, they might realise it’s part of project A they’re involved in. They can merge those tickets. The resulting merged ticket will still contain my comments, so a bit of useful requirements gathering for the detail of X is available to project A.

How this helps

Critically, RT offers a single, unified space where these deliverables can be organised and the relationships between them spelled out. It’s not the only tool that does this; it’s just one you can look at now to see how this might be made to work.

Furthermore, whilst particular projects can retain their groupings and allocated staff, project queues etc, dependencies between strands of work can also be highlighted. Tickets don’t have to be on the same queue to be related.

Most importantly: this is a machine-processable representation. RT can already draw you a dependency graph. It’s possible to cleanly extract this information and say:

“you want some NFS space? Sure. It depends on the NFS service, which depends on posix namespace mappings, which depend on group management, which depends on the IdM toolkit. We can do that for you in 18 months’ time, give-or-take; or if it’s important, we can escalate that strand of the IdM project”

(Again, I use a concrete example to indicate how this would work. Workarounds and cheap hacks are also representable within this system, just as easily as formal deliverables, and their existence is also formally recorded.)

What this gives you

• A dependency graph that encapsulates everything you’re trying to do. (For this to work the tool needs to be lightweight in operation and capable of handling that number of tickets.)
• Critical paths. A way of taking that information and actually having a sound engineering (I was gonna say “scientific” but it’s not) basis for deciding what our priorities are, in terms of impact, average delays, etc, rather than which project owner shouts the loudest. This will inform management decisions, although I’d hesitate to say their function can be completely replaced by some simple discrete mathematics :-)
• Something concrete to base management meetings around.

It’s machine-readable. It’s processable. You can actually SEE when things were asked for, when they were delivered, why they took that long to deliver, and so on. It’s measurable. You have things that you can look at that tell you how often the cross-team work is stalling.

It’s not perfect; it needs people to work. But it’s the kind of thing I’m talking about when I say “we need tool support”. Hopefully this makes some kind of sense!

(Note: you don’t automatically get this from ITIL, although it’s related. What ITIL will provide are some processes and tools; how you profile those processes, how you use those tools, determines whether you’re an effective organisation or not. ITIL can help structure some of those decisions, but it’s definitely not a panacea.)

As a footnote, please understand that when I talk about getting measurability out of a ticketing system, I’m talking about this kind of dependency tracking, recording start and due dates, etc. There is no mention above about reporting on time taken by an individual staff member, number of tickets closed, etc, and with good reason: the notion is inhumane, it ignores that fact that tickets represent both short- and long-term deliverables of vastly varying size, and nobody around here is foolish enough to think it’s a good idea, fortunately! The goal is to support our staff, not produce a stick to beat them with.

https://svn.cse.bris.ac.uk/svn/jan/trunk/calendar/

This represents a bit of a rejigging of the oracle_calendar.py to give Events the ability to turn themselves into iCalendar-style records. They also pick up vAlarms.

The Event output isn’t (yet) complete; most importantly, attendees aren’t re-serialized. However, it suffices to be able to provide a one-way output to an unjailbroken iPhone – or anything else that can consume the .ics format.

This is coupled with a little flup-based AJP service. I originally used the flup fcgi WSGIServer instead; however, I was having trouble configuring it with Apache 2.2 and mod_fastcgi (I had multiple FastCGI services, only some of which I wanted to pass the HTTP_AUTHORIZATION through to. Looks like mod_fcgid manages all this a little better) so I wound up using the AJP connector instead – without a hitch.

The service requires HTTP-Basic authentication, which it passes through sight unseen to the Oracle Calendar “soap” service.

I’ve probably complained about that before now. It’s not really a SOAP service. It’s a mostly-incomplete access point that will let a user see only their own calendar (no proxy or delegate authentication options, no way to choose another visible calendar) that accepts a precisely-formatted sublanguage of valid SOAP requests; mostly by dint of being half-implemented with a hand-written parser, as far as I can tell. The parser, amongst other things, is extremely sensitive to having namespace declarations in the right place with the right prefixes, and so on. Oh, and if you see an empty calendar (and your server’s throwing expat exceptions) that’s because Oracle calendar won’t always return well-formed XML. Mmm, taste the quality.

Anyway, this is enough to get simple calendar details slurpable through to an external calendar on the iPhone. It’s set up as follows:

<Proxy balancer://fcgi>
    BalancerMember ajp://cmjg.localhost:8642
</Proxy>

and then

    ProxyPass /some/url/path balancer://fcgi/some/url/path
    <Location /some/url/path>
       Allow from all
    </Location>

The calendar.fcgi is launched externally to Apache.

I’ve run one of these up for testing and it suffices for an aide memoire; the iPhone turns vAlarms (of any kind, the ACTION:EMAIL ones that Oracle produces in particular) into bleeps and dings.

Two nodes, alike in dignity. We preconfigured MPxIO on them prior to installing the cluster software. We also allocated a small LUN that was visible to both nodes prior to the installation, and made sure they could see it: this was intended to act as a quorum device.

That just worked. Did the install on the first node (moose); it allocated a DID to the multipathed device, no problems. Bring in the second node (moron) and scinstall wound up identifying the to-be-quorum device as the same DID.

Finished the setup using scsetup to drop out of installmode and we were done. Next steps.

We zoned up the two nodes to be able to see our SATABeast pair that the first data LUNs would be coming off. (Note: I’d prefer that the quorum device actually hold some data too, but we were getting started first.)

We allocated a single LUN and presented that to moose only. On that host:

1. Use fcinfo to scan for the new LUNs and get their multipathed device nodes created; /nfs/bin/luns is a handy script that does this for you.
2. Use cldev refresh to allocate a new DID.

Fine so far. At that point (we were going a step at a time to check this) we LUN masked moron to be able to see the new LUN too, and repeated the process there. The existing DID now showed its availability on both nodes.

This worked so smoothly that in future I think we could simply mask all nodes to see the LUN, run /nfs/bin/luns on each node, then use cldev refresh all in one go.

Zope (modern zope) comes with a configured object that lets your proxy supply alternative server host and port settings. The following sufficies in /etc/httpd/conf.d/zenoss-proxy.conf:

Order deny,allow
Deny from all
Allow from 137.222.12.
Allow from 127.0.0.1
ProxyPass http://127.0.0.1:8080/VirtualHostBase/http/hita.cse.bris.ac.uk:80/VirtualHostRoot/

# /sbin/lspci

04:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
04:00.1 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
05:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
05:00.1 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)

Two dual-port cards, effectively.

Kernel module for these:

# /sbin/lsmod | grep ql
qla2xxx 1079969 0
scsi_transport_fc 73673 1 qla2xxx
scsi_mod 188665 7
sg,qla2xxx,scsi_transport_fc,mptsas,mptscsih,scsi_transport_sas,sd_mod

qla2xxx, already loaded.

To find WWNs, on RHEL 5.x (centos 5.x),

WWN information and other FC stuff is under
/sys/class/scsi_host/hostN/device/fchost:hostN/port_name
for values of N.

# ls /sys/class/scsi_host/host*/device/fc_host*/port_name
/sys/class/scsi_host/host1/device/fc_host:host1/port_name
/sys/class/scsi_host/host2/device/fc_host:host2/port_name
/sys/class/scsi_host/host3/device/fc_host:host3/port_name
/sys/class/scsi_host/host4/device/fc_host:host4/port_name

(there’s a host0 on this box which is the onboard SAS controller)

# cat /sys/class/scsi_host/host*/device/fc_host*/port_name
0x2100001bxxxxxxxx
0x2101001bxxxxxxxx
0x2100001dxxxxxxxx
0x2101001dxxxxxxxx

So these are the addresses that need zoning.

Scanning for LUNs:

# echo "- - -" > /sys/class/scsi_host/host1/scan
[wait a bit]
# more /proc/scsi/scsi
[new paths to luns show up, mpaths appear under /dev/mapper if that's configured]

After scanning for luns (so they show up in /proc/scsi/scsi)…

- /sbin/chkconfig multipathd on
- edit /etc/multipath.conf to look like the one on bastet. I made the following changes:

blacklist {
devnode "*"
}
blacklist_exceptions {
devnode "^sd[b-z].*"
}

(since sda was always the local SAS root device) and modified the defaults {} section to use “failover” rather than “multipath”, “multibus” etc, which appears to be fine.

After you’ve done this you should be able to do the following:

/sbin/multipath -v2 -d

-d is for dry-run (make no changes). It’ll tell you that it’ll make mpath0, mpath1, mpathxxx, tell you the ID of that volume (a long hex string) and what paths are available to it, what SCSI vectors those use, etc.

You can then add a multipath{} section to /etc/multipath.conf which lists that WWID and gives it an alias (ie, doesn’t use “mpath0″ etc) – we used “sb2cc-lun5″ as an example alias.

Start multipathd if it’s not running: /etc/init.d/multipathd start and (with a possible /sbin/multipath -v2 to search for new paths and make them available or to rescan multipath.conf) you’ll find /dev/mapper/sb2cc-lun5, etc, as new block devices.

We labelled these with e2label, made an ext3 filesystem (takes several minutes for a multi-TB filesystem) on there and put entries into /etc/fstab as follows:

/dev/mapper/sb2cc-lun5 /sb2cc-lun5 ext3 defaults 1 3

You probably don’t want to use ext3 for a large filesystem.

These come back happily AND IN A STABLE FASHION on a reboot, which is just as well because the raw “path” devices, sdb..sdi were juggled on the reboot – this is just down to how fast the various HBA scans come back.

- Listing current multipath settings: multipath -v2 -l
- Listing what would be changed (no changes made, dry run): multipath -v2 -d
- Making those changes live: multipath -v2

Getting rind of unwanted LUNs:

# /sbin/multipath -v3 -d
[nothing changes, need to rescan]
# echo "- - -" > /sys/class/scsi_host/host1/scan
[wait a bit]
# more /proc/scsi/scsi
[new paths to luns show up, mpaths appear under /dev/mapper

At that point I noticed that the backup host group on sb3cc has access to some stuff it shouldn't; turned that off. Now need to flush the paths to the LUNs that've disappeared:

# /sbin/multipath -F
# /sbin/multipath -v2 -l

only two paths show, scan for the LUNs on the other controller...
# echo "- - -" > /sys/class/scsi_host/host2/scan

Now you can create the multipath devices.
# /sbin/multipath -v2
# /sbin/multipath -v2 -l

two large LUNs show up...

mpath6 (36000402001fc475761ee919c00000000) dm-5 NEXSAN,SATABeast
[size=6.4T][features=0][hwhandler=0]
^^^ this is sb3mvb (...4757 in WWID)

mpath5 (36000402001fc46db60ef903200000000) dm-2 NEXSAN,SATABeast
[size=6.4T][features=0][hwhandler=0]
^^^ this is SB3cc (....46db in WWID)

At that point you I edit /etc/multipath.conf to give these useful device names, and run /sbin/multipath -v2.

This is exactly not what I’m after. Apparently this is fixable by adding

__allow_access_to_unprotected_subobjects__ = 1

to the class definition. It’s irritating to have to modify external methods in order to make them Zope-ready; at least the fix is (apparently) a one-liner.

Mind you, there is also a bunch of lack in the Oracle calendar SOAP interface: no WSDL, no querying of delegate calendars, a limited SOAP implementation that appears to have been written in terms of SAX events and that’s particularly sensitive to namespace declarations, and so on.

Anyway: I got enough of this working that I can query a user’s calendar in a fairly Pythonic fashion – https://svn.cse.bris.ac.uk/svn/jan/trunk/calendar/ for details. It’ll suffice to reimplement the guts of Shirley, the ILRT’s automatic receptionist.

This also marked the first time I’ve tried using new-style classes to augment the Python str class with a secondary attribute.

Having tested this tonight, we can indeed pull out a list of ilrt-visitor meetings, together with their organiser’s details (mailto and cn).

our discipline has a tried and tested way of going faster. Cut corners, cut quality. That way you can produce more crap.

So, how does this relate to the DFS?

Well, I’m coming under pressure to declare that the DFS is production-ready. This isn’t a technical thing, this is purely one of PR. Where does this pressure come from? From “on high” – ie, from someone who is one hop away from seeing anything other than that we have a working cluster, what’s holding things up? (Bob’s actually pretty reasonable about this – he’s stuck between a rock and an opinionated git.)

There have been unavoidable supplier and technical delays involved in getting this far. The trouble is that dates have been randomly selected on no basis whatsoever (actually, on the basis of having a meeting and me saying, “it will take n days of uninterrupted work by all involved with nothing else getting in the way, assuming no impediments, no unforseen hitches, and the continued availability of the emotional energy required to sustain that velocity by all involved” – and then n being added to the date of that meeting); and then those dates have been missed because, for example, we require additional FC ports in order to plug in our development array and the vendor arbitrarily cancelled our order and didn’t tell us about it. That kind of thing. I work hard on it, at a rate that I consider to be sustainable – I’ve already managed to get to the point where I looked like a corpse and couldn’t focus my eyes whilst fixing the mess that the previous kit had put us in. It’s about expectation management. So stuff slips.

So, I’ve repeatedly resisted that pressure. Why?

First, I should point out that there is no difference in what we do now compared to what we would do with a system that is production – providing nothing goes wrong. The difference happens when something does go wrong.

What would happen now is that we would run ourselves ragged making stuff up on the fly, recovering the system as quickly as possible, but basically choosing our path on the basis of our best expectation (which would be reasonable except that empirically, we’ve come to understand that Windows clustering seldom meets our best expectations).

In a production system, we would have already simulated that problem, developed and practised the recovery process, and have it documented and understood by at least two people.

That’s the difference.

Now, in a recent meeting, I was challenged with this:

If we had held any of our current production systems up to the same standards of delivery, they wouldn’t be in production.

That may be true, but it isn’t a reason to cut corners and produce crap. It’s a comment on the other services that leak into production, not the state of the DFS.

Why is this the case?

• A lack of project management. We lack well-defined milestones. We’re not in the habit of setting them. Instead, our teams tend to operate in silos, interrupt-driven, incrementally doing development work when the stoking of production systems isn’t in the way. We don’t run clean iterations with clean, measurable, achievable milestones.
• A lack of project teams. We’re not geared up to fix the first problem because of the way that our department is organised. People live under fixed organisational structures. It means that sorting out the logistics (find a DBA; find a sysadmin; find some kit; find a project manager; etc) is difficult because you are naturally trying to squeeze attention out of a small number of vital people who don’t directly live in the same organisational branch that you do.
• A lack of capability to fix the above. The kind of short-term, well-defined, project-related work needs effort from middle-management. It needs a willingness to devote people to well-identified pieces of work for a fixed period, to let them get on. It needs up-front planning and project-management skills. It needs a bit of vision and a bit of courage.

So that’s where we are and what I think is wrong. We’re departmentally in a rut. It’s the role of the departmental directors to sort this out. I’m not sure it’s perceived as a problem; but it’s certainly the case that we could use people better, in more varied ways, and identify the key resource shortages if we were to stop spreading those key people so thinly. We should be giving everyone a more varied, more rewarding experience at work.

And if, when it comes to resourcing a project milestone, 60 people are left in the room because the key people have already been earmarked for working on a particular thing that month: well, then that’s a result, not a failure. Better to identify that problem than to settle for a working practice that permits you to ignore it.

The good news is that entropy is letting me fill in addition sections in my list of “what to do when part X craps out” anyway: we had a fairly hard switch failure the last week and that went swimmingly well; although obviously the nodes that lost paths to the array behind it needed a reboot to find them again afterwards.