Apollo 13

A theatre production is a team effort and it can bring out the best in people. Strenuous efforts where you sprint for the finish line (opening night) are the norm, followed by the relentless feedback to refine the show performance by performance until that last night where you can relax, sit back and enoy your work – whilst mentally steeling yourself for one last burst of effort to take the show appart at the end of the night. It can be all consuming, tiring and pushes people beyond what they normally would consider reasonable in say their normal work or home life. With age comes experience, less youthful energy, knowing when to give in for the night and take a break and when pressing on is injurious to the goal. You have a calibration of how hard you can sprint for how long. I’m immensely proud of some of the shows I’ve worked on and to date in life they are the benchmark by which I’ve been able to measure other things.

Until now that is. This blog is written on Day 25 of Apollo 13. After 24 days of extreme effort I am uncalibrated once more as to how hard I, and indeed the people around me, can sprint. My employer is part of the government’s ventillator challenge in response to Covid-19 and our internal project name is Apollo 13. There are some good parallels – think “Houston we have a problem”, running out of a scarce resource, having to invent solutions from limited components and above all a sprint of human endeavour to do something extreme. Our morning team leads meeting is “Mission Control”. Today is day 25 – an enforced day off and midnight-to-midnight down tools for the whole team. What follows are some incoherent ramblings about our situation and some thoughts I need to unpack.

I time the project from its offical start on Thursday 19th March. The preceding Friday had seen a call with government and our medical technologists had worked the weekend to demonstrate a rapid prototype. Rapid meaning I was able to view the demonstrator video on the Tuesday. I remember over lunch how impressed I was but also how odd it felt not to be part of the response commenting “if the world needed 5G base-stations to fight a global pandemic I’m sure we’d have responded – shame we can’t contribute.” This had rather changed by the Thursday when the job of software verification was hurtling around the company looking for a team lead and a hasty zoom with a rogues gallery of likely suspects convened on the Friday. The best qualified person could not be unplugged from other commitments. Other equally well qualified people were ruled out as the role is incompatible with supporting young children at home (we’d moved to home working that week). Two minutes is not a long time to ponder a major commitment and my glib response to the brief “we need an awkward bastard” may have swung it in my direction. And that was that. Relieved to say I got the person best qualified onto the team to support me – a vital sanity check.

I spent the next hour extricating myself from ongoing commitments. It was timely in one sense – I’d recently delegated most of Technical Authority role on a long-standing project to several people all of whom hit the ground running and I’m pleased to say have proven me superfluous over recent weeks. The other two projects on my plate were doing final reports that week or the following so some loose ends needed dealing with. When the dust settles on Apollo 13 its team will be commended – but so too should be the people who took on extra things to free up people for the team. Everyone I spoke to said “well duh! this out-prioritises, go do, we’ll cope”. Cambridge Consultants prides itself on responding quickly to new projects – but I’ve never seen a team ramp so quickly. By Friday afternoon we were making inroads into our bit of the problem. Nobody even discussed whether we should work the weekend – we just ploughed on.

I’ve never had a sleepless night questioning whether I should take on a project responsibility before. That’s not meant to sound arrogant. Put simply, nothing has mattered before. The worst a signal processing project can normally go wrong is to miss deadlines or go over budget. Class C medical software (translation “get it wrong and you kill people”) is not my comfort zone. At the time it was believed the UK would run out of ventillators in 14 days. The morals of the situation were terrible. Do nothing – people die. Do something too quickly or carelessly – people die. Do it to perfection and you will be late – people die. There were serious unanswered questions from the MHRA as to what concessions against normal product standards would be allowed – even what specification we were aiming for. The critical path is never a great place to be in such circumstance as software was likely to be. The motivation – do something that could save lives – was clear enough and widely understood. I know I struggled with my composure trying to explain our situation as we brought people into the team. How on earth do you describe what commitment is expected from people in such circumstances? How do you set expectations? With hindsight I needn’t have feared – the team around me share an understanding of the situation and everyone is focussed on the goal. It helped to ponder a conversation from years earlier with Simon Long – ask an ex employee about what they worked on at CC and they’ll pour good humoured scorn on the pointless bluetooth widgets and other daft things they worked on – while telling you that however dull/miserable/difficult the project the thing they took pride in was the thing that made a difference in life (in this case I think some sort of kidney dialysis machine). Anway. Sieze the moment and press on. No time to ponder.

The first week was a blur. Think get up at 6am with thoughts racing, ingest the overnight changes and test data, plan the day ahead in time to sync with the team at 9am and escalate any issues to other team leads by 9.30am’s Mission Control. Inevitable calls and followups. 12.30 team meeting to absorb the mornings work and reset the direction. Not uncommon to get to 4pm when working from home and think “Crap. Should have a shower now and maybe some breakfast”. Then work on in to the evening and often the night. Oddly it’s not a technical challenge – the pneumatics, mechanics and software in a ventillator are not ground breaking – the challenge is to do it in a context where every day counts and where what you have to aim for depends on when you can achieve it. We are not resource limited – but there is a fundamental limit to how fast you can align people and solidify a design and plug in people and hardware to help contribute.

Last weekend was surreal. On the Friday we had a project team video conference with David Baker – one of the original Apollo 13 mission control people – a fascinating insight into a very famous moment in history and he had some thought provoking things to say about what people are capable of when they push as a team. Also a good motivator going into the weekend – our product would leave 5am Monday for government test. Day 17 Saturday saw the first assembly of a full product prototype – electronics, mechanics, software, casework the lot and it was a hell of a relief to see it go together and spring into life around midnight. My corner of the world was a close run thing – we were trying to clear four software milestones in a weekend for a 4pm Sunday code freeze and 5pm delivery to system test. Milestone 3 (minimal viable product) was looking good so we were free to enhance test coverage and push for the stretch target milestone 4. All good in theory until a late breaking bug in 3 came to light at 4.36pm, the test framework had hit issues due to the push for enhanced tests and 4 hadn’t been tested at all – with the test cycle at >20 minutes. We had nothing at that moment – and a timely prod from the TA “will it be ready by 5” was met with a bold text *Maybe* as the best certainty we could give. My longest and most intense 24 minutes of concentration of the project followed as the test framework came together and both milestone releases went through test in parallel. We delivered 30 seconds ahead of the deadline. And breathe(!) The system test went ahead and were done by 2am with 3 hours to spare.

It’s hard when you’re pushing for a target and the target moves. The global picture is complicated and we’re about plan D – plans A,B,C involving pushing pre-existing design production harder, scaling e.g. formula 1 companies to make the parts that cannot be sourced for existing designs, ramping production of more basic export products for the application and so on. Making a new design from scratch is not the obvious answer but it can be tailored both to the disease and to the enormously complex supply chain issues – basically what is in stock or can be manufactured in volume in the UK. And there is a lot of sense to backing multiple approaches. Our plan now has a different end date because we’re aiming for something more sophisticated – the clinical feedback is changing in response to what is being learnt. The world is awash with “dead simple” ventillator concepts that will keep someone alive for 24 hours; less so the more sophisticated support needed to nurse someone to recovery over 14 days without doing more harm than good. And the early designs that do get approved canibalise the supply chains for the designs that follow and the designs have to iterate. Another sadness around an end date that moves is that if you’d known it was going to move on day one you would have gone about it differently.

It’s bizarre when people are so focussed to think we don’t have certainty the design will see the light of day. If we are stood down it’s won’t be because we haven’t delivered a viable product in an insane timescale – it will be a competing design getting there first or because the need has receded or because the government’s plans A,B,C have over-delivered. Whilst all these things should be celebrated such an outcome would be gut-wrenchingly hard for such a focussed and comitted project team.

I try and avoid the press. The companies working on this all agreed to avoid publicity; it’s not helpful and adds enormous stress to the people involved. Dyson is a dirty word for breaking ranks and talking rubbish on this front. The government and associated press releases linked above were probably a response to this – although putting it in the public domain did help in enabling people to talk to their families and friends about what they were working on. It’s sad but I’ve been very careful writing this to avoid naming names and companies involved or to release details into the public domain that should not be.

I struggle with social media. People spout forth with complete certainty as to what should be done or what mistakes have been made. Such people are idiots. The situation is far more complex and nuanced than many imagine. On no day has it been clear what the right course of action is. A vigourous but tolerant debate is needed not such simplistic and entrenched positions. I’ll avoid a long rant here but people need to think harder. Lockdown has its costs – in mental health alone to say nothing of social, cultural and economic harm it does. The economic worries should not be off topic – without it the supply chain for health services is unviable and no you can’t easily delineate what is “essential”. The people who blame the government for everything simply mark themselves out as people who would blame the government for everything in all situations regardless. I try not to get stressed arguing with such people.

Life has its other oddities. If I’m in work I can pretty much chose an entire empty floor of an office building to go work in. Mostly I’m there to ransack labs for test equipment and to build remote-access lab rigs. The doctor doing the evaluations was isolating so while you can’t meet him the cameras for the video links are operated by his daughters because he’s not isolating from his family. Working with a sub-team of 12 people only one of whom you’ve worked with before and only two of whom you know would be bizarre enough in normal circumstances – now with video conference and home working for added wierdness. Michael Gove has written to us as “dear friends” confirming we’re key workers. And so on….

There’s humour too. I remember the general channel in teams carrying “Rumours are circulating it’s gloriously sunny outside. This is FAKE NEWS.”

The hypochondria gets the better of you. The worry that you might be ill or worse might make a project team ill who have to deliver in about the length of time the illness knocks you out for is not a happy thought. I have no idea how much I cough occasionally in normal life – because I do that when I’m in perfect health. I suspect despite pleading with us we’re not being careful enough on social distancing in labs; people are putting it to the back of their minds.

I am grateful to the people who’ve checked up on me. I’m sorry if I don’t seem to listen when you tell me to take a break. I am managing. Theatre taught me well and I know the warning signs when I am tired and I’m monitoring how well I rest. And it’s not like there’s anything else to do at the moment. The team around me are rotating through rest days and I’m trying to look out for them. After 24 days of 14-16 hour days I was pretty tired by the end of yesterday having only had a few hours break on Thursday. I’m not sure of the wisdom of today’s shutdown given I know the challenge of the week ahead but I’m using it to go back maximally recharged. On the 50th anniversary of the Apollo 13 launch it was nice to sign off for Sunday to a message from David Baker encouraging us in our endeavours. There have been phone calls to friends and a BBQ in the sun and so on today. I have food – a large pile of cow does three roast dinners and several curries so my major food groups are covered.

I am deeply proud of the people I work with; they are doing an amazing job in difficult circumstances. I have faith everyone reading this would do the same in the same circumstances which is why I’m embarrased people compliment me on what I’m doing. I’m just being me and doing what I do with all the energy I can summon. We’re not out of the woods yet. For me it’s all about the end goal. It could all come off the rails in the week ahead just as quickly as it’s come together so far. Mis-steps are costly when the timescale is in days. We all desperately want to see this in manufacture. We steel ourselves for the coming days.