Getting Back to Work…At Work (The HYPER Guide to Returning to the Lab Safely)
A new semester has begun, summer is upon us, and research is kicking into high gear…but we are not operating under normal conditions. The pandemic has laid new challenges at our feet which has completely changed how we approach even the simplest of tasks. It is important, now more than ever, to band together as a lab community and apply our HYPER ethics to tackle these obstacles. As quarantine restrictions loosen in our state and we can begin returning to work, albeit restricted, we have created a plan for returning to work that help us to stay on track with our research goals while maintaining high safety standards to prevent the spread of disease. Follow these guidelines below to ensure success for yourself and HYPER. Before entering a lab space, ask yourself these questions: Am I feeling ill? If you are feeling ill at all, do not come into the lab. Stay home, rest, and get well. Do I need to go into the lab? Currently the policy is, if you can work from home, you should. Only essential work is allowed on campus. Essential activity is a combination of work that can only be done on site and is deemed appropriate by the supervisor (more on that later). Have I taken the required safety trainings? There are three trainings that must be taken before you can set foot on campus. Here is the list: “Pandemics: Slowing the Spread” “Disinfecting the Workplace for COVID-19” “WSU COVID-19 Safe to Return to Work” They can all be taken online at https://wsu.skillport.com/ . Do I have pre-approval to return to work from my supervisor? VCEA requires that you fill the Return to Work Request Form before returning to work. This is to verify that you have permission from your supervisor to return to work and that you have taken the proper safety training and read the COVID-19 required materials. Once you have established that you are well, trained, and have essential work, and have permission from your supervisor to return to campus, follow these steps before entering the lab space (note: all documents discussed in this post are uploaded to the HYPERawareness channel in Microsoft Teams):
- Fill out an Essential Activity Log.
- Submit your Essential Activity Log for approval.
- Block out time on the schedule.
- Arrange to get access to the lab.
- Upon entering the lab, find the cleaning cart and put on your PPE. Masks are now required, and gloves are highly recommended.
- Gather your tools and materials, keeping track of everything you touch. The red and white buckets are useful to help you distinguish between what tools/materials are clean or dirty.
- Complete your work as you laid out in your EAL. The red and white buckets are useful to help you distinguish between what tools/materials are clean or dirty. Be sure to follow social distancing guidelines if others are also working in the space.
- Once your work is done, put on a clean pair of gloves, grab the paper towels and disinfectant, and begin sanitizing your tools and workspace. Utilize the cleaning checklist for guidance. (We use alcohol as a disinfectant which needs to be left on a surface for at least 30 seconds to kill germs and bacteria).
- Make sure you clean every doorknob, light switch, drawer handle etc. Be thorough.
- Fill out the cleaning checklist, sign, date, and turn in.
- Exit the lab, PPE still on, walk out of the building and dispose of your PPE after you exit the facility.
How HYPER uses 6S for Success
It’s really simple. Continuous improvement results in speed, more success, and safety. Procedures are easy to continuously improve — they handle complex tasks with ease, they take input freely from many, they always take blame when something goes wrong, and they never complain about change. I enjoy good procedures. So… Why do we bury procedures in complex lab manuals that are never where we need them? Why do we leave procedures in passive black and white prose instead of fun active colorful mental maps? Why do we teach procedures in individual lab exercises that nobody else will ever use? This might be why we seldom see good procedures and why we so often struggle to continuously improve. HYPER had a system for developing procedures called Did you know? ‘Did you know?’ had people create a fun meme and a list of instructions that followed. The memes were fun, though often involved inside humor (e.g. see the movie Dr. Strangelove): But the resulting list of procedures were usually terrible due to a lack of standardization, people thought they were weird, and they often failed to catch the eye when memes were not used. ‘Did you Know?’ faded into irrelevance… The new HYPER lab manager, Mark Parsons, asked after a month on the job, “What systems do we have in the lab that have been standardized?” — which was the final nail in the ‘Did you Know?’ coffin. After they were pointed out Mark said, “Well if you use 5S for continuous improvement, why not use 5S to write procedures for our key lab systems?” 5S is from the Japanese philosophy for Lean Manufacturing and stands for:
- Systemize (set-in-order)
- Shine (sweep)
So just how dangerous is hydrogen fuel?
When I tell people I work on hydrogen fuel, they immediately say something very wrong like, “Are you worried about a mushroom cloud over your lab?” — Mushroom clouds are from a nuclear bomb detonation, and I don’t plan on starting thermonuclear fusion anytime soon in my lab, and if I did, it might save the planet. The other statement I often get is, “Wow, don’t want another Hindenberg!” Again, very wrong. Detailed studies from NASA and others have shown that the Hindenburg disaster was likely exacerbated, but not caused, by hydrogen. The Hindenburg’s sister ship, the Graf Zeppelin flew more than a million miles for nearly a decade on hydrogen before being grounded after the Hindenberg disaster. Go in and read the studies for yourself. The Hindenberg cut several corners the Graf Zeppelin did not, and you can’t expect the hydrogen to blow-out a diesel fire. The final one I sometimes get is, “Oh, the Challenger Shuttle!” again very wrong, Challenger was caused by the failure of a solid-oxide rocket booster o-ring. Sigh… So really, if I’ve just debunked the three most common misconceptions about hydrogen ‘incidents’ in a single paragraph, how dangerous is hydrogen fuel? “Tests were devised in which tanks containing liquid hydrogen under pressure were ruptured. In many cases, the hydrogen quickly escaped without ignition. The experimenters then provided a rocket squib (a small powder charge) to ignite the escaping hydrogen. The resulting fireball quickly dissipated because of the rapid flame speed of hydrogen and its low density. Containers of hydrogen and gasoline were placed side by side and ruptured. When the hydrogen can was ruptured and ignited, the flame quickly dissipated, but when the same thing was done with gasoline, the gasoline and flame stayed near the container and did much more damage. The gasoline fire was an order of magnitude more severe than the hydrogen fire. The experimenters tried to induce hydrogen to explode, with limited success. In 61 attempts, only two explosions occurred and in both, they had to mix oxygen with the hydrogen. Their largest explosion was produced by mixing a half liter of liquid oxygen with a similar volume of liquid hydrogen. Johnson and Rich were convinced that, with proper care, liquid hydrogen could be handled quite safely and was a practical fuel — a conclusion that was amply verified by the space program in the 1960s. At the time, however, Johnson and Rich filmed their fire and explosion experiments to convince doubters.” https://history.nasa.gov/SP-4404/ch8-6.htm That was during the 1950’s “Project Suntan” days with Kelly Johnson as project lead (yes that fabled Johnson that started Skunkworks and led the design of the SR-71 Blackbird, among others). For whatever reason, perhaps to remove all doubt, the Air Force Research Labs (AFRL) decided to reproduce Kelly’s experiment in the early 1980’s at Wright Patterson Air Force Base. This experiment involved shooting each container with .50 caliber incendiary rounds and simulating lightning strikes. This time hydrogen (on the top) is being compared with kerosene (on the bottom, aviation fuel or JP-1). This confirmed Kelly’s findings that the hydrogen fire ball dissipated quickly, providing less damage to the structure in every case versus the JP-1 test. The lightning test was inconclusive due to the container being obliterated in each case. The end result: hydrogen is safer than aviation fuel for aerospace applications involving an incendiary round penetrating the fuel tank. if you want more info on hydrogen safety in aerospace applications, NASA has loads of documentation of the history on-line, you can also check out Daniel Brewer’s book “Hydrogen Aircraft Technology.” Don’t believe me so far? It’s hard to believe that hydrogen can be safer than conventional hydrocarbon fuels. The following video was developed by AR Little corporation in the late 1950s and posted to YouTube by Aaron Harris, technical director at Air Liquide. This is the single most convincing video on liquid hydrogen safety I’ve seen: Those experiments established the safety to transport bulk liquid hydrogen (16,000 gallons +) on US highways. The results have been confirmed after nearly 50 years of practice. Even in the extreme — AirProducts had a full tanker of liquid hydrogen get rear-ended, while stopped, by a glycerin truck going at 65 miles per hour. The driver of the glycerin truck was instantly killed, but the glycerin slammed forward into the liquid hydrogen tanker and caught fire, fully engulfing the liquid hydrogen tanker in glycerin flames. The strength and insulation of the liquid hydrogen tanker was such that I’m not even sure the tanker lost vacuum — a small tube outside the tank busted and caught fire, sending a hydrogen flame up into the air that was eventually put out. As the director of liquid hydrogen safety at AirProducts, David Farese, is fond to say, “Nobody has ever been killed by a liquid hydrogen spill.” But we’ve talked about blimps, planes, rockets, and liquid hydrogen tankers so far. How does this translate to conventional automotive vehicles? The National Highway Transportation and Safety Administration has a comprehensive report on the hydrogen safety studies for vehicles. The report reviews international research as well as US research. The report identifies one direct comparison between hydrogen and conventional gasoline vehicles conducted by the US Department of Energy. Here’s a few pictures from the study, try and guess which car is fueled with H2 and which car with gasoline: Very similar to the aerospace studies, when a hydrogen storage tank ruptures and assuming a leak ignites, a hydrogen flame tends to burn out, and up and away from the structure, very quickly. One number that I remember from an introductory hydrogen technology class is that hydrogen diffuses away at 20 miles/hour, often straight up to space. Hydrogen literally is so fast that it has escape velocity and will eventually dissipate into space and the upper atmosphere. This is one of the inherent safety features of hydrogen — it doesn’t stick around long outside of a container. So as long as you don’t capture hydrogen beneath a structure where it can accumulate in dangerous quantities, you’re probably fine. Sadly, this excludes most research labs and garages where hydrogen sensors and ventilation must be carefully considered. Thankfully hydrogen is relatively easy to sense due to it’s high chemical activity. But what about the very high pressure within the fuel tanks? Toyota has conducted numerous studies of high pressure hydrogen tank damage and has posted the videos on line to access. Two that are particularly relevant are the tank being shot at point blank by a bullet (note that the hydrogen doesn’t ignite, which is why the airforce had to use incendiary rounds). It’s a pretty uneventful video. The next is a rear-offset impact collision — a severe loading case on the carbon-fiber fuel tank. Any other fuel tank would crumble. Remember that these carbon fiber fuel tanks are the strongest structure in the entire car. If there was a place to put a black box in the car, it would be on, or even inside the fuel tank. The Army decided to do their own field tests to determine whether the high pressure hydrogen storage tanks were safe for the battlefield during development of the ZH2 Colorado. They started similar to the Air Force study with armor piercing ammunition. Proceeded to a Rocket Propelled Grenade (RPG) strapped to the side of a tank. And concluded by strapping C4 plastic explosive to the side of the tank. As you can see from the figure, if your car is hit by an RPG there will be a hydrogen flame that will dissipate fairly quickly, but the tank will remain intact. This Army group strapped the RPG to the side of the tank (a perfect hit) and ignited the RPG with C4. They had C4 left over and didn’t want it to go to waste. So they strapped the rest to the side of the tank to see if they could truly get a full on tank explosion (perhaps to simulate a high-level state sponsored terror attack). In this final case the biggest danger was shrapnel from the aluminum liner. The final sentence of the report concludes, “Storing the hydrogen at pressure will not cause any more significant safety issues than liquid fuel in the event of a ballistic penetration or explosion due to the inherently safe design of the storage systems.” A final concern about hydrogen is that it is an emerging technology that you are probably unfamiliar with. Do we have enough experience to use this technology safely? There are currently over 25,000 hydrogen fuel cell powered forklifts operating around the world in shipping fulfillment centers the likes of Amazon, Walmart, FedEx, and more. These systems have been in service for well over a decade and operate 24/7, having completed >16 million refuels. Whether you knew it or not, your recent purchases were likely moved by a hydrogen fuel cell vehicle at some point during the routing to you. We just never see and interact with these vehicles. Moreover, these systems are often operated by someone that may or may not have a high school diploma — no Ph.D necessary. Still not enough? There are over 7000 hydrogen fuel cell vehicles on the road around the US. Given how much the movies like to bash hydrogen fuel (see Wonder Women, among many, many others), if there was a problem the media would have already informed us. It would be easy pickings given our embedded bias against hydrogen. But I’ve just compared hydrogen to other fuels. What about hydrogen safety relative to lithium ion batteries like those in a Battery Electric Vehicle (BEV)? These are very different energy storage technologies, and I’m not aware of direct comparisons between the two, but we’ll try. The key difference is that a BEV cannot rapidly dissipate the energy stored in batteries like a fueled vehicle can. This means that once a cell is damaged, neighboring cells in the battery can continue to catch fire or explode at a later time. This issue has led to BEVs requiring special storage and observation after a crash. You can search for videos of first responders trying to put out a BEV that is on fire. It’s not easy and the fumes are terribly toxic. The latest article I read is not clear whether BEVs have fewer post crash fires than gasoline vehicles. This is not an admonishment of BEVs! The fact that this emerging technology is already on par or better than gasoline in terms of safety is remarkable. It would be great to see a direct study comparing BEV safety with hydrogen fueled vehicles, once both technologies have had a chance to mature. We work with hydrogen every day in the lab. Ample opportunity for thought. I’ve often wondered why everyone has such an ingrained fear of hydrogen. I actually think the problem is convenience. Think back to your first introduction to hydrogen. Probably in a high-school or college chemistry class. Mine was in 8th grade when we electrolyzed water, filled a test-tube with a stoichiometric (ideal) mixture of hydrogen+oxygen, then lit the mixture with a match. It’s the easiest way to get a bang that gets people excited about chemistry. It’s also very impactful to see and feel the incredible energy release from such a small amount of gas. One of the most commonly watched videos of hydrogen on youtube is of a balloon filled with hydrogen and oxygen being lit on fire. But what is never added to the ends of these demos is that it’s pretty rare (and easy to engineer against) accidental mixing, within a pressure vessel, hydrogen and oxygen at the ideal ratio for an explosion. Regardless, this extreme introduction to hydrogen is embedded in our culture and memory. So how dangerous is hydrogen fuel? In many situations where a vehicle is located outdoors hydrogen is safer than conventional liquid fuels or natural gas. This in no way implies that hydrogen is not dangerous — there are many situations where hydrogen, like any other fuel or energy storage device, can cause an accident. As one life-long hydrogen expert said to me once, “Hydrogen is no better, nor worse, than any other fuel. You just have to know the rules for working with hydrogen.” Hence our work and mission. One final note — If you’re thinking about doing a hydrogen experiment at home, best to use caution. Hydrogen, indeed, has the highest flammability range and lowest required ignition energy of any fuel (4%-80% H2 by volume is flammable with air and a grain of sand caught in an escaping gas jet has enough kinetic energy to ignite a flow). The H2tools.org website has a wealth of information, including accident history to help guide you. Even the pros get caught in tough spots from time to time. Read about our near-miss hydrogen leak event sometime to get a feel for how very un-expected situations in complex systems can lead to risky repercussions. Regardless, with careful engineering, hydrogen fueled vehicles have a bright and safe future.
The Art of Safety
Let’s talk about Safety
One of the promising undergraduate students within the lab I worked in at Wisconsin was machining a part one day on a mill. He passed on the unsupervised lab-specific machine shop for risk of safety and was in the established student shop in the College — a fancy facade of a facility with a carefully organized tool closet and a windowed observation office where the head machinist, a disliked authoritarian of a person with decades of experience, could watch the shop. The student was very sharp, but left the chuck key in the mill head and turned it on. The key spun around, flew out, and took with it two of his fingers. As he’s holding his bloodied hand the head of the shop comes running out and begins yelling at him, “why did you do that!!” This would surely be a mark on his safety record. The student, in shock, ran away to the hallway outside where other students applied paper towels to his hand and helped him to the hospital. The problem here was not a lack of authority and control, or severity of consequences, but a lack of community connection and continuous improvement in the shop practices. A chuck key with an ejector spring prevents people from leaving it in the chuck, but is more expensive. The buddy system with a mentor can help spot some of these mistakes, whatever they may be. While these improvements may seem obvious to some, common sense isn’t so common. The WSU administration, led by the Office of Research, is undergoing an effort to re-emphasize and improve safety at our institution. I was recently informed by my chair that “at least one significant incident occurs at a university laboratory every month.” And OSHA (Occupational Safety & Health Administration) shows “that researchers are 11 times more likely to get hurt in an academic lab than in an industrial lab.” What is it about our authoritarian-legalistic structure of academic bureaucracy that naturally leads to this sub-par performance in such a critical area, and what can we do to improve?
Why Universities have a hard time with safetyI’ve written previously about how Universities evolved tree-like hierarchies. Nearly all of the reward system and feedback loops are geared towards promoting researchers to become power-driven authority leaders in their fields, which reinforces the extant authoritarian-legalistic system structure. The problem with these structures is communication. There is a very low amount of duplex communication, i.e. real conversations,,, talk. There just isn’t time for an administrator to sit down and spend quality time actually working with someone in a lab to mentor them — let alone knowing the people in their division. This results in a natural disconnection and un-grounding of administration from the people actually doing the work. I recently asked one of my friends, who is an administrator: “When was the last time you actually got a training by sitting down and doing the activity with someone, or a group of administrators?” He couldn’t remember a workshop that wasn’t primarily the traditional one way data dump. Couple the difficulties in communication with declining resources, increasing performance pressures, and a 2-5 year graduation timer on all your primary lab personnel, and you have a recipe for a safety nightmare. This means that it’s all too common to hear safety bulletins from administrators along the lines of the following: “make a new resolution to make this year accident free,” or to add “safety to annual performance evaluations,” or to “please report even the minor accidents,” and emphasis that “failure to report an incident… does result in consequences.” This is the easiest thing for an administrator in a power structure to do. Aside from invasive intrusions into labs, what else can they do? But this leads to other problems. I once knew an administrator who still conducted research in their lab. One day, a post-doc accidentally mixed two substances in the fume hood, leading to an explosion that destroyed the hood. The administrator, under pressure to reduce accidents in their unit, did not report the incident to others as they were the only required chain of reporting. Months later, a young faculty member in their unit had a similar incident that destroyed another fume hood. A year later, a similar accident sent 16 people to the hospital at a neighboring institution. When framed like this, the lack of communication almost seems criminal. Clearly, the sad reality is that these authoritative declarations coupled with punishments, within our communication-deficient authoritarian-legalistic system structure, can lead to corruption and actually be detrimental to the broader cause they intend to help. This command and control approach boils down to what is known as the deterrence hypothesis: the introduction of a penalty that leaves everything else unchanged will reduce the occurrence of the behavior subject to the penalty. I’ve previously written about the problems of applying the deterrence hypothesis to grading of coursework. In this case, safety is connected to my performance evaluation — which is primarily used for raise allocations and promotion. So in short, if an accident happens, my status and pay within the institution will suffer. So does this feedback mechanism promote better safety or lack of reporting — the most direct effect is lack of reporting. This is also presumes that the permanent disabling damage from losing fingers or another accident is not deterrence enough — the approach assumes that faculty delegate all risks to students rather than doing the activity themselves. In a famous study titled, “A fee is a price” researchers investigated the efficacy of the deterrence hypothesis at mitigating the undesirable behavior of picking a child up late from daycare. This is low — abusing the personal time of a lower-paid caretaker charged with the health and well being of your child. In many ways this parallels the minor accidents, cuts, and knuckle bangs we’re being asked to report. In order to couple these to performance evaluations, a non-arbitrary metric must be created to decide how big the penalty, or price, should be. Contrary to expectations, the researchers performing the study found that adding the penalty actually increased the negative behavior that it intended to deter. The researchers deduced that the penalty became a price — if I’m late, I’ll pay the $20 and everything is ok — regardless of whether the caretaker had other plans. Perhaps the most troubling finding from the study was once the penalty was systemized, the bad behavior continued regardless of whether the penalty was removed or not. Once you marginalize or put a fee on a person, it’s tough to treat them as a person with rights and dignity again. I’ve seen this play out many times with daycares, teams, and communities I’ve been involved. Reliably the diminishing of people and disruption of personal connection leads to the demise and under performance of the organization. When an authoritarian is presented with this evidence contrary to their belief, they reliably counter with, “oh I’ll make the penalty severe enough to deter the behavior.” What else can they do? This approach, in the absence of appropriate developmental scaffolding, leads to a depressed environment adverse to uncertainty. Everyone becomes afraid to report safety, afraid to discuss safety, afraid to try new things and push the limits (isn’t trying new things and pushing the limits called research?) — often simply because trying new things is no longer the norm. When something is not the norm, it becomes an uncertainty risk and threat. I once was having a discussion with an administrator about a new makerspace on campus. This prompted the statement, “But we’ll never be able to control the safety!” To this I immediately responded: 3D printers are robotic hot glue guns with safety shrouds! Every campus in the US has a gym with a squat rack (people put hundreds of pounds on their back on a daily basis with poor form), climbing wall (someone could fall!), pool (but what if someone drowned!), and a hammer/discuss/shotput/javelin toss (yikes!). Arbitrary targeting of risk/blame is another characteristic of authoritarian/legalistic organizations because they lack established heuristics, a.k.a. processes, to work through safety scaffolding of new activities. Shot put and the hammer toss are established activities that our culture has normed to, where the risk in developing the established safety protocol was encumbered centuries ago. Less of a need for an administrator to CYA. Moreover, a command and control approach isn’t what makes them safe — it’s connections and discussions with people. The disincentive for not using the squat rack correctly is chronic back pain, something I deal with on a daily basis. That risk didn’t stop me from squatting incorrectly! The problem was ineffective coaching/scaffolding. Telling the coaches to coach better won’t explicitly fix that. And we can’t always rely on starting a new facility fresh with appropriate safety from the beginning. I once attended a safety seminar, led by a well respected researcher at another academic institution. The researcher described the brand new building they were having built, and all of the safety protocols they implemented to make it safe. Afterwords I asked the researcher their approach for improving safety within established student clubs. The response stunned me: “I’m not really sure. We have another building for that. We never allow students to work after hours unsupervised.” They had nearly entirely avoided teaching intrinsic safety culture! The students were never allowed autonomy to make decisions! I told myself I’ll never bring in a student from that institution. This exemplifies what happens when we are granted huge resources without having to perform or evolve to a level that justifies them like in industry. It was almost Orwellian. Certainly not the future our society and university needs. After having a string of safety incidents in their unit, an administrator and safety board required every club and lab to have a “designated safety officer” or a designated authority to control safety for the group. After a few months in this position, one lab’s “safety officer” lamented to me, “Sometimes I need to be the bad guy because people don’t take safety seriously. But it gets tiring. They dislike me for it, blame me when stuff goes wrong, and they still don’t take safety upon themselves.” This is directly analogous to the problem of quality control faced in Lean Manufacturing. In Lean, the question comes up of whether something you’ve manufactured meets the design specification. Do you hire a quality control czar to stop production if product starts coming out not to spec or unsafe? Ever heard a story of someone who was frustrated with the quality cop coming over to tell them things were wrong yet providing no explanation what was wrong or how to fix it? Moreover, the only way to ensure 100% quality/safety is 100% inspection — not a sustainable or scale-able approach. The Lean approach is to design quality/safety control into the production process — if the part can’t be made wrong/unsafe, it’s much easier to achieve 100% safety/quality. Moreover, if everyone is responsible for checking safety/quality during the production process, you just made everyone in your group a safety officer and multiplied the odds of spotting a risk before it’s realized. Another common characteristic of authoritarian-legalistic approaches to safety is the posting of negative signage/reminders. “No ___ allowed.” “Don’t do this!” etc. Here’s a great counter example from Seth Godin titled, “How to make a sign.” The problem is we become numb to these negative associations and quit paying attention. That’s why we have “Did you know?” documents in our lab that just describe the right process for doing something. We try to include a funny meme at the top to get people to positively associate and look at these. Here’s an example posted near a compressed gas bottle area: It boggles my mind why lab leads do not have safety procedures posted by all key lab processes and equipment. It’s really simple — if something goes wrong you change the procedure. Changing the procedure is orders of magnitude cheaper and easier than changing the equipment or personnel. It’s therefore much easier to continuously improve procedure. So we’ve shown through multiple ways the safety shortcomings of traditional authoritarian-legalistic bureaucratic structures. How do we get beyond these to cultivate a sustaining community and culture of safety within such institutions?
Let’s talk about Scaffolding Layers of SafetyIn short, the real solution to safety is performance based funds from a diverse array of sources, like in industry. This naturally dovetails with a diverse, sustaining and supporting lab community. If you’re operating efficiently and effectively, you can’t stand the loss of a well trained person, even for a few days. But that’s a chicken or the egg conundrum for us in universities. I’ve written previously about the challenges and tips for building sustaining lab communities. It’s not easy! In short, you have to scaffold multiple orthogonal value sets. But the end result can literally be a life-saver! About 6 months ago we had a near-miss hydrogen venting event in the lab caused by a power failure and a pressure relief valve freezing shut. Because we had multiple layers of safety engineered into the experiment, and multiple layers that we could communicate within the lab, and university, a potential tragedy was avoided. In the end, instead of being reprimanded, we got a 5 month extension on the project, upgrades to the lab, and were told by administrators, “This was not an accident because you are working hard to do everything right.” In a recent post I provided a scaffold to grow agency in engineering education. The key premise being that values change, and we need a scaffold that relates to many different value sets. Safety is no different. This provides the “layered” approach to safety that is popular in software security and other forward thinking fields. Here are several levels and examples of what we do in the HYPER lab to help activate the appropriate values: Authority: Typical to most research labs. A grad student, or preferably a team of 2 grad students and 2 undergrads are responsible for maintaining an experimental or fabrication facility. Their names and pictures are associated with the project both in person and virtually through the lab website. They also are given an instant communication channel that the lab can see specific to the experiment/facility. Notice I was careful to connect authority to responsibility and have carefully steered clear of the power-command authority traditional of academia. Legalistic: Each experiment has a Safety Protocols and Procedures manual that is continually refined (send me a note if you want to see ours, I don’t want to display online in case of nefarious actions.) The safety manual includes a Failure Modes and Effects Analysis (FMEA) that predicts all of the likely safety issues and emergency protocols. We implement the buddy system for changes to experiments and manuals — you need to have someone else there to approve. We also are continuing to develop a common lab-rules, standards, and values banner that goes above the doors to spaces. We are working to develop standard trainings for the right and wrong ways to utilize plumbing fittings and seals common in our work. We emphasize use of engineering standards wherever applicable. Performance: Once a student is proficient with the responsibilities, literature, trainings, and practices in an area, they develop a did-you-know? heuristic process document. This informs people of the necessary steps unique to the space for accomplishing a task. Students at this level are expected to begin bringing in their own resources and recruit their own students to working on their project. We are also implementing a traveling safety award for the lab and tracking days without incidents. Community: All of the lab members (without me) go to lunch together once a week. In addition we work together as a lab for a 3 hour time-block once a week on lab community builds and needs, including safety. This is greatly enabled by allowing all of the students to contribute to our community website (this site) and our Slack message board. We offer tours of our lab as frequently as possible to gain critical feedback and advise from potential stakeholders or partner labs. I’ve written previously about Tradings Places and Ways. Systemic: We’ve established the expectation of all lab members to contribute and cultivate our system and community by looking for and enhancing restoring feedback loops that improve our efficiency in each of these levels. We do this by building our people from the ground up — we seldom import talent into our culture. This is very similar to Toyota and other lean production environments. No surprise, our lab has the Lean Philosophy of 5-S posted throughout: Sort, Sweep, Systemize, Standardize, Sustain. So far things seem to be working. We have equipment and builds that I’m sure my colleagues think are ludicrously difficult and safety risks. We’re the only lab in the country that focuses on cryogenic hydrogen — which has the highest thermal, fluid and chemical power gradients. Hydrogen should not be taken lightly! But I also know that the students are developing in incredible ways and coming together as a community to make it happen, safely. One of the reasons I know this is the fact that they’re not afraid to talk about safety, and they are having fun with it! So let’s talk about safety! Send me your comments and suggestions: jacob.leachman<at>wsu.edu
Our near-miss hydrogen vent in ETRL 221
Yes, rumors about a hydrogen bomb in ETRL are exaggerated. On August 2nd around 10:00 am, the HYPER lab had an uncontrolled hydrogen vent into ETRL 221. There was no damage to equipment or personnel, leaving the event classified in accordance with U.S. Department of Energy (DOE) criteria as a “near-miss”. While no critical flaws were identified with the experiment design or procedures for handling the event, the subsequent expert review by the Hydrogen Safety Panel has valuable lessons learned for the WSU and cryogenic hydrogen research communities.
Event TimelineAround 9:00 AM — A bird flew into a sub-station and shut down power to the entire WSU-Pullman campus. Campus-wide backup diesel generators kicked on and began powering back up campus sectors in a pre-determined order based on need. However, this was exactly the time that the HYPER lab was planning to begin a test of a hydrogen vortex tube in the Cryocatalysis Hydrogen Experiment Facility (CHEF) which had operated for 4 years without incident and had been working for a week to accumulate 4.25 liters of liquid hydrogen. This quantity of hydrogen is less than 40% the quantity required to reach a flammable mixture if the entire room was equally mixed. However, a hydrogen vent easily reaches the lower flammability limit near the location of the event. An addition risk is over-pressurizing the containment vessel. Hydrogen expands ~780 times in volume when heated from a liquid to room temperature. 9:20 — I was across campus and received a WSU Alert text about the outage and called the HYPER lab to learn that all power was down, including the fume hood ventilation, and a mechanical pressure gauge on the experiment read 90 psi. Instructions were given to manually vent the experiment if the pressure rose to 120 psi. I got in my truck to drive to the lab. 9:35 — Power is restored and the ancillary support systems for CHEF were turned back on. However, by this time the cryocooler had warmed to the extent that the thermal wave moving towards the stored liquid hydrogen could not be mitigated in time to prevent rapid boiling. 9:40 — I arrived in the lab and the pressure had risen to 120 psi, a manual vent of the hydrogen through the fume hood began. However, pressure continued to build in CHEF as the temperature had risen above 30 K and the liquid was rapidly boiling while transitioning to a supercritical fluid. CHEF has three, redundant, proportional, pressure relief valves that are designed to vent through the lab fume hood. All three valves are adjustable and tested within the last 8 months to open at 150 psi, 160 psi, and 165 psi. The 150 psi valve failed to open. When this occurred the manual vent was increased. The 165 psi valve was the first to open, however, a small liquid vent port on the side of the 165 psi valve was jetting a hydrogen stream towards electrical equipment. 10:05 — While we debated on ducting the venting hydrogen stream away from electronics we heard a “pop” that we did not know the origin of. In retrospect, the pop was likely the 160 psi pressure relief valve opening. However, we could not definitively say the noise was not a small hydrogen ignition. Since we did not know the amount of hydrogen entering the room versus the fume hood, and personnel and equipment remained at risk, we evacuated the lab, pulled the fire alarm, and called 911 to explain the situation. 10:30 — Pullman fire department was on-sight and briefed about the situation, hydrogen danger, and maximum quantities that could have been released in the room. We had a real-time readout of the experiment dashboard handy on a cell-phone and could see that the experiment had vented the remaining hydrogen and was de-pressurized. Out of precaution, the fire fighters were going to give the hydrogen time to diffuse before entering the building. 11:00 — Fire fighters enter the building (picture above) with a thermal imaging camera, hand-held hydrogen detector, and CO2 extinguisher. The maximum hydrogen level they could detect in the space was 1% of the lower flammability limit, so the vented hydrogen was already gone. 12:00 — We were allowed to go back into the lab to ensure the experiment was safe. 12:30 — The all clear to go back into the building was given. The experiment was safed with inert gases. I went over to the college communications team to explain the event if a reporter called and begin typing up this timeline to explain to WSU staff and administration.
Post Event AnalysisThis was about worst possible time for a power outage. Given an additional hour, the planned experimental test would have dissipated the hydrogen. Total power loss with the maximum quantity of liquid hydrogen in the lab was a scenario we had discussed and had thought through, but knew was unlikely. Practicing this kind of event is difficult, dangerous in itself, and we had not. The immediate feedback from the first responders and WSU response team was very positive and supportive. We were commended on our knowledge of the situation, promptness with responses, use of multiple redundant safety systems, and proactive actions to mitigate risk to personnel. The lab went out to lunch the next day to discuss Strengths-Improvements-Insights (SIIs) of how we handled the event. I was honestly expecting a degree of fear from some of the newer lab members, however found confidence instead. We had just made it through a serious situation. While we had procedures and trainings for nearly all aspects covered, we realized the need to systemize and standardize our trainings to ensure continually improvement. Specifically:
- We are now developing a safety demonstration kit/system where all lab members are expected to master the demos and develop their new contributions to the kit/system. While we had completed demos in the past, we had not done so on a regular basis or in a systematic fashion that requires experiential contributions from the lab members. And it should be pretty fun.
- We need to further systemize our lab’s procedures for Management of Change (MOC) and Failure Modes and Effects Analysis (FMEA). We currently have separate forms for these. We have plans to integrate MOC and FMEA into our House of Quality system for designing our experiments. The result will hopefully be a single procedure that integrates all aspects into a natural and expected outcome after all changes to systems.
- We need to update our now outdated lab banner above the main door on the way out to increase relevance and efficiently remind all of lab safety procedures and our commitment to a safe community. We have initial ideas of how to make this banner space one that continually improves.
Important Notes for the WSU CommunityWhat the expert review really uncovered were my flawed assumptions in how the building we were in, ETRL, functioned. Specifically:
- The hallway bottle locker was not sprinklered nor ventilated. Believe me, I had asked several people about this prior to the event and you can feel and hear the air when you open the door. I assumed the sprinkler was above a duct that runs through the space about 16′ up near the top. That was wrong.
- The fume hood system shuts down when the power is out, and ETRL is not high on the priority list of buildings to re-energize when the power goes out.
- When you pull the fire alarm for the building, the supply air to spaces shuts down while return stays on to create negative pressure in each lab space.
3 Years LaterPeople continue to say, “This wasn’t an accident because of how you handled it. We want this to be an example for other faculty.” I really hate heros — people who have to come in and save the day. No system that is continuously improved within a university should ever need a hero because there is a tested fail-safe for every situation. After the event the department and college stepped up to help pay for installation of a dedicated hydrogen vent line (built to CGA hydrogen vent stack standards) to the roof of ETRL. We’ve test this vent on a regular basis whenever we run the experiment. We now also have dedicated hydrogen sensing systems specific to each experiment that will vent the entire experiment if hydrogen is ever detected at an amount close to the lower flammability limit. We also have backup power procedures to handle common power outages. Finally, we have a 100 page safety document specific to CHEF that has been reviewed by the DOE Hydrogen Safety Panel. In spring of 2019 we became the only university that is a founding member of the AIChE Center for Hydrogen Safety and participate in regular meetings. In many ways I look back on this event as the turning point from when the lab transitioned from hacker class to professional. Had I been proactive from the beginning though, I would have engaged these hydrogen safety professionals before there was a near-miss. That’s my closing message to everyone else starting a hydrogen research project — use the tools available to you at h2tools.org and don’t hesitate to reach out for advice.
Compressed Gas Bottle Safety
Compressed gas bottle safety is important! Follow these simple rules to ensure your gas bottle stays a container – not a rocket.
- Bottles should be chained at all times to prevent them from tipping over.
- Steel caps need to be on bottles when not in use – especially for transportation.
- Transport gas bottles on bottle carts.
- Always use pressure relief devices when attaching high pressure bottles to systems.
- Ensure lines are depressurized and bottle valve is shut before disconnecting the bottle from a system – even when the bottle is “empty”.
- Flammable gas bottles should always be grounded before use to avoid static ignition