Yes, rumors about a hydrogen bomb in ETRL are exaggerated.
On August 2nd around 10:00 am, the HYPER lab had an uncontrolled hydrogen vent into ETRL 221. There was no damage to equipment or personnel, leaving the event classified in accordance with U.S. Department of Energy (DOE) criteria as a “near-miss”. While no critical flaws were identified with the experiment design or procedures for handling the event, the subsequent expert review by the Hydrogen Safety Panel has valuable lessons learned for the WSU and cryogenic hydrogen research communities.
Around 9:00 AM — A bird flew into a sub-station and shut down power to the entire WSU-Pullman campus. Campus-wide backup diesel generators kicked on and began powering back up campus sectors in a pre-determined order based on need. However, this was exactly the time that the HYPER lab was planning to begin a test of a hydrogen vortex tube in the Cryocatalysis Hydrogen Experiment Facility (CHEF) which had operated for 4 years without incident and had been working for a week to accumulate 4.25 liters of liquid hydrogen. This quantity of hydrogen is less than 40% the quantity required to reach a flammable mixture if the entire room was equally mixed. However, a hydrogen vent easily reaches the lower flammability limit near the location of the event. An addition risk is over-pressurizing the containment vessel. Hydrogen expands ~780 times in volume when heated from a liquid to room temperature.
9:20 — I was across campus and received a WSU Alert text about the outage and called the HYPER lab to learn that all power was down, including the fume hood ventilation, and a mechanical pressure gauge on the experiment read 90 psi. Instructions were given to manually vent the experiment if the pressure rose to 120 psi. I got in my truck to drive to the lab.
9:35 — Power is restored and the ancillary support systems for CHEF were turned back on. However, by this time the cryocooler had warmed to the extent that the thermal wave moving towards the stored liquid hydrogen could not be mitigated in time to prevent rapid boiling.
9:40 — I arrived in the lab and the pressure had risen to 120 psi, a manual vent of the hydrogen through the fume hood began. However, pressure continued to build in CHEF as the temperature had risen above 30 K and the liquid was rapidly boiling while transitioning to a supercritical fluid.
CHEF has three, redundant, proportional, pressure relief valves that are designed to vent through the lab fume hood. All three valves are adjustable and tested within the last 8 months to open at 150 psi, 160 psi, and 165 psi. The 150 psi valve failed to open. When this occurred the manual vent was increased. The 165 psi valve was the first to open, however, a small liquid vent port on the side of the 165 psi valve was jetting a hydrogen stream towards electrical equipment.
10:05 — While we debated on ducting the venting hydrogen stream away from electronics we heard a “pop” that we did not know the origin of. In retrospect, the pop was likely the 160 psi pressure relief valve opening. However, we could not definitively say the noise was not a small hydrogen ignition. Since we did not know the amount of hydrogen entering the room versus the fume hood, and personnel and equipment remained at risk, we evacuated the lab, pulled the fire alarm, and called 911 to explain the situation.
10:30 — Pullman fire department was on-sight and briefed about the situation, hydrogen danger, and maximum quantities that could have been released in the room. We had a real-time readout of the experiment dashboard handy on a cell-phone and could see that the experiment had vented the remaining hydrogen and was de-pressurized. Out of precaution, the fire fighters were going to give the hydrogen time to diffuse before entering the building.
11:00 — Fire fighters enter the building (picture above) with a thermal imaging camera, hand-held hydrogen detector, and CO2 extinguisher. The maximum hydrogen level they could detect in the space was 1% of the lower flammability limit, so the vented hydrogen was already gone.
12:00 — We were allowed to go back into the lab to ensure the experiment was safe.
12:30 — The all clear to go back into the building was given. The experiment was safed with inert gases. I went over to the college communications team to explain the event if a reporter called and begin typing up this timeline to explain to WSU staff and administration.
Post Event Analysis
This was about worst possible time for a power outage. Given an additional hour, the planned experimental test would have dissipated the hydrogen. Total power loss with the maximum quantity of liquid hydrogen in the lab was a scenario we had discussed and had thought through, but knew was unlikely. Practicing this kind of event is difficult, dangerous in itself, and we had not.
The immediate feedback from the first responders and WSU response team was very positive and supportive. We were commended on our knowledge of the situation, promptness with responses, use of multiple redundant safety systems, and proactive actions to mitigate risk to personnel.
The lab went out to lunch the next day to discuss Strengths-Improvements-Insights (SIIs) of how we handled the event. I was honestly expecting a degree of fear from some of the newer lab members, however found confidence instead. We had just made it through a serious situation. While we had procedures and trainings for nearly all aspects covered, we realized the need to systemize and standardize our trainings to ensure continually improvement. Specifically:
- We are now developing a safety demonstration kit/system where all lab members are expected to master the demos and develop their new contributions to the kit/system. While we had completed demos in the past, we had not done so on a regular basis or in a systematic fashion that requires experiential contributions from the lab members. And it should be pretty fun.
- We need to further systemize our lab’s procedures for Management of Change (MOC) and Failure Modes and Effects Analysis (FMEA). We currently have separate forms for these. We have plans to integrate MOC and FMEA into our House of Quality system for designing our experiments. The result will hopefully be a single procedure that integrates all aspects into a natural and expected outcome after all changes to systems.
- We need to update our now outdated lab banner above the main door on the way out to increase relevance and efficiently remind all of lab safety procedures and our commitment to a safe community. We have initial ideas of how to make this banner space one that continually improves.
The DOE flew out the lead member of the Hydrogen Safety Panel, along with a top liquid hydrogen expert with nearly 30 years of experience from industry, for an event review the majority of the day yesterday. The WSU response team also participated in the event. The first order of business was determining why the pressure relief valves failed to function as designed.
The first pressure relief valve was a Circle Seal CSC 500 series that is adjustable and was set to open at 150 psi. The valve is positioned on a buffer reservoir far away from anything cryogenic. After the event we tested the valve and were shocked that it did not open until ~230 psi! After removing the valve for inspection, we noticed that vacuum grease had deposited on the inner surface of the valve from a vacuum pump positioned closely nearby. When the cold hydrogen vapor was boiling and moving back into the buffer reservoir the vacuum grease likely froze. The valve may not have opened until well above 300 psi during the event. While serving as a temporary fix when we had insufficient funds for appropriate valves, several mistakes lead to this being an issue.
The second pressure relief valve to open was the 165 psi valve connected directly to the liquid chamber. This adjustable valve is a Rego 9400 series that is both cryogenic and hydrogen rated. This valve would have removed all excess gas from the system and we would not have had a “near-miss” had a jet of hydrogen not been escaping from the side of the valve towards electronics. A picture and drawing of the valve is shown below. Can you see the problem? You may have to look closely.
Not seeing it? Neither did we. On the bottom right corner of the drawing above, you’ll see two horizontal lines from the inside of the valve to the exterior of the valve body. This is the roughly 0.1″ diameter unmarked hole that the hydrogen was escaping from. You can buy the valve without this vent hole, which is to help with two-phase flow through the valve.
A secondary issue to the valves not opening was the size of our lines running to and from the valves. The Compressed Gas Association and API have standards on pressure drops associated with pressure relieving systems. While we had completed calculations to not run into “choked-flow” where a shock wave forms in the line in a maximum flow scenario, these estimates do not conform to the standards. Although the combination of valves we had worked, we need larger and cleaner lines running to and from our relief devices.
A test that could have identified these problems is a cold over-pressure test. This is a difficult test to accomplish as you risk creating a hazardous situation. However a cold helium over-pressure test can be conducted in a controlled manor and we plan to do this in the future.
Important Notes for the WSU Community
What the expert review really uncovered were my flawed assumptions in how the building we were in, ETRL, functioned. Specifically:
- The hallway bottle locker was not sprinklered nor ventilated. Believe me, I had asked several people about this prior to the event and you can feel and hear the air when you open the door. I assumed the sprinkler was above a duct that runs through the space about 16′ up near the top. That was wrong.
- The fume hood system shuts down when the power is out, and ETRL is not high on the priority list of buildings to re-energize when the power goes out.
- When you pull the fire alarm for the building, the supply air to spaces shuts down while return stays on to create negative pressure in each lab space.
One final note, I was under a lot of pressure to not pull the alarm and evacuate the space. My tenure appointment would not become official until August 10th and an accident could put that in question. Also, I was under pressure from the Department of Energy to get measurements from our experiment before the end of August. Pulling the alarm put both in jeopardy. The response from WSU and the DOE has been entirely supportive through this. Nobody has questioned that decision. In the end, we’re coming out of this even more together as a lab with better safety designs and procedures. Thanks to everyone for the support!
3 Years Later
People continue to say, “This wasn’t an accident because of how you handled it. We want this to be an example for other faculty.” I really hate heros — people who have to come in and save the day. No system that is continuously improved within a university should ever need a hero because there is a tested fail-safe for every situation.
After the event the department and college stepped up to help pay for installation of a dedicated hydrogen vent line (built to CGA hydrogen vent stack standards) to the roof of ETRL. We’ve test this vent on a regular basis whenever we run the experiment. We now also have dedicated hydrogen sensing systems specific to each experiment that will vent the entire experiment if hydrogen is ever detected at an amount close to the lower flammability limit. We also have backup power procedures to handle common power outages. Finally, we have a 100 page safety document specific to CHEF that has been reviewed by the DOE Hydrogen Safety Panel. In spring of 2019 we became the only university that is a founding member of the AIChE Center for Hydrogen Safety and participate in regular meetings.
In many ways I look back on this event as the turning point from when the lab transitioned from hacker class to professional. Had I been proactive from the beginning though, I would have engaged these hydrogen safety professionals before there was a near-miss. That’s my closing message to everyone else starting a hydrogen research project — use the tools available to you at h2tools.org and don’t hesitate to reach out for advice.