April|2019 – FOG!

Fog!

I am skipping February and March blogposts – but I will write them, I promise 🙂 I just wanted to write April’s post while the memories are still fresh.. What happened that it is so important?

Let’s start from beginning – on 8th of October 2018 we made a tube #6059. There was nothing special about that tube with one exception – we noticed a subtle gray fog developed on internal side of the glass envelope. I remembered that exactly this problem appeared once in 2016 – it happened just once, so I decided not to spend time (the most valuable asset) to investigate it deeper (which I always love to do – sometimes wasting too much time). So the #6059 tube went to box with rejections as well, I probably concluded that it was a dirt on glass which wasn’t cleaned with our standard cleaning procedure. It was just weird that it went through visual inspection of envelopes before assembly (done by Lenka).

Four days later, Renata brought me another tube (#6101) with fog which she found during inspection of finished tubes. More tubes soon showed same problem #6127, #6181, #6213, #6221, #6243. This brought my attention to this problem, I decided to make the cleaning process for glass more intensive (longer, temperature increased etc..) – the idea was that maybe we are introducing some dirt into the tubes, so intensive cleaning should fix it. But it didn’t, more tubes followed in November. And December. In January, 5% of production was rejected because of this issue. I tried around 10 proven troubleshooting techniques which I developed over the years, but none of them worked. The occurence was random and I also couldn’t measure any difference from good tubes in pressures (gas loads) during pumping process. The most tricky thing was that electrical characteristics of finished foggy the tubes were identical to good ones – this is usually where we can distinct good tube from bad.

Long story short, in February my all focus was on this problem, all other projects went aside, I turned off notifications and started replying on emails only once or twice per day. In middle of April, the worst possible thing happened – 100% of tubes produced developed this issue and we had to stop production. This is something I never had to do before – I was always able to solve problems in few days or weeks (only improving our sealing techniques in 2017 lasted longer). The problem was that hard to solve.

Consider this as a teaser, this problem was so frustrating that I want to explain it in a video and show it more in details. It deserves it.

Finally on 1st of May, I solved the problem and from that date, we run production at normal pace – with few minor improvements I learned on that problem.

Days To Death

Since 2018 we have new website.. To step up our opennes, we introduced new feature – a footer with live statistics. Visitor now can see how much we sell, average failure rate of our tubes, bank account balance etc.. Later there will be a page with much more detailed statistics with explanation for people who love numbers and charts just like I do 🙂 OK, one of the stats is named DTD – Days To Death. This kind of “metrics” was suggested by Lukáš Greň from Moravio.com in one of their blogposts, I can’t find it right now. Its purpose was to give an idea to their employees-developers how much time the company still has before they go bancrupt. The calculation is simple: Cash/DailyCosts, the number will give you the amount of days.. I liked this idea so much I decided to add it to the stats. In our case, DTD means amount of days we would have in case we would have to stop production and payments from customers.

At that time when we were making the website I thought such situation is very improbable – I was thinking about a fire in our lab or other disasters.. It was more an ironic joke than a serious metrics – few months later I actually had to stop the production and the DTD number became the most watched metrics. We went from almost 90 days to below 40 during that investigation period.. Thats how things works, thats the irony at its best..

You can imagine the pressure – I was solving difficult problem while watching our cash reserve getting lower and lower. During that time I had to buy several pieces of expensive equipment to be able to analyse the problem better. All that time I was thinking about the money spent on Designblok and development of new products. That $11,000 what the design fair cost would give us at least another 14 days.. Fortunately the problem was identified and solution found before we went belly up.

Conclusion

Year 2018 was successful and brought very little troubles, I got confident and started to explore new possibilities of products and projects we could do. The “fog problem” brought me back to ground the hard way – it reminded me once again to be humble and never underestimate demanding character of vacuum. My current feeling is that we do too many things with too small team – apart from tube manufacture, we also develop electronics, software and all what a final product needs. I now want to bring more focus on tubes itself and introduce several control mechanisms to make problems like this less probable in future.

A little joke for vacuum scientists.. When searching for information on internet, I found a forum where a guy asked for help with some ultra high vacuum related stuff. One of the replies was: “I’ve just changed my job – seeing your problem I am so happy I don’t have to go to 10E-10 anymore!”.. I totally udnerstand that frustration, even when I tried only 10E-7 🙂