On turning up at office on Tuesday, after a 3 day holiday, heard about the problem blowing out here and there in our kiosks due to a broken package in our repository. Thankfully it was happening only with our ongoing new rollout image which was installed in around 80 kiosks in Karnataka. We decided to make a field trip and asses the situation, find a fix that can be executed in the affected kiosks with the help of the field engineers and operators itself. Our earlier plan of making the trip on Tuesday evening itself was dropped as it would have been too late when we could reach even the nearest affected kiosk.
We started the field trip around 8 a.m. on Wednesday, getting a cab to take us around. This will help us in visiting as many kiosk as possible in a day. We started with a 2 hour travel to Yadiyur near Kunigal falling under the Tumkur district. The cab driver suggested us a nice hotel in Kunigal, where we had plate-idlis, vada and tea before continuing our journey.
The Yadiyur kiosk is located right in front of the cattle market of the village. We started hunting for the bug which caused all the mess and soon found it to be an extra netmask line in the network interfaces file. Removing this line and rebooting the machine brought the network up and the machine started to operate normally. But when we tried the second machine, it was more than this. The firewall seemed to have messed up during the upgrade, so we needed to clean it, then clean the interface file and bring up the system back. We also started calling up the field engineers, asking them to go to the nearest problematic kiosk and guide them to perform the same fix we found.
Having found the fix, we wanted to test it again. So we next moved to our kiosk in Maisandra, which is some 20 Kms from Yadiyur. We had a bit of trouble in finding the kiosk as it is located in a building interior to the main road. The kiosk was small but neatly maintained. We fixed the first machine following the same procedure, but the second machine did not face the updates because of lack of network switch to connect both the machines and hence it was still unaffected. We finished our job and left to the next location, but it was already afternoon.
Our next location was Huliyurdurga, which is on the other side to Yadiyur and it took us almost an hour to reach. We again had problems locating the kiosk as the name board is not put on the building it is located at and people could not get a clue of what we were searching for. Thankfully the operator was standing out in front of the building, could spot our car and I could spot her waving. We quickly fixed the first machine, made it operational as customers were waiting and moved to the second machine. With some extra effort, we managed to fix it too.
We were hungry and decided to feed us with some snacks from the bakery. We decided not to visit the next kiosk Tiptur as it was very far and it was already 5 p.m. We started our way back to Bangalore and after crawling through the traffic around Nelamangla and Yeshwantpur, we reached office by 7.40 p.m.
We have learnt our lesson that we need more stricter Q.A. process and more rigorous testing before we update our stable repository next time. When we were in the kiosk, probably a hour or two at each location, there were indeed customers turning up at least once every hour 🙂 On the whole, it was a nice day of traveling through rural Karnataka and visiting few more of our kiosks.