Monday, September 28, 2009

How SingTel Broke The Apps: The Gory Details

SingTel's engineers broke the three apps that check data usage--SG BBOM, iBBOM and Singtel Data Usage--by making two different changes to the way the server accepts requests for data. Interestingly, it looks a little like a cat-and-mouse game as the engineers made the second change after one developer was able to get around the first change.

At about 8.50pm on Sunday, iBBOM developer Meiwin found out that both his app and Singtel Data Usage had suddenly stopped working. Both apps would close abruptly, almost immediately after launch. SG BBOM, on the other hand, handled the problem more gracefully, by asking users to check their input. It is not known when the change was made but it was estimated that the change took place some time after 6pm on Sunday evening.


A flurry of tweets between the developers quickly alerted all of them to the issue and they began to examine the problem. According to Singtel Data Usage developer Cheah Chu Yeow, they found out that Singtel had added an additional check to the request so that the HTTP-Referer had to be "http://home.singtel.com/bbmobile/checkusage.asp." Without this referrer, the call to get the data would return an empty string.

Only Singtel Data Usage could work around this change quickly because the app sends data requests through the developer's proxy server. Both iBBOM and SG BBOM send the data directly from the app to SingTel. For them to fix their apps, they would have to resubmit a new version, which would take time to be approved.

By 10pm, Chu Yeow had gotten around the problem and his app, Singtel Data Usage, was working again. This, however, wasn't the end of the story.

At about 1am on Monday, SingTel's engineers made another change. The second change required cookies to be set before a query for data usage is accepted. Chu Yeow only discovered the problem when he woke up around 5am and saw emails about how the app was crashing again. He found out the cause, tweaked his proxy and the app started working again at 6.40am.

Don't hold your breath that this will solve the problem permanently though. If SingTel really wants to prevent the app from working, all it will have to block is block data requests coming from the proxy server's IP. Said Chu Yeow: "I'm actually surprised they didn't do that first. That'd have been the hardest problem for me to solve because IP spoofing is not something I know off-hand."


See related stories



0 comments: