Frigging Blackberry service is down again! No emails, no BBM, no browsing. Anything that needs the internet access is not working.

Yesterday it went out at 10am and didn't get fixed until 11pm. Today it went out again at 11am. I am not an expert but for the IT company 99.7% (24h down) uptime is pretty poor. I had my BB for two years now and I can remember only couple of outages that lasted 1-2 hours so they've been pretty good until now but this is a disaster for them! The only press they can get at the moment is a bad one so letting half of the world down (EMEA) is not really a good idea for RIM.

I can't understand why your emails go through their servers...BBM and Blackberry specific services, yes, services provided by others, no...

AFAIK - All mail sent to and from BB's are routed via the BB servers. That's why BB are banned in many countries - their dodgy governments cannot gain access to them.

What strikes me about the outages is that they seem to be focused on the RIM facilities in Slough.

So, either RIM have decided to put all the critical hardware that deals with EMEA in one place with no apparent fail over/back-up system in place or they've lost multiple sites simultaneously... twice in two days. Neither of these can possibly be good for their long term reputation and continuing business, especially from the corporate customers that their services are designed to appeal to.

The Times was saying that yesterdays outage didn't affect corporate customers.

Ah, that might explain why there was an e-mail about it at work today but not yesterday.
Still, not brilliant in any case.

<Tinfoil hat>Maybe the government is testing systems to prevent a repeat of the riots</tinfoil hat>

Not just me wondering about that then?

No, I've heard it elsewhere.

It's an 'agave cactus up the back passage' grade pain in the arse for me. We are currently planning to move an entire department to a new mail service this week. That department includes several hundred Blackberry users. We probably won't be able to move the department to it's new mail service this week, because the BES accounts for those hundreds of BB users won't activate because, like apparently every chuffing other thing related every chuffing other blackberry on two chuffing continents, it has to be trafficed through a data centre in Slough so RIM can deign to allow it to happen. A data center which is apparently staffed by half-wit chimpanzees rather than sysadmins who know what they are doing.

Enterprise? Jesus, I wouldn't give you buttons for it. The BES service costs us 20K a year in licencing. 20 GRAND. It's highly resilient, entirely redundant and will carry on working subject to anything short of an extinction level event, yet RIM who demand all this money from us apparently don't have a sodding clue.

I'm going to get some 'you should have bought a chuffing iphone' t-shirts for the helpdesk staff in that department to wear. And I'm damn well not going to take any of the blame for this thing falling through.

Yours, an unsatisfied Blackberry customer.

Well that aint true as had several sales people yesterday and today saying their BB's keep dropping out re. receiving emails and then a while later they will suddenly start receiving them again - that is using BES.

To get somewhat technical for a minute - when you BES activate an email account, the system tells RIM to send an email with an encryption key to the account's inbox. The BES system spots the message, reads the key and stores it against the user's account. When the phone is activated against the account RIM also gives it a copy of the key and deletes the original, which is why RIM can claim not to be able to read anybody's stuff - the phone and the BES server share a key RIM don't have any more and all RIM are doing is passing the encrypted data back and forth. I suspect in the debacle with the Indian government, what RIM actually did was simply agree not to delete their original copy of the key for Indian customers, so if necessary they can decode the traffic as they pass it on.

The RIM infrastructure is currently so buggered its simply not sending out the emails with the keys in the first place. So the encryption key is never shared. So the phone activation never works. Even if did work a bit, the keys have a very short lifespan (roughly half an hour) so if the email isn't sent out quickly enough, by the time it arrives it's already considered compromised and won't work.

So, possibly it's affecting some enterprise BB customers whose devices have already been activated but lots of 'consumer' customers. I suspect that's because RIM are desperately sacrificing other core services to keep it's existing enterprise customers happy, because that's where the big money is. However, it's definitely affecting BES customers who are trying to add new users to their systems. I'm in touch with our upline support (which we also pay for, wahey) and they're pretty straight to admit there's not much they can do until RIM gets it's infrastructure back up and running properly.

RIM decided to route everything back through them so they could make a profit on every stage of the transaction - server, handset and ongoing licence fee for the service. Charge for the razors and the blades, as it were. This has now come back to not so much bite them in the arse as savage them like an enraged grissly bear.

Jon

We had a notice on our intranet saying that delivery of emails is slow due to backlog of messages on Vodafone's hands (we use BES obviously).
My private BB got some of emails around 6pm but the BIS is still down.

Ironic thing is that I have upgrade my contract to BOLD 9900 and as it is sold out I am still waiting for it to be dispatched. I can't imagine myself getting an Android (not to mention iph*ne) but I do mind paying £31p.m + £99 one off for that BOLD now. I am pissed of with RIM right now and I am thinking about cancelling that upgrade and waiting for it come down in price because that are asking a lot of money and they can't provide a reliability to justify the costs.

I know all it companies have problems but the way they've handled this PR-wise is just shocking

I thought corporates, with their own BB servers didn't use the BB main servers? Or does the local BB server just encrypt and pass the e-mail on to the main server?

It is affecting our BES server today, yesterday it was fine. I've been on the phone to someone at O2 and who gets affected day to day is random. He also said that 50% of users in EMEA are affected.

So I woke up this morning and all my outstanding emails have been delivered to me overnight.

However I still can't browse the net, I've received one email so far today that took 20 minutes to get to me and the message I've sent still hasn't arrived after 90 minutes. So after 13h of outage on Monday we now have 24h outage since yesterday morning. In addition the problems have spread from EMEA to LATAM. Absolutely shocking!!

I called Vodafone this morning and cancelled my upgrade. They didn't say a word...


Companies use BES, average Joe uses BIS. BIS services are down but it is obviously affecting BES users due to sheer volume of data that Vodafone has to deal with.

Even BES users actually get traffic routed via RIM. According to the network diagrams I've seen, the phones don't ever actually talk directly to the BES server, they only ever talk to RIM.

Jon