It happens in a moment – playing with my mobile before going into a meeting; I put it down for a moment and suddenly my host is standing in front of me. Two hours later I’m desperately searching for my phone. Rush back to reception but it’s not there. Here I am in Dubai and my phone is gone! I need to call my provider to block it but the provider’s number is in the phone. I have visions of my wife calling and suddenly panicking that the Somali pirates have got me – like the time I forgot to call from Dublin and she’s waiting for the ransom demand – still living that one down but that’s what happens when you go on a business trip to Dublin the week before Christmas!
My host comes up with all kinds of useful suggestions about who I should call but since my whole life is in that stupid thing I can’t remember any numbers. All my contacts, email addresses – like I said my life is in that stupid thing!
All it takes is a small distraction and before you know it you’ve disconnected your business critical applications. One small change on the firewall or the router and suddenly you’re users are disconnected. If you’re a service provider just imagine the revenue loss! If you’re an airline taking online bookings, or a bank, or any kind of business suddenly you are losing money and/or customers just because of a momentary distraction.
And like my phone, recovering the situation is not necessarily that simple. Logically I could say that my phone was somewhere, but where the somewhere is, is the other question. You would think that if one of your admins made a simple change to a firewall or a router you could just immediately reverse the process, but in reality it is often like looking for a needle in a haystack. Some organisations have hundreds or thousands of rules in their configurations and they are being changed and modified constantly. Maybe even worse they are being changed at weekends when everything is quiet and then the proverbial hits the fan on Monday morning!
Maybe you didn’t change anything, you only upgraded to the newest release from your supplier. Only to discover that there are problems because the new release has different defaults to the old release. So how do you now validate your baseline configuration against all the devices that have been upgraded?
And of course someone is always looking to place the blame! I have to say my initial suspicion was that my telephone was in the “careful keeping” of one of the guys at the security desk. Frequently I hear network and security administrators complaining that as soon as something doesn’t work the firewall guys are always the first to be accused. Network connectivity problems are some of the most common – and aggravating – for business users.
With distributed systems, as soon as an application does not behave as expected, the firewall is suspect. There are many other possible points of failure – the client application, the user’s PC, intermediate switches, routers, filters, load balancers and the application itself. But, because of its nature (secretive and designed to keep people out) the firewall is a prime suspect. As a firewall administrator, you are guilty until proven innocent – like my thoughts about the guys at security.
You can of course take the usual approach to “solving the crime”. Start to analyse the firewall traffic logs. Contact the user, obtain his IP address and ask him to access the application again. Ideally, this should trigger the connection in question. Then you can review the firewall traffic logs and locate the dropped or accepted packets. How easy this is depends on the tools – unless you have a smart log browser, you may have to work with syslogs. Normally there will be a lot of logs so a filter on the source IP and, if possible, on the destination IP or port will make things easier. But this costs time, money, and above all the user with the problem is not always totally rational in the situation – just ask the guy who was trying to help “this user” find his mobile!
Using a Policy Analysis tool is like having a video of what is actually going on. It simply allows you to create a policy analysis query and you will see exactly where the problem is. Policy Analysis will quickly determine whether the firewalls are allowing the user’s traffic or not. If it turns out that the firewall is, in fact, blocking traffic, Policy Analysis will point you to the rule that’s causing the problem as well as when it was last changed, and by whom. In fact if there was an equivalent “Lost Phone Analysis Tool” I would have been able to identify exactly who found the phone and where they were at that exact moment.
Providing network security for any organisation has become an extremely complex operation involving many infrastructural components and security teams around the world. Regardless of how experienced someone might be it is impossible for them to be constantly up to date on what is going on. At the same time, organisations must comply with rigorous standards of transparency and accountability. Planning, implementing, enforcing and auditing organisational security policies are now business-critical.
Sometimes you happen to be in the right place at the right time and you get lucky. For example if you’re ever going to lose your mobile I highly recommend Dubai as the place to do it. It’s not everywhere that someone picks up a 16Gb iPhone, calls the last number dialed, drives 50Km in heavy traffic, and then waits 45 minutes for someone to pick it up. And he didn’t even give his name – and my wife never knew!
You just might be lucky and spot the problem on your firewall immediately but the chances of doing so are about as slim as being in Dubai when you lose your mobile!