On Geographically Distributed Load Testing

Welcome to my first public blog post!

Occasionally I get asked to perform load tests from geographically distributed locations as opposed to within the same local network that the application under test is running in. The idea being that we'll find out how well the application will perform for clients in different locations and do "something" about the results. I've decided to write this blog post so that I can refer people to with my opinions on the matter the next time it's brought up.

Pros:

• If you're extremely short on hardware, companies will run your load testing scripts for you on their hardware.
• You'll test the external interface of the hosting provider you're using. However this could also be a con if your current production environments are using the same interface. Also, it's not infeasible to test this locally. Plus, surely you'd have enforceable SLAs with your hosting provider around this.

Cons:

• If you find a problem outside the DMZ, you probably won't know where the problem was as each packet can take its own route. Even if you did find an issue on a particular node, you couldn't do anything to fix any issue. (Are you going to ring them up and tell them to get their act together? Although I guess there are some companies that have the kind of power to make change here.)
• Unless you're somehow crowd sourcing the load testing, you'll be using a very limited subset of ISPs and potential routes to the destination. Different upstream providers can mean very different routes.
• The only thing you're going to be able to do in order to fix any problems you find with traffic from a given location is to look at optimising your application's network profile which you could do without testing from another geographic location.
• Your application has to be exposed to the internet. Security will be a major concern.
• You'll likely have to work with another company to provide remote hardware or to run the scripts as well which means:
• You shouldn't use real data in your tests as it will all be visible to the third party.
• They'll have your scripts and with that knowledge of your application and how it works. Plus knowledge of any testability facilitating methods you use. (Which should never wind up in production, but I believe we'd be lying if we said we hadn't heard of this happening before.)

So in conclusion, I'm not in favour of going to the effort of running geographically distributed load tests given the low value it offers in relation to running performance tests locally. I find hard to believe an organisation could be so strapped for hardware resources as to make it their only option given hardware's relative inexpensiveness in relation to other resources (e.g. Engineers) and the option of running everything in a cloud environment (I've been using AWS a lot lately and am very impressed BTW).

 Thanks for reading what was hopefully my first post of many!

Regards,

Anthony Fisk