Last week, a bug was discovered in FireFox 3.5 on Windows machines. To help generate a random number, FireFox reads the garbage data from the temp files and cache folders to “seed” the random number generator. Internally, random numbers are used within FireFox for security-related encryption functions. Unfortunately, on Windows machines with large Internet Explorer Caches, or Temp folders, reading the data used to seed the random number generator can take up to 40 seconds, and FireFox is barely usable.

This problem was presumably unanticipated because the development machines did not have large Internet Explorer cache folders, because FireFox developers mostly use…FireFox! It is an unfortunate reality of software development that code which worked well in a development environment may fail in the real world, especially if it hasn’t been tested and vetted against real-world data and usage patterns. The only way to escape the “works on my machine” problem is thorough testing with real-world data.

The same principle applies to testing the performance characteristics of web applications. The test data and scenario must be as realistic as possible to avoid this type of problem. For example, many SQL queries are blindingly fast with 2,000 records in a database, but when there are 2 million records, those same queries may return their results shortly after pigs fly. SQL queries (typically joins) that scale exponentially are a common scenario that can run into this type of problem. Although it can be difficult to get right, the more realistic your testing, the less margin of error for surprises after release, so test environments must be built with a careful eye on realism.

Tweet This Post