null is not an object

I stumbled across a blog by Jonathan Boutelle from almost a year ago now about pre-loading data and latency issues in web applications. With the recent rise of Ajax to super stardom(ish) it seems even more applicable today. He makes good points about comparing the value of reduced latency to the cost of download and factoring those against the odds that data will be requested.

Today, however, in the age of Ajax what may be a more important question is how to determine when and how much data to load rather than if one should simply pre-load data in general - of course the two are inherently intertwined. The most important reason for using Ajax in my mind is to be able to pre-loading a small subset of data (data that is going to be needed almost no matter what) and allow the user interactions to define what other data is loaded.

I would say that Ajax is the solution for dealing with large data sets on the web.

For example, if you are browsing your emails in Oddpost you only want to pre-load the latest 10 emails that the user needs to see on the screen (and realistically maybe a few more) and then if the user begins to scroll through their emails, requests to the server need to be made to get more data. To determine how many new email records to request one must consider the rate at which the user is scrolling (this will also determine how far ahead we need to buffer) and the time it takes to retrieve the data - there will be some optimum number of records to request and some optimum record buffer size given the latency of the requests to the server.

Of course this also brings up the issue to bloated XML requests (read SOAP) to get data from the server but I will talk about that tomorrow maybe.