The File Upload Algorithm of Iptanus File Upload

The first job of the Iptanus File Upload plugin is to upload files, and it uses a careful algorithm to do that reliably and securely whatever the size of the file.

Why stability matters with large files

Consider a 1 GB file that takes a couple of hours to upload. A great deal can go wrong in two hours: the connection drops, the laptop suspends itself, someone closes or refreshes the page, the web server has a moment. Having all of that time thrown away is a miserable experience for the person uploading.

Uploading in chunks

The plugin uploads large files in chunks: the file is sliced into small pieces, the pieces are sent to the server one at a time, and the server puts them back together. It is more involved than a single upload, but it sidesteps the web server’s own limits on upload size and request duration, which is what makes files of unlimited size possible.

A 1 GB file becomes a thousand chunks of 1 MB — in effect, a thousand separate uploads. Starting all of them at once puts a heavy load on the browser, especially on a phone, and not every browser handles that load well; the bottleneck can be enough to fail the upload. The plugin therefore caps how many chunks may be in flight at any moment and feeds the rest through as slots free up.

Auto-resume when a chunk fails

With a thousand chunks, the odds of at least one going wrong are considerable — a momentary connection problem, a busy client, a hiccup on the server. The plugin watches the chunks as they go, and any chunk that fails or takes too long is simply sent again. Nothing is lost, and the upload carries on until the file is complete.

Auto-resume is internal and invisible. The person uploading does not have to do anything, and does not need to know it happened; the only requirement is that the upload window stays open.

The feature has been tested hard: connections interrupted over and over on purpose, the web server restarted mid-upload, files over 2 GB, connections both very poor and very fast. Uploaded files were compared with the originals bit by bit to confirm the algorithm’s integrity.

Security

The algorithm uses AJAX, and is guarded accordingly. Checks run on both the client and the server to verify the user, the referrer and the contents of what is being uploaded, so that neither an unauthorised user can upload files nor an attacker slip a malicious script through. WordPress nonces protect against CSRF, and exposed strings are sanitised.

Older browsers

The plugin still works with browsers that support neither HTML5 nor AJAX. It detects what the browser can do and adjusts, though some features — chunked uploads among them — are not available in that case.

If you would like to know more about the upload algorithm, please contact us.

The Iptanus team