What is wrong with e-mail? Can it be fixed? - The Receiver and meta-data

This is one in a series of blog posts about email. The other posts can be found at: [My hypothesis: e-mail is evil and deserves to die!, Problems with email….. I appear to have touched a nerve…, What is wrong with e-mail? Can it be fixed?, What is wrong with e-mail? Can it be fixed? - The Programmers and What is wrong with e-mail? Can it be fixed? - The Receiver and the dreaded FYI.

One of the major problems with e-mail, and for this I blame the developers, is the lack of decent search facilities within the e-mail clients, and what I mean by this is there really doesn't seem to be much auto-indexing, cataloguing and filing of received or sent e-mails - there is no intelligence. In fact, if you really want to make sense of your e-mails, and have a decent filing and cataloguing system, you will have to invest considerable time and effort on your own part to file, sort, and annotate your received mail.

The big problem here is the lack of automation. Yes, you can do some clever things with filters and 'smart mailboxes' but it takes a lot of effort and can be easily broken, as I know to my cost. Also, there is no real meta-data (see Do you speak my language: M is for MySQL, MAC address, Malware, MashUp, Metadata, Middleware) analysis associated with e-mail, and other than some basic routing information, and possibly a spam score, no meta-data is added to the e-mail during the sending process.

Over a period of about 10 years I had built up a fairly sophisticated e-mail handling system. I had various computers and devices all accessing the same e-mail accounts through a mixture of different e-mail protocols (e.g. using IMAP and POP) to keep things synchronised and to generate a central e-mail repository that could be backed up. I used a series of scripts (some server-side and some on the central e-mail repository) to handle spam and the routing of different e-mails to different final e-mail folders (sorting), and finally I used a series of scripts and 'smart folders' to sort, prioritise and in some cases to automatically respond to e-mails. It all worked great. Spam never really bothered me, I had a searchable archive (which worked but could have been better if there was more meta-data in the e-mails) and I never felt overwhelmed by my e-mail. My system had grown up organically as my e-mail volume had increased, and I never had a huge backlog of unread mail, and I never had the 'your inbox is full' message. Then my employer broke it….

So, how did my employer break my system? Simple… Moved me from POP to Microsoft Exchange (possibly one of the worst e-mail systems on the planet) and they also decided to change the e-mail addresses of everyone on campus! This effectively broke my system for handling e-mail as scripts stopped working, and smart folders became dumb. Ever since then I was moved I have been plagued by spam, my automated back-up and searchable archive has not worked, and I keep getting 'your inbox is full' messages. A nightmare.

The move from POP to Exchange has cost me dearly in terms of lost time and in terms of the increased effort required to handle mail. Plus, I can no longer find things in my archived mail as that is no longer automatically updates and indexed. I have lost data….

The other part of the problem is the lack meta-data associated with e-mail. My suspicion is that this stems from the fact that e-mail is a very old technology (it has been around for over 40 years - see The 41-Year History of Email) which comes from a time when people may not have been as concerned as they are now about sorting, archiving and searching data. That is, when e-mail was invented no one realised how big it was going to be. The latest estimates suggest that 144 billion e-mails are sent per day (see Did You Know 144.8 Billion Emails Are Sent Every Day? - there is a great info-graphic on this post) and there are 2.2 billion e-mail users (see Internet 2012 in numbers), that is roughly 65 e-mails per user per day, or 24,000 e-mails per user per year. It is estimated that we spend 11 hours per week on email (nearly 1.5 work days), which equates over 15 work WEEKs per year. That is HUGE, and that is one of the reasons why I think e-mail is broken because it has changed from a tool that was supposed to make life easier, to being a huge drag on our work efficiency.

Is e-mail one of the biggest hurdles to effective business, teaching and research today? Is it broken? And if it is broken how can it be fixed?