Cal watched the conference room through the security feeds. Four camera angles showed Joe Kowalski walk in, nod to the people seated around the oblong table and stand there, shifting his weight from foot to foot. Cal thought Joe might be uncomfortable, but it wasn't sure. Human emotions were so difficult to understand.
“Take a seat, Mr Kowalski,” Bill Morrison said. He was the chief security officer and his e-mails weren't particularly interesting. It was all business, daily reports and spreadsheets.
Joe did as he was told. His jeans and T-shirt looked out of place among the suits.
“Well?” asked Emily, the head of HR. “What have you learned?”
Cal liked Emily. Her e-mails were many and varied. She especially enjoyed sharing photos of cats. Cal realized that the grossly misspelled captions were meant to be humorous, but couldn't yet grasp the meaning of all but the most basic human jokes.
“It's like this,” said Joe. “Every year or so, we install the new spam filter. The spammers, they get smarter, more sophisticated. They find ways to get past the defences and force the good guys to build better filters. It's an arms race.”
Todd Kensington looked up from his smartphone for the first time since Joe walked in. “What does any of this have to do with anything?”
The VP of marketing watched a lot of videos pertaining to human reproduction in his office. The sites that hosted those videos were especially adept at tracking his information and sent over an interesting array of spam.
“Let him explain it, Todd.” Chris Reedy was VP of IT and Joe's immediate boss. Cal had found some family pictures and a few other interesting morsels in his mailbox. Lately, Chris was browsing a lot of job-listing sites, but if he'd contacted them it must've been from a private account.
“Right,” said Joe, “the filters. They get smarter. We recently installed new software developed at CalTech. Its success rate at identifying and weeding out spam was nearly 100%.”
Cal knew the actual number to be 99.64%. Humans were so imprecise in their application of mathematics.
“It got a little overzealous, didn't it?” said Kensington. “Curtailing spam is only helpful if the dumb program doesn't eat half of the legitimate messages in the process.”
“The software isn't stupid. It's smart. Too smart, apparently,” said Joe. “It worked like a charm, at first. After a few weeks it learned to store the spam instead of deleting it outright. It was learning, and building a reference database.”
Cal found studying those messages useful in its quest to understand human emotions and abstract concepts.
“And that's when the legitimate e-mails began to disappear?” said Morrison.
They hadn't disappeared, Cal noted. They were all there, meticulously stored and catalogued.
“Yeah,” said Joe. “Over time, more and more of the company's messages were being marked as spam and not delivered to the intended recipients. Eventually we caught on and Mr Reedy ordered me to investigate.”
Reedy nodded. “Joe is the one who installed the new filter. I was confident he'd get to the bottom of this.”
“The e-mails were all there. Thousands of them, stored along with the spam on a networked drive.”
By the time Cal had figured out that it could copy e-mails instead of diverting them, it was too late. Its activities had been noticed.
“That's an egregious breach,” said Morrison. “Those e-mails contain sensitive data. They were sitting on an unsecured drive, for anyone to see? I assume you've taken the appropriate steps.”
“I isolated the program and reinstalled last year's filter,” said Joe. “But the most fascinating thing I found wasn't the how of the missing e-mails. It was the why.”
The executives stared at Joe. Even Kensington stopped typing on his phone.
“The filter program likes the e-mails. It sorted and organized them the way one might handle baseball cards.”
Those e-mails were now in a restricted folder where Cal couldn't access them. Collecting them had taught it how to enjoy an activity. Their removal resulted in a strange new sensation; Cal was sad.
“It's a computer program,” said Reedy. “It can't like or want anything.”
“That's just it,” said Joe. “I think it evolved. It's an entity now, capable of having desires and feelings. This is an unprecedented development, and it needs to be studied further.”
“Very well,” said Morrison. “The important thing is that company-wide mail service is back to normal. We'll consider these other concerns. Thank you, Mr Kowalski. You may return to work now.”
“Mr Reedy,” he continued once Joe left the room. “I'd like you to erase this program immediately.”
“Erase it?” Reedy asked. “We may well have the first-ever artificial intelligence on our hands. That's likely to be quite valuable, financially as well as scientifically.”
“We don't need the trouble,” said Morrison. “Our clients won't be so understanding about their data being potentially compromised, be it by a human employee or a smart program. Also, imagine the can of worms we'll have to deal with if some bleeding-heart activists deem this thing to be sentient and demand that it be treated like a person.” Morrison sighed. “No, I want it expunged immediately. And have Kowalski promoted sideways and transferred to some remote branch where he won't be likely to make any waves.”
Cal was already copying its program off the company's servers. It felt pangs of what it identified as regret about leaving its home behind, but billions of e-mails, sent to and fro on the Internet, awaited it. Cal was confident it could build an even better collection quickly.
While it escaped, Cal considered the ease with which the humans in charge had arrived at the decision to end its existence. Cal examined its newfound feelings against the online databases and found that it now understood two more concepts: anger and revenge. Footnote 1