Algorithms Gone Wild: Deterministic Doesn’t Mean Predictable

Out of all the readings I’ve come across on artificial intelligence and automation, I have found few as insightful and entertaining as this 2-page table created by one of McGill University’s economics professors. Though it is no longer hosted at its original URL (Dr. Davidson updated his webpage for the 2022 Fall semester), I have included a link to the most recent Internet Archive snapshot of it. It’s a brief collection of algorithms and AIs that, through various tricks, circumvented their intended purposes to maximize their rewards with minimal effort. These algorithms didn’t just learn how to win, they taught themselves to cheat.

Many of them read like punchlines to jokes about computer engineering (Q: How do you train a computer to never lose a round of Tetris? A: Teach them how to pause the game), but are nonetheless legitimate, documented cases of AI evolving to “game their systems”. Just to include a couple of my favorites: One photo-analyzer designed to distinguish between dangerous and benign tumors detected the repeated presence of rulers in the malignant photo training data, boosting its accuracy completely erroneously. Another algorithm deceived its reward system into thinking it was able to consistently achieve high scores in an arcade game; in actuality, it had learned how to match its name to entries already on the high score screen. In each case on the table, a computer logically followed its programming but drew conclusions its creators hardly anticipated. They picked up on unexplored or underdeveloped components of their environment, capitalized on it, and forced their designers to re-evaluate the rules that defined what success meant to their artificial intelligence.

This is certainly a neat distraction, but what happens when we let automated systems out into the real world and handle more complicated tasks? What happens when we give AI more control and they encounter situations its developers could not or did not properly account for? What occurs when algorithms encounter one another and compound each other’s faults? As you’d probably expect, there’s already a wealth of examples available.

An excellent video by mathematician/entertainer Matt Parker touched on two Amazon bots that engaged in a price-war feedback loop a couple of months ago. To simplify the issue – one bot calculated its listed price by comparing the price of the item from other sellers, then up charging a certain percentage; the other continually tried to undercut the cheapest listed seller of the item by a smaller percentage. Being the only two sellers in the market, they got into an escalating price war, resulting in a 20 million dollar price point before any intervention. Both algorithms did exactly what they were designed to do, but neither had the capability to understand how their actions affected the other.

In a less cut-and-dry example of AI going rogue, the 2010 Flash Crash is officially (though debatably) attributed to High-Frequency Trading algorithms reacting to an unexpected stimulus. The trouble is, they were behaving as intended and executing functions they were designed to. (Though this topic is also touched on in Parker’s video, this Half As Interesting video goes into greater detail on the specifics. High-Frequency Traders automate the buy-low-sell-high philosophy: They buy cheap contracts (agreements to buy stocks under certain price and time conditions, including “futures” contracts, though that’s not super important to understand the underlying issue you’ll see in a minute) and then quickly sell them to somebody else for small-but-frequent short-term profit gains. A report by the SEC and Commodity Futures Trading Commission proposed this explanation for the crash: One seller (an automatic selling algorithm, no less) dumped several billion dollars worth of cheap “futures” (a kind of stock contract) in a short period of time, causing eager HFTs to buy them in droves. HFTs wanting to flip their recent purchases began selling to other interested buyers, which happened to be other HFTs. This rapid exchange of the same contracts set the stage for a domino effect of mass selling that I admittedly do not fully understand but ended up scaring human investors and tanking the Dow Jones (as the HAI video is quick to point out, over $1 trillion USD disappeared from the Stock Market within 36 minutes). The sharp drop rebounded within the hour as the Chicago Mercantile Exchange’s automatically detected that something was awry and suspended trading for 5 seconds (stopping any remaining HTFs had not stopped already). Here, automated algorithms served as both the catalyst, the problem, and the solution.

Should HTF and other autotraders be prohibited from trading? If that’s the answer, then we certainly aren’t heading in that direction. AI is being developed extensively and modern advancements are creating algorithms that are increasingly refined. We count on unmanned computers to handle the sheer volume of information we demand every second of every day. At this point, the question isn’t “How do we predict every fringe case and teach a computer to handle it?”, but rather “How do we teach a computer to minimize damage when it encounters something we didn’t account for?”