Magicians, marketers, and politicians all count, to some degree, on the belief that people (at least collectively) behave in fairly predictable patterns. Each has their own subculture and, history, and research as a guide, but the core phenomenon is the same. Magicians are probably the easiest to demonstrate – if you have ever been to a quality magic show you have likely been amazed at what you saw. This is because magicians exploit predictable patterns in how people direct their attention and process information, knowledge built over centuries of trial and error. Politicians, rather, seem to go on personality and social instinct, with the good ones rising to the top. Although they are increasingly supported by a campaigning industry which is very data driven.

Political campaigns, therefore, are increasingly like any marketing campaign, which is intensively data and research driven. Not only do marketers read and benefit from the psychological research, they have their own research, complete with their own scientific journals. All of this is premised on the notion that people are far from the unique snowflakes we like to think, and are more like predictable sheep that can be herded. Research generally supports this latter view.

Now of course anyone can rise above the herd with knowledge, critical thinking, and consumer savvy. We can learn the tricks, read reviews, learn something about cognitive biases, and take more control of our own purchasing decisions. While I highly recommend all of this, the result is partly an arms-race where marketers try to get more and more subtle with their manipulation, trying to fly under our critical thinking radar. With the advent of big data, social media, and artificial intelligence (AI) their ability to do so is getting frighteningly powerful. This is not something we should underestimate.

For example, the BBC recently discussed one company that examines hundreds of thousands of news articles and has AI sift through every click to look for patterns in individual behavior. The goal is to learn things about consumers that they might not consciously know themselves. For example, typically people who read articles about celebrities do not read articles about political news. However, they found that Boris Johnson was an exception. They conclude from this that much of the public thinks of Johnson not just as a politician but as a celebrity. This is likely not a conscious thought or decision, it is simply how people react.

This, of course, is not new. You may remember the story from 2012 where Target’s marketers developed an algorithm whereby they could score how likely it was that a customer was pregnant by their purchasing habits. They would then send them coupons for maternity items, hoping to beat out the competition and create a long-term customer. However, they found that some customers were a little creeped out by how quickly Target was able to find out they were pregnant. So Target adjusted further – they sent them books of coupons for random stuff with maternity-related items mixed in, to disguise their true marketing intentions.

This kind of thing is now an off-the-shelf service any company can use. Here is how one company, Resonate, describes its service:

The Resonate Ignite Platform is the only AI-driven consumer data and analytics platform that seamlessly accelerates insights into action. Log in for instant access to hyper-relevant, fresh consumer market research on today’s consumer. Ignite delivers more than 13,000 real-time, proprietary, values-driven insights on 200 million individual consumers.

That’s a lot of information. The combination of AI and big data is getting extremely powerful. The ultimate limitation of marketing manipulation is now – how predictable are we? A century of psychological research indicates that, statistically speaking, very. You cannot guarantee how any individual will behave, just like you can’t predict how an individual molecule of air will behave. But you can predict to some extent population behavior just like you can predict the behavior of gasses.

This predictability with what the science of cognitive biases and heuristics is all about. A simple example is that we have an oversimplification bias – we try to make things simpler so we can understand and remember them. This is a method of reducing cognitive load. One manifestation of this is the left digit bias. We tend to simplify a number to an estimate based on the left most digit. So one hundred and something we think of as 100. This is why prices are so often $19.95 or something similar. Yes, this actually works. We respond to anchoring biases (“now how much would you pay”), to peer pressure, availability bias, to “health halos” or other associations, to celebrity endorsements and appeals to authority.

While this has always been the case, it feels like there is a difference between being manipulated by marketing experts sitting in a room trying to figure out how to appeal to their customers, and being manipulated by a powerful AI with massive amounts of personal data at their disposal. The AI could exploit biases we don’t even know exist yet. They can predict our subconscious preferences that we are not aware of.

But I have to say, while I am concerned about privacy issues, I am not really frightened by the notion of being effectively marketed to. There are a couple of manifestations of this phenomenon that I do find scary. One is the use of marketing to promote pseudoscience, even if this is primarily about selling products and services. I am thinking of things like alternative medicine and organic farming. At first this may just be a way to sell snake oil, but then the marketing becomes the product itself and takes on a life of its own. Now customers identify with an entire philosophy, one based on pseudoscience and that is hostile to science-based alternatives. GMOs were demonized in order to promote the organic brand.

And of course, I am very concerned about high level manipulation being used by politicians, political parties, and ideologies. This is, in fact, exactly what happened over the last two decades. Social media algorithms, looking to maximize click-based revenue, created large subcultures of conspiracy theorists, apparently devoid of common sense or the ability to detect blatant fantasy (at least within their belief system). Just look a Q-Anon – if 20 years ago you boasted that you could convince millions of people that the world is being run by Satan-worshipping pedophiles, to the point that believers would effectively capture a major political party, you would have thought they were nuts.

Toxic populism, the ability to emotionally manipulate large segments of the population, has always been Democracy’s Achilles heel. Now social media algorithms generating rabbit holes of conspiracy theories seems to have democracy in a death grip. It remains to be seen how this will play out.

More than ever, the sustainability of our society seems to depend on widespread teaching of critical thinking skills.