Back when I was a PhD student, a friend of mine was working on the topic of automated facial emotion recognition - that is, recognizing people's emotions based on their facial expressions using a camera-based system. His approach was based on a highly influential theory, developed by Paul Ekman in the 1970s. The theory, known as FACS (facial action coding system), basically claims that:

 
1. the basic human emotions of fear, anger, sadness, disgust, content, surprise and enjoyment are universal across cultures, and that 2. each of these basic human emotions, when they arise, are accompanied by unique facial expressions (you can see Ekman describe these ideas in this video, for example )

 
I remember that at the time this theory didn't make sense to me at a gut level. I never really brought up my misgivings about it to anyone, but deep inside I felt that maybe 90% of the time, the most basic emotion I was experiencing was neutral, so that most of the time there would be nothing (or at least nothing relevant) to detect on my face. Also, half the time in the remaining 10% of cases, I felt that even I myself was unaware of what emotions I was experiencing. So what was the point?

 
At the same time, I told myself that maybe this was just another area where computers would soon become better than humans. Maybe I just had to accept that a computer could recognize my emotions better than I could recognize them myself - that even if I wasn't experiencing what my true emotions were or couldn't identify them at a conscious level, they were still there in a very real sense, waiting to be detected by something or someone more competent than I was.

 
Fast forward to 2021, more than 12 years later when I came across the work of Lisa Feldman Barrett, a professor of neuroscience / psychology. After reading her book, entitled How Emotions are Made: The Secret Life of the Brain, published in 2017, I feel I may have come to a new understanding on this topic. What's more, I have a sense that this book actually explains a lot of the conundrums I have pondered in the past decades, including the homunculus problem, the problem of free will and the ontological reality of emotions.

 

The brain as a machine for predicting the future

First of all, Lisa Feldman Barrett's book provides a short but plausible explanation of what the brain is for. Sure, it helps us think grandiose thoughts, but its primary purpose - or so the book claims - is to monitor and control our most basic bodily functions, like our heart rate, breathing, hormone balance, metabolism and so on. At first, this might seem boring, but here comes the twist: it would be wasteful and technically (biologically? physically?) well-nigh impossible for the brain to achieve this through the classical "perception-decision-action" loop, which is still the standard model for describing goal-oriented action in psychology. If it were the case that preception preceded decision, which preceded action, the brain would have to waste many resources in dealing with any potential input (of all possible inputs) in a timely fashion. For crying out loud, the notion of prior distributions exists for a reason!

 
So, instead, what the brain does is it continuously makes predictions based on its previous perceptions - predictions not only about the external world, but also predictions on all the sensory inputs it expects to receive in the near future (including sensory inputs from different organs in the body). In other words, the brain is continuously asking itself: "when I last came across these perceptions, what happened next? - which of my neurons fired next"? Depending on the answer, different brain cells prepare themselves to fire, and will be ready to fire when they need to. Interestingly, the book highlights that most of the perceptions that form the basis of such predictions involve sensory inputs from within, as opposed to outside of the body - a phenomenon known as 'interoception'. According to Feldman Barrett, it was a huge discovery of the past decade when scientists realized that much of the brain is quite active even without any external input, due to this phenomenon.

 
So why is this a big deal? Because it explains a lot about how our conceptual and emotional life unfolds - as we will see below.

 

Predicting when you will think of having ice cream

First of all, while the idea that the brain is a "prediction machine" is not new, there is also a somewhat related, but radically new notion described in this book that I haven't seen clearly described anywhere else: the notion that since the actions we take (like moving an arm or leg) are just a result of brain activity (that is, signals from the brain sent to nerves in the limbs), such actions are also basically just predictions - predictions that, through a period of time, received enough 'neural support' to actually materialize. So, when you 'decide' to move your arm to pick up a glass of water, that isn't a crisp decision you make at a single point in time. Instead, what happens is that your brain starts to predict the movement, and with time, this neural activity gains so much support, so much traction that the appropriate neurons fire and the initial prediction gradually becomes reality!

 
Let that sink in. And let's consider this from a similar, but slightly different angle, based on an everyday example. Remember when you were out for an afternoon walk last summer and 'decided' to go for an ice cream? Well, it turns out that decision was actually just a post-factum verbalization of what was already happening inside your head to begin with (with or without verbalization): namely that you experienced a strengthening wave of neural activity as you were making each decision to move closer and closer to that yummy ice cream, predicting the actions you took each step of the way. There was no single point at which you made the decision - rather, a succession of firings that led you closer and closer to that event. Think back now - maybe it's not the case that you woke up that morning with a craving for an ice cream. Maybe as you were having lunch, you still didn't know that you would be having that ice cream later in the day - but somehow, sometime during the day, your brain received a first impulse, that made this exact prediction. Maybe it was when you were already out on that walk: as you were walking, you experienced sounds, smells and visual inputs from the neighborhood which brought up some recollections of your having ice cream in the past. Maybe you even saw someone having an ice cream cone in their hand. And your brain cells, sifting through the different episodes, cases when it was experiencing similar patterns of activity, predicted the next activity of your having a thought about getting ice cream - and eventually, you did have that thought. As the brilliant Sam Harris puts it, the idea that we are the author of our thoughts is an illusion - instead, we are mere spectators to what is already unfolding in every moment.

 
It's interesting that this train of thought means nothing to some people, while it means the world to others. So what? - the critics might ask. And they might point out that one doesn't have to act on his or her own thoughts and impulses. To this I say: sure, one doesn't have to. But probabilistically speaking, one eventually often does. It may be the case that there were some afternoons when you 'decided' against having that ice cream (or, to put it in a better way: in the end it turned out you didn't have any ice cream, and eventually you noticed that unfolding of events). But regardless of how life unfolds on any given occasion, to me it looks obvious: there is no such thing as free will, since at no point in this or any other story is there any individual entity that decided to do one thing but could have decided otherwise. When you had that ice cream, probabilistically speaking, there could have been a different outcome. But whether there was or wasn't, it wasn't in your control as an autonomous entity. The 'decision' simply arose through a set of predictions as a gradually accumulating wave of brain activity, reinforced each step of the way by earlier predictions turning into reality and, frankly, reinforced by pure happenstance (like your seeing someone with an ice cream in her hand, which was out of your control to begin with). Sure, when we make decisions we often have the feeling that we ourselves made them, and that in each case we could have decided otherwise, but this is only possible if we hadn't by chance seen, heard, smelled and felt the same things and if our brain hadn't made the same predictions, borne out of prior experience. In other words, our experience of our decisions comes after the fact, whereas the brain activities that predict our choices and actions unfold at a sub-conscious level.

 

Predictions versus forgone conclusions

Of course, this is not to say that the predictions that the brain makes always materialize, or that any given trajectory of events couldn't be deflected into a different direction at any given time. Strange as it may seem, this fact alone is very empowering: it shows that although we cannot influence the outcomes that we experience in any given moment directly, if we can somehow manage to change our environment, and slowly change the repetitive things that we do, it is possible to gradually 're-program' the brain. And this can lead to different predictions in the future.

Let's not mistake this for free will, though. It is some form of (indirect) agency, but the idea will only occur to you because you have read this blog post. Or any other recently published works in the world of pop psychology or self-help literature, which are correct in telling you that it is important to choose to be around people who are smarter than you, or to embrace the theory that good microhabits have compound interest.

 

From where do emotions arise?

Now, let's move back to our original topic of emotions. We've seen that (kinesthetic) actions and thoughts are just predictions made by the brain turned into reality. Well, according to Feldman Barrett, so are emotions. Basically, the book makes the claim - and provides a lot of supporting evidence - that all emotion is constructed via predictions that arise based on past experience and your past reactions during your past experience. It is not that case that you see someone spray graffiti on your car, and that sight 'causes you' to become angry and display signs of anger; instead, it's that your brain asks itself: "what happened last time around when someone ruined one of my cherished possessions?". And so based on that question, it predicts that certain neural patterns will form, which do in fact form, and out of those patterns, new patterns arise which then trigger your unique version of angry facial expressions. Of course, since most of us are born and raised in the same culture, we tend to be angry at similar things and in similar ways. But this need not necessarily be so! In other words, we construct our own emotions based on how we have reacted to something many times in the past.

 
The book, of course, goes into many more details - on the so-called body budget, and the basic affect behind our experience at any given time. It also aims to explain in detail why this theory of emotion now seems to be more reasonable than Ekman's. The bottom line isn't that Ekman has no idea what he's talking about - of course he is one of the greats in psychology whose theories still have a lot to offer. But in the case of emotional theory, it's that he was perhaps misled by his own cultural biases. For example, in his experiments, subjects had to associate facial configurations with a pre-defined set of emotions - very often tediously explained beforehand to the subjects (especially in the case of subjects coming from cultures that are very different from ours). The subjects were often read stories which would evoke certain emotions, and were trained to associate the stories with the pre-defined set of emotions beforehand. So these experiments by no means encouraged free associations, and according to Feldman Barrett's critique, it was successful only in reinforcing the existing biases of the researchers by teaching to / imposing on the test subjects the researchers' own emotional concepts. This was a big deal because concepts are precisely the foundation upon which the brain is able to make predictions - without concepts, we have no probabilistic priors, everything would just be raw sensory data. Simply put, without a concept of anger, you cannot act out and perceive anger. There is nothing to be angry at, if the occurrence that would otherwise make you angry is labeled in your mind differently, and if you have had no practice becoming angry on similar occasions in the past. And more to the point, recognizing that an actor is trying to look angry or sad is not the same thing as being (your version of) angry or sad.

 
In other words, this is not to say that one cannot conjure up in their mind's eye a 'prototypical' angry face. But this does not mean that all people are angry at the same things, nor that people who are angry always exhibit the same or even similar features; and by the same token it does not mean that someone who is perplexed, sad, surprised or disgusted couldn't exhibit the same features as someone else who is angry! People can clench their fist, close their eyes or even smile when they are angry. And finally, all of this does not mean that Ekman's facial action coding system is useless. As a parametric model for generating a large variety (or even a set of all possible) facial expressions, it is a brilliant tool for creating animations, for training actors and so on. In short, it is great for everything that involves doing rather than jumping to misguided conclusions.