Empathic AI and ethics: notes on RightsCon 2019

Last week, more than 3000 people gathered at Tunis to join this year’s RightsCon. The 3-day event is packed with 450 sessions in total, and there is no way for one to attend all of the sessions of interest. However, I feel I was lucky enough to joined quite a few good sessions, and had many interesting discussions with some very smart people.

One of the sessions I would like to note here is a workshop titled “Rights and Ethics: When technology knows how you feel“, hosted by Gawain Morrison and Ben Bland from Sensum.

The slides they used for presentation are here, and there is a summary of key feedbacks in the group discussion part, both of which I urge you to take some time reading.

Some general impressions

The session is characteristic of the best kind of discussions you could find in RightsCon, with technologists, lawyers, human rights activists, and policy makers sitting in the same room, giving insights from their respective disciplines, while still sharing a common background knowledge to actually get the message through.

My impression is that the majority of the attendees are coming from social science background. Being a data scientist and a software developer, I was impressed by the clarity of their grasp of technological concepts, and their abilities to apply critical thinking and give thoughtful, meaningful feedback.

Here is a comment I especially like from one of the attendees, who I believe is a policy researcher:

Any technology should start with values, and should only serve to empower people, transparently, rather than exploit them for the selfish goals of the provider.

Empathic technology

The technology in discussion here is biometric sensor. Combined with algorithmic models, they form the foundation of Empathic AI technologies.

For the purpose of making empathic decisions with algorithms, several types of data can be gathered from human body, for example:

  • Gaze
  • GSR
  • Thermal image
  • Facial coding
  • Heart rates
  • Breathing rates
  • Skin temperature
  • Voice analysis
  • Ultra-wideband scan

These types of data can be gathered from both contact sensors and wearables, or contactless sensors such as biometric radar. From these inputs, an algorithm might infer several human emotional features such as stress-level, distracted or engaged to the current task, arousal, fatigue. Then algorithms can take these labelled information into account.

These technologies are not only possible, but in fact very close to real world applicable, according to the two speakers’ experiences in the industry.

This is obviously driven strongly by commercial interests, because almost all companies in the world want their products to be more personalized in order to engage their users more personally. They are pursued not only by consumer industries such as automotive, but can also be valuable in places such as medical use, social welfare, customer service, mental health, robot cares, etc.

As Gawain said in his presentation, “these [technologies] are coming, and they are coming fast.”

Some scenarios

The two hosts gave several scenarios for group discussion. These scenarios are largely composed from project proposals they have seen (or turned down) in the past, so they are highly plausible. Some of the highlights for me are:

  • Empathetically responsive cars: Cars with biometric sensors the can “know if the driver is feeling stressed, tired, distracted, intoxicated, relaxed, happy or angry. The vehicle can then respond to help the driver shift into a better mood, or recommend that it takes control of driving – for their safety, comfort and entertainment.”
  • Monitoring staff emotions in workplace: Using fitness bands, cameras, and microphones, “staff will be monitored continuously for key emotional metrics such as stress, confidence, relaxation and aggression. These metrics will be provided to each staff member via a personal dashboard, as well as to human resources managers for them to discuss with staff during their regular.”
  • CCTV monitoring in public space: “The council of a large city has announced plans to add emotion detection software to its network of CCTV cameras in public spaces…. Artificial intelligence software was added to the camera systems to identify potential criminal behaviour before it happens, such as people loitering in areas that are vulnerable to crime. Now the council is planning to add the capability to measure human emotions from facial expressions and body language. With this the council hopes to spot aggressive intentions so it can prevent violent behaviour before it happens.”

These are some scenarios that can be both fascinating and terrifying, and we had vigorous discussions around them. Two clusters of comments I found particularly interesting are concerned with prediction errors, and autonomy.

Prediction errors and false negative errors

Quite a few comments are about the false positive and false negative errors in these algorithms, and what kind of bad outcomes could be resulted from them. To me, these discussions often do not surface when we talk about new technologies, especially when they are presented by companies for commercial interests.

In short, to measure the performance of an emotion-labeling algorithm, we can talk about its precision, which is defined by the number of correct positive predictions (true positives) divided by the total number of subjects predicted to be positive (predicted condition positive). Precision measures an algorithm by asking: of all the subjects that it has predicted to be in an emotional state, how many of them actually are in that state.

The problem is, this is only half, or less, of the picture.

An algorithm can make an error by falsely claim a subject to be positive (being in an emotional state), or falsely claim it to be negative (not being in that emotional state). These predictions are called false positive or false negative.

If an algorithm tries to detect possible crimes with CCTV images, both of these errors are critical to its performance. While false positives, which arise when someone is wrongly accused of being dangerous, concerns us a lot, false negatives are also important. A false negative in this case means a subject has been classified as safe (no criminal intentions) by your algorithm, but turns out to have criminal intentions. If this performance metric is not monitored, it is hard for us to know how many criminals we have missed by following this algorithmic system.

However, for some proposals of algorithmic system, it is very hard to get feedbacks on false negatives. The CCTV monitoring scenario above is one of these cases.

In short, these systems can have big impacts, but are hard to audit to ensure they are doing right.

Discussions about prediction errors are important, especially because it is one of the sources of discrimination and abuse. If there are significant differences in prediction errors among different demographics, the algorithmic system might as a result encode an unfair social structure for a particular demographic, just like we have seen in some real stories.

Autonomy

Another major issue that was constantly brought up during group discussion is that of autonomy. This is a strong value that urges us to be prudent in adopting these technologies. In the words of one of the attendees:

Technology should always provide enablement to the user, assisting their autonomy, otherwise it can be an undesirable substitute for autonomy.

The insight she gave is that, technologies are good when they are solving problems for the users’ qualities of life, enabling them to do things they intend to but unable to do. However the moment when they are making enhancements, they become problematic. Autonomy requires free will, and intention. Algorithms do not have intentions. Making enhancements without necessities are substituting users’ intentions, thereby taking over their autonomy. This is problematic.

This is a consideration that, although from my perspective seems to base a little too much on Western believes, we should all bear in mind when designing algorithmic decision making systems.

Conclusion

This year’s RightsCon consists of quite a few sessions such as this one. It serves as a good platform for stakeholders to discuss about the social impacts of emerging technologies.

From the perspective of a technologist that takes social responsibilities seriously, I am glad that many of the vocabularies and frameworks are shared by the group of people, whether they work in commercial companies, research, development, or policy making. As one of the hosts said in the workshop, all of these concerns have to go into policies, to make sure that this technological development goes right.

Leave a Reply