Stanford HAI 2019 Fall Conference – Race, Rights and Facial Recognition

Stanford HAI 2019 Fall Conference – Race, Rights and Facial Recognition


– My name is Daniel Murray. I’m the executive director
of Stanford’s Center for Comparative Studies
in Race and Ethnicity. We’re a research and teaching center here committed to advancing racial justice through the comparative
understanding of race. Before we get started, I’d like to begin by acknowledging that the land that we are
standing on here at Stanford at the Hoover Institution, indeed in Silicon Valley, is
the ancestral, unceded land of the Muwekma Ohlone tribe. I think it’s important
for us at our center and for us here at Stanford to recognize that every member of the community has benefited
and continues to benefit from the use and occupation of this land. I’d also like to thank HAI, the organizers of the conference, Rob Reich, Fei-Fei Li, John Etchemendy, and Celia Clark. Thanks for putting together
a great couple of days to talk about some very important issues here at Stanford and in the world. At CCSRE, our center, we’ve recently launched a
race and technology initiative of which this panel is a part. And for this work we’ve recently won a national challenge grant, competition for public interest technology hosted by New America, funded through Ford, Hewlett, and Rick’s foundations, and through this an in
partnership with HAI and other Stanford partners, like the Stanford
Digital Civil Society Lab and others, will be hosting a
cohort of practitioner fellows starting in January throughout the year. We will have a conference in Spring on digital technology and racial justice, and an array of other programs, research and educational opportunities throughout the year. So we hope you will join us for that. You can find out more
information on our website and follow us on Twitter and
social media @stanfordccsre. Now to our panel. In our discussion today, we will focus on racial justice in the development and
deployment of AI technology, with some specific focus on facial recognition technology. As we’ve heard throughout the conference, there are dangers in biased data, technology, and implementation. Joy Buolamwini, who unfortunately is unable to join us today, but, hopefully some of you were able to hear her talk yesterday, if not, can find it online, illustrated how facial
recognition technology works great, so long as you look like me, in her words, “pale and male.” Now at present, this flawed technology leads to challenges from inconvenience to wrongful arrests to much worse. So do we solve this particular problem through better data sets? But this raises concerns too. In one troubling recent example, Google contractors in Atlanta attempted to address this problem by capturing the faces of
homeless people of color without their consent. But the problems are not
only about flawed technology and better data. Increased surveillance, particularly in communities of color that are already over-policed, raises equity and civil liberty concerns. And emerging technologies
like affect recognition, which purports to
determine emotional states, raises another host of concerns. This technology can be used in classrooms, workplaces, and hiring to determine whether someone is engaged
and other emotional states. Stanford’s own psychology
professor, Jeanne Tsai, one of our faculty affiliates, and others have shown how
culturally-specific emotion is. How do we define engaged
as an emotional state and how does that relate to one’s ability to do a successful job? Will it lead to more disciplinary action for children of color in classrooms? These are some of the challenging concerns that the new technologies raise. They require a diverse interdisciplinary and cross-sectional debate as we design and implement
new technologies, as we determine the ethical
and legal guidelines, and that’s what this
conference and this panel hopes to do today. So I’ll introduce each panelist in turn, and they’ll present for
about 15 or 20 minutes on their work and research to help ground our discussion. Then, we’ll discuss as a panel and open for Q and A with all of you. So, with that, I’d like to introduce our
first speaker, Matt Cagle. Matt is a technology and
civil liberties attorney at the ACLU of Northern California, where he works on digital rights issues, including privacy,
surveillance, and free-speech. Matt focuses on ensuring
modern digital systems are publicly debated and implemented with equality and justice in mind. This includes work on the use of surveillance technology by local police and the promotion of best practices for online platforms. He was instrumental in the recent ban on the use of facial
recognition technology in San Francisco in Oakland, as well as the recently
enacted California law that blocks police from
adding facial recognition technology to body cameras. He wrote and helped to pass those laws. He’s a graduate of Stanford
Law School as well, where’s he’s a member editor
of the Stanford journal of civil rights and civil liberties. Join me in welcoming Matthew Cagel. (audience clapping) – Thank you. Hello, everyone, and
thanks for having me today. I want to thank the Stanford institute for hosting us today and bringing together
this amazing conversation. So my name is Matt Cagel, I’m with the ACLU of Northern California, based in San Francisco, and today I wanna talk about the work that the ACLU has been doing, with our coalition partners, to push back against facial recognition and really sound the alarm
about this technology and its impact on our society. So, with my time, I wanna broadly go over,
sort of, the concerns that an organization like The American Civil Liberties Union has about facial recognition and other biometric surveillance, and then talk about two
ground-breaking laws that the ACLU has led coalitions to pass this year, here, in California. And in doing so, I want
to persuade this audience, and folks watching online, to support a strong, restrictive measures, like moratorium, or
bans on this technology. And that’s for really three simple reasons to sort of outline where
I’m gonna go in this talk. The first reason is
that facial recognition poses an unprecedented
threat to civil rights and civil liberties. This is the technology
that gives the government power it’s never had to track people, to identify people, and to effectively control people with automated surveillance. The second thing is that, because this is so
powerful and unprecedented, we think it’s very radical and dangerous for us to be allowing
governments and companies, for that matter behind them, to be testing facial recognition, affect recognition, and
biometric surveillance, on the public right now, which is what is happening in many places where we lack any sort of regulation. And then finally, it turns
out that policy makers and the public strongly
support bans and moratorium on facial recognition and
other biometric surveillance and they’re passing them. So this is where the energy is right now and I’m gonna talk a
little more about that as we go forward. So, the slide clicker is. Early last year, I was on Amazon’s website for their cloud services product, AWS. As one would do, I was
browsing in AWS’ website, and I noticed they had a product called Rekognition with a K. So, that seemed interesting. I didn’t know Amazon had a
facial recognition product and all throughout their page there, they were quietly, sort of hidden in plain sight advertising a product that they said could identify and track hundreds of faces in a still image or in a
video that could track, quote, “persons of interest,” and that on the website, they also talked about their ability to, they touted a use case
that was to add this to officer-worn body cameras. That was the first use case, if you think about software makers, they hold up sort of
your primary use cases. This was right at the top. And that rang an alarm bell for us because we have been working
with our coalition partners to make sure that body
cameras are deployed fairly and that they do their job that they were promised to do which is to hold officers accountable and make them transparent for the public. Yet, Amazon, on their website, was advertising the use of body cameras for surveillance. And so that just showed to us, it suggested a misunderstanding of how, one, policing
worked at the local level, and two, how surveillance
could impact communities that the police are supposed to protect. These are devices that were promised to make police more accountable in the wake of well-publicized shootings of black and brown men
across the United States, and so seeing that set off alarm bells for us at the ACLU. So we worked with two affiliates of the ACLU in Oregon and in Florida. We crafted and we sent
public records requests. It’s something that all people can do. You don’t have to be a lawyer to do this. We sent it to Orlando Police Department and a prominent sheriff
in Washington state or in Oregon, excuse me, that we knew to be using this software, because it was listed on Amazon’s website as a customer. They were listed as a
customer on Amazon’s website. So we got back a lot of records and we were surprised to learn that Amazon wasn’t only providing this technology, but that it was providing
technical support, it was providing free
engineering resources, it was even soliciting ideas
for new product features and inviting a person who
was sort of a prominent user to Amazon’s conference in Las Vegas, where they could talk to
other potential customers. And what also striking
about this was that, and it is, very cheap. So for the cost of a large cheese pizza, a law enforcement agency now can upload hundreds of thousands of images of mugshots or DMV photos or whatever sort of, source photo they
have, create face templates, and a matching database that can be used to identify persons, or
attempt to identify persons as they’re arrested,
but more concerningly, or a sort of separate concern, is that that sort of
data set in that system can be turned and applied to body cameras roaming around our streets. It could then be applied to public-facing video cameras that are increasingly in our neighborhoods and that have higher and
higher quality video footage and that could allow for the establishment of more sort of, eyes watching us as we go down the street. So, the cost matters in
part because, you know, the limits on government surveillance, some of those limits
have traditionally relied on the cost of surveillance. It was something that
was sort of infeasible 20 years ago for law enforcement to watch where you go throughout your day or week continuously, they
would have had to devote personnel resources,
potentially, a van and equipment to follow you and there
would have to be a sort of, investment of resources
to track where you go. Facial recognition helps
to eliminate that friction. So, I just want to also highlight this, which was in the records we’ve received from one of the law enforcement customers. I think this is funny because
the internal email chain and the voice of this sheriff’s employees, well they were actually correct. The ACLU was concerned and
we started a public campaign to demand that Amazon stop providing facial recognition to
government entities after that. So, yesterday Joy talked about the bias and inaccuracy in facial
recognition technology and it’s true that inaccurate systems will result in false arrests, they’ll result in encounters
between law enforcement and the government and the public that should not occur, and they will result in the wrongful death of people in our society
and in our communities. But, even perfectly accurate facial recognition systems pose a uniquely dangerous threat
to our democratic system and to our civil rights and liberties. So, facial recognition, we like to call it face surveillance because that more accurately conveys what this actually functions
as in public spaces. That means the power to
create a profile of somebody, to identify them without
lifting a human finger, to create a persistent record of people’s movements across town, their interests, their
potential emotional state or at least their
perceived emotional state, there are many products now that purport to identify things about our interior, simply from exterior features, which is snake oil at best in many cases, and very dangerous at worst. So, what else does it mean? In this world, if you have facial
recognition in a community, that means the government has the power to know who you are
without asking for your ID. That means the government has the ability to know where you worship and who you worship with without visiting your church. That means the government has the ability to log and create a roster of you and your political associates, whether you attend the Women’s March or a march for The Dreamers, and know who you were there with and when you were there, all without infiltrating your
political associates or group. That’s not just a creepy set of powers that the government gets
with facial recognition, as I was saying earlier, it’s very unprecedented in many ways. And of course, with the ability to impose more surveillance means
the ability to impose more political control and social control. This is a screenshot,
as was the last slide, from a great series The
New York Times has done on facial recognition showing
how it’s being used in China. I wanna point folks to this slide from one of the features they did, which I think really illustrates something called the chilling effect that facial recognition has. It’s not only the government’s ability to know more about your private life, but it’s the government’s ability to impact your ability to
freely express yourself, to freely associate, to freely exercise your own religion. And so, in this story, the author talks about how after the Chinese government
set up facial recognition in a city square, or town square, where they had previously
been prayer happening at all times of day,
lots of civic activity, after the system of facial
recognition was established, the author comes back
and he sees, you know, a desolate city square
where no one wants to go to. Now China obviously has a
different system of government and they are, in many ways, much further ahead on their
use of face surveillance in public spaces, but the chilling effect that is occurring in this sort of situation right here, is not going to be, that’s
not uniquely a Chinese thing. In the United States,
under the first amendment, one of the main sort of principles or concerns motivating our
first amendment protections is this idea that when people are watched, they change their behavior. When people are watched,
they feel intimidated and that is exactly what will happen if we establish facial
recognition in public spaces. Many of us will feel less comfortable going to a public protest if we know the cameras there, whether they’re on officers
or on streetlights, are going to potentially log who we are, whether we’re there, and
who we’re there with, and that’s going to be true even if we have laws in place that say, okay here’s a complex bureaucratic process the government has to go through before they can use facial recognition, they have to mark X, Y, and Z boxes. That chilling effect, I would argue and the ACLU would argue, will exist even if that bureaucratic
procedure is in place behind the cameras,
because what you will have are cameras that were previously asleep, right now when we walk down the street, we don’t think that camera
is actively watching me, we’re gonna have cameras that were previously asleep
coming awake in a new way, and that is going to have an impact on our first amendment rights. And in many communities, it will be safe to assume the worst, that people are actually watching, that government agents
are actually watching, because the history of
policing in the United States is the history of racist policing. It’s a history of policing
being disproportionately aimed and policing resources being
disproportionately aimed at communities of color,
at immigrant communities, and at activists who seek
to speak truth to power. So, how do we know this? Well, these systems are
vulnerable to mission creep, is one thing we know as well. So what does that mean? That means when we build a
facial recognition system, we create a system that is
vulnerable to expansion. This is because, in part, to create a minimally viable
facial recognition system, you need a large set of matching photos in order to identify a
person who’s arrested when they come into the city jail. You need plenty of photos
of identified people in order to find a match. Well with facial recognition,
a few lines of code, a few cameras can be established
and added to your system, and suddenly, you have a
city-wide surveillance network. The friction to creating
and expanding these systems, is something we like
to call, mission creep, that friction is even lower in the facial recognition context than it would have been, say in the context of buying drones, where you have to spend
a few thousand dollars each time you buy a new
drone to expand that network. How do we know that this sort
of system will be abused? History tells us that facial recognition and other sorts of surveillance
have and will be abused. So this first example
is from a set of records that the ACLU released earlier this year and what this was is an examination of license plate readers. These are technology that
American Law Enforcement have been using for over 10 years now. They look at license plates, they snap a picture and
tag the car with a GPS, many cop cars have these and streetlights, and over time, the collection of this data allows for the collector
to know more and more about the locations of drivers. The ACLU and many actors in civil society have been sounding the alarm
about license plate readers for many years now. But then last year, early last year, we learned through records requests and reporting by The Verge that license plate reader systems, that were previously
pitched as a way to find amber alert vehicles,
to find stolen vehicles, they had expanded their use, and suddenly, ICE had purchased a subscription to one of the largest
databases of driver locations in the United States. And so with that, ICE gained a new ability to target, locate, and deport immigrants living in our communities. We obtained records showing that ICE was talking about somebody’s habits. They were searching the system to find out somebody’s habits of
where they went to work, where they went home. We saw the tracking of the
associates and family members of immigrants using these systems and these systems, this ALPR system, license plate reader system, was originally pitched for
something completely different. Over time, the system grew,
and the new uses were found, and profitable ones at that. We’ve also seen it in the facial recognition context already. So, in 2016, the ACLU, we obtained records showing the use of social media monitoring software, so it’s a software that
scrapes social media, and then provides this sort of Google Maps sort of interface to law enforcement so they can conduct investigations. We uncovered the prominent
use of this software and in some of the records, we found that there was a
use of facial recognition. Law enforcement in Baltimore pulled images from social media, they ran facial recognition
against those images, and then they went and arrested
people who were protesting at the Freddie Gray protest in Baltimore. So this is a protest following the police killing of a black man. Law enforcement used facial recognition to target protesters at that
event and to arrest people. And so, we don’t need to speculate, we don’t need to theorize,
to understand the sort of, writing on the wall when
it comes to what happens when we build sophisticated
surveillance systems and we let them grow and
we let them be deployed against our communities. So, the problems in facial
recognition are not just limited to the technical problems
that Joy discussed yesterday, you have your inaccuracies
in prominent products, there are bias issues in
many prominent products. We can point to the problems
in facial recognition even if it’s perfectly accurate. I talked about the privacy issues, the free of speech and
first amendment issues that exist even if these
technologies are perfected, even if they’re perfectly accurate. Another set of problems that exist, even if the technologies
get to a perfect state, is misuse by humans. Facial recognition like any
other surveillance technology is deployed in an imperfect world. Even perfect facial recognition
is going to be deployed in an imperfect world. We see this in Florida
where there’s a case brought by a man named Willie Lynch. The ACLU and the Electronic
Frontier Foundation have filed a brief on his
behalf because he was not told about the use of facial
recognition in his case to identify him when there
lacked an eye witness. Until just before his pre-trial hearing, he was not provided crucial information about the use of the facial
recognition technology. He was not provided the
other set of possible matches that were given to law enforcement before they went and arrested him and placed him in prison for eight years, and he wasn’t provided the opportunity to scrutinize the algorithm, to cross-examine the law enforcement agent who operated the system, to ask questions about whether they were
even trained properly. So, again, perfectly
accurate facial recognition and I’m not assuming this
particular product was, but even if we say this
is perfectly accurate, there are tons of problems there. Another situation in records uncovered by the Georgetown Center
for Privacy and Technology. Earlier this year, they released a report showing law enforcement simply
feeding in the wrong data into facial recognition systems. So in one instance, the NYPD, not being able to find a match based on the blurry image on the left, gave the facial recognition
system a picture of Woody Harrelson who they,
in there sort of, guesswork, decided was the closest-looking person to the person in the blurry image. Facial recognition systems compare pixels, and so, what the facial
recognition told them was not based on the
sort of human analysis of well does this person look alike? How much do they look alike? In fact, they just took the results, and then they went and
arrested that person. So, what are we doing to
fight back and to push back? You know, policymakers
and the public really do understand these threats that
I’m discussing right now. Facial recognition
resonates with the public and policymakers in
the way I’ve never seen a surveillance system
resonate with the public or policymakers in my
seven years at the ACLU. So we know this is resonating, like I said earlier, because of two recent
campaigns that the ACLU has lead here in California. So, in San Francisco, we put together and led a coalition of 25 civil rights groups to pass what is the first ban on the government use of facial recognition by
governments in the United States or perhaps the world. Inside of city hall, I
think this is a really important point when we
talk about this sort of, where regulation should fall. Inside of city hall, the
ban was uncontroversial. What was controversial was this idea that policy makers, or excuse me, police and manufacturers
themselves, acting alone, should be the ones to decide whether a new surveillance technology is deployed against the public. That’s radical. What is reasonable and
makes complete sense to policymakers is the idea that the community should decide and residents should decide whether or not a new surveillance technology
is deployed against them and what the sort of proper
public safety framework and solution should be. That argument, that
sort of frame resonated with policymakers and this
passed by a 10 to one vote. It was nearly unanimous in San Francisco and there were hardly any sort
of public discussions about the sort of, ways to weaken
the facial recognition ban or anything like that. It was a very sort of, I
think consensus based approach to this regulation. And so, that has triggered a nationwide movement in many ways, Oakland and Berkeley have
passed their own bans, there’s a ban passed in
Somerville, Massachusetts, led by our allies and partners
at the ACLU of Massachusetts, and there are bans proposed
now in Portland, Oregon, we’ve heard potential bans
in Southern California, in New Jersey, and other
cities in Massachusetts. So, building on the momentum
from the San Francisco ban, we also led a coalition to
pass a statewide law this year that addresses a sort of pressing issue in the facial recognition and
biometric surveillance space. So, again I talked about body cameras and how they were promised to communities for keeping officers and police, officers and sheriffs transparent and accountable to communities. But those cameras are
at risk, in many places, of being turned into surveillance devices roving through our communities. And we had seen chatter
from prominent vendors talking about the potential
addition of facial recognition and other biometric
surveillance to these cameras. We had seen patents filed, and so, we banded together with
the coalition of partners across California working on
an immigrant’s rights issues, working on behalf of the unhoused, working on behalf of criminal defendants, and we brought a bill forward
that said law enforcement in California may not use
biometric surveillance which includes facial
recognition, affect recognition, voice recognition, recognition
of the unique way you walk, they may not use that in
conjunction with officer cameras, so the cameras attached to
them or the cameras they carry. This bill was just signed into law by Gavin Newsom three weeks ago. And so, it goes effective on January 1st, and that would make California
the largest state yet to pass this sort of measure, arguably the largest market
for officer-worn cameras, and so we’re really excited
to see it made effective here in California. And so, I’ve taken a
bunch of time I apologize, but I just wanna conclude
by making the point that I think is often lost on those of us working in this space, which
is that this technology, whether it’s facial recognition or another sort of surveillance technology being deployed or pitched
in our communities, it is not inevitable
that facial recognition has to be deployed, simply because the
technology has improved. We as a Democratic society, not technology companies,
not police, acting alone, we’re the ones who get to decide whether and when these technologies are ever going to be deployed against us. And I think what you saw in San Francisco and in other local communities
as well as California really does show that
democratic policymakers or democratically-elected
policymakers I should say, because these were bipartisan efforts are listening, they’re hearing, and they’re excited to
bring forth legislation that draws a line in the sand and at a minimum presses pause on the use of these dangerous new technologies. And just to close, I think I wanna highlight
that none of this would have happened if we
had not banded together with allies who don’t traditionally work on privacy issues to work
on these legislations. In San Francisco, at the
California State House, we brought together
coalitions, like I said, immigrants’ right, racial justice, the unhoused,
Muslim-Americans, communities who could stand up before policymakers and explain the impacts
of historical surveillance and how it harms people. That resonated with policymakers in a really meaningful way, because on one side we had
sort of theoretical benefits of how infinitesimally, rare situation facial recognition could
help address potential crime. On the other side of that, we had actual people who talked about what happens when law
enforcement infiltrates a mosque and uses a license plate reader to scan everyone who’s parked at the mosque. Or people of color who could talk about the use of social media surveillance to infiltrate Black Lives Matter and seek to shut down those movements. Those voices carry the day
and they are why California right now is really
leading on this legislation and this sort of topic. So, thanks for having me, and I look forward to the discussion (audience clapping) – Thank you, Matt. So Matt has grounded our conversation in facial recognition
technology, thank you. Our next panelists are gonna
expand our thinking on this. To think about biometrics and other ways in recognition more broadly. So first up, Dr. Bridget Algee-Hewitt is a senior research scientist at CCSRE, Center for Comparative
Studies in Race and Ethnicity, and a biological anthropologist with over 20 years of experience
in social justice work in the U.S. and abroad. She merges machine
learning, data modeling, and hands on laboratory
approaches for the estimation of the personal identity parameters, like sex, ancestry, stature, and age, that are essential components of the biological profile used
in forensic identification of unknown human remains. She also applies cultural anthropology and critical race theory
to understand how biology and culture interact to shape
personal or group identity and reflect individual experience. Dr. Bridget Algee-Hewitt. (audience clapping) – Good afternoon. First, I’d like to thank
HAI for having us here today and I’d like to thank
Daniel and the center for inviting me to speak. I have to also say how excited I am to have such a diverse panel. I think we talk in academia
a lot about the value of interdisciplinarity
and I think more than ever dealing with issues
like facial recognition where it involves those
who are the academics, who are developing the methods, those who are the computer programmers who occupy that liminal space
between academia and industry, and then those who are the policymakers need to be all agents
of discussion and change and development as we move
through the thorny issues and come to an understanding
of what we don’t know and then how we ultimately
deal with what we do know once we reach that point, in terms of regulation,
policy, and moving forward. So this is, I think, a very exciting letter-shed
moment to have all of us here to discuss collaboratively,
drawing upon our strengths as policy and academic professionals. So, Daniel asked me to
give an overview of my work so that in case you’re
not all familiar with the minority of us who
do this kind of research. So, I have a multi-pronged
research program as Daniel said I’m a
biological anthropologist, which means I study
what it is to be human, biologically and culturally, and I use that information
distilling it down to make identifications in the individual. So we look at large, worldwide patterns, develop mathematical models for understanding those patterns and then use those
models on the micro-scale for coming to make an
identification of the individual. So that means that a large part of my work is situated within the field
of computational biology. I cover both the skeletal, genetic, and the phenotypic areas of
what it is to be a person, and I combine those with
demographic information, so how we see ourselves within society, so what our labeling systems are, and also those numbers
within society that help us situate the math of doing identifications, so population density,
issues of diversity, are all very important when we think about developing models. And then as I said, I look at that from the global perspective to understand how we vary. We’re all different in
terms of populations is what we talk about as anthropologists, which means we have differences that are often geographically located. But we also know that in our genetic, in our phenotypic, those are what make up how we look, or what is expressed by our genes, and in our skeletal system. We have commonalities and those commonalities form clines, so we have distributions, so while we’re all very unique, and then combined as populations
that cover the globe, globally we’re all very similar as well, so teasing apart those
different levels of identity is important and that’s how I’ve come to AI and machine learning because it gives me a very powerful tool for developing much more nuanced models that allow for that type
of analysis to happen. And then, in addition to
doing the mathematical work on that more theoretical perspective dealing with lots of big data sets that have been collected
over long periods of time, I also deal with the immediacy of the identification problem. So, I merge what we know
from human variation and from these large data sets that cover that swath of diversity
with what we can gleam from individuals. So what we study in the lab when we’re given a forensic
case so that micro-analysis and the goal there is to use these models, or develop better models
that take that information that we have in the laboratory
and we merge them together so that we can make a
positive identification for an individual for whom we have nothing but a DNA sample, for
example, or their skeleton. So it’s to bring social justice to the identification process. So, the ultimate goal as Daniel said, is to estimate those identity parameters so that we can say from all
that remains of your biology, the things that would have
been important to you in life or socially important in life and the things that then
would have then been reported on a missing persons report, or told to law enforcement, or a NGO if you’re working in a humanitarian context, about you. So I always tell my students
that we like to think, we wanna relay what’s
on your drivers license. We wanna say all those phenotypic and demographic information
that may help define you as who you are so that we can
take these unknown remains and match it up with the known individual and ultimately serve social justice by returning those remains to the family. So, I also then work in
the medical legal system, I’ve taught and aided law
enforcement in understanding how this process works and being able to produce the reports that can aid them in their investigation, and then I also serve
as an expert witness, especially in asylum cases and in the documentation of trauma and humanitarian aid contexts. And then, as hopefully you’ve
gleamed as I’ve been talking, social justice runs hugely through my work and is a great motivator for why we do these kinds of analysis, model building, it’s what inspired
development that applies AI and ML technologies. So, I apply cultural anthropology
and critical race theory, as Daniel said, to
understand that intersection between biology and culture, because if I want to speak to who you are and how your family or your
friends would have defined you, then I need to speak to all of you, and as a person, you’re both your biology
and you are shaped by the culture in which you exist. So important to that, then, is understanding that interaction between what is innate in your
skeleton or in your DNA that reflects your ancestry and how that can be translated
into more social categories, such as geographic
categories for populations, or more commonly, what gets translated
into race and ethnicity. So there’s always that thorny allusion between biology and culture and its something that
needs to be probed deeply and treaded upon lightly. And my actual research integrates a lot of these complexities
of dealing with biology and culture and also geo-politics as I work on the identification models for improving the work of those people who are on the border. The boots on the ground, very amazing people who
I work with in Arizona who generously afford me the data and the opportunity to work with them in their medical examiner offices, on the identification of missing migrants who have made the perilous journey from Central America and
Mexico and have perished only at the point at which
they’ve reached the border. In that, since then I work
with refugee populations and I also work with the living. It’s important to be able to understand the patterns of the dead. We need to understand the
patterns of the living. So, being able to have the opportunity to understand migration movements and how people self-define and trends that shift
in that self-definition and in those migration patterns over time are critical for me to build models that accurately reflect the people who are crossing and dying on the border. And then, more recently, I have work that is also focusing acutely on social justice concerns and how geopolitical
situations actually change our biology by causing trauma and that trauma is played
out in our epigenetics and those traumatic events
and those consequences of those events can be passed
on inter-generationally. So understanding how, what
happens to us culturally affects our biology is huge in terms of being able to create models that allow us to serve social
justice in the best way. So then how did I end up on a panel that deals with facial recognition? Because I haven’t mentioned
that yet in anything that I do. My specialty is actually
the estimation of ancestry and ancestry is something that reflects largely continental
variation that goes back to our evolutionary history, and the skeleton is a really great source of population history information because while we do have heritable traits in other parts of our body, the skeleton is very, and
the cranium in particular, is highly heritable,
which means that it is very stable, so it reflects
neutral genetic variation, which is essentially what we need to study in our models to be
able to estimate ancestry. So the cranium, as I said, retains that kind of ancestry
information very well. And how do we study the cranium? We take measurements, and
we’re taking measurements of the underlying structures
for which facial recognition is ultimately built upon. So, the sort of fundamentals
of biometrics are similar in terms of facial recognition
and ancestry estimation from the cranium. So the processes, the mathematics
of it are complementary and the issues that surround it, the problems of partitioning
people up by labels and imposing estimates of what we think someone’s biology tells us are equally problematic when
you’re using facial recognition for estimating someone based
on their outward appearance, so not to make a match, but to estimate someone’s skin color and then extrapolate from that what the racial identity would be. The same issues can arise when we try to make that allusion
between ancestry and race, which is problematic and complex. So we share in some of
these same struggles as a field as we move forward. So then as I said, craniometrics then are a form of biometrics, and we need to be cautious as we approach biometrics from a facial recognition
perspective looking back to the long and very torturous history of how craniometrics have
been used in science. So we need to need to think
of Blumenbach and phrenology and the racialization of immigrants using
measurements of the body, and in particular, the cranium. So we can learn a lot as we move forward with what is a new technology
by looking backwards at something that has
been around a long time and people like myself are hopefully doing a much better job at bringing
this established history of anthropology and biometrics forward into something that does good work in promoting social justice. So then, as I said, like
biometrics, craniometrics have had that fraught
relationship with race, but they’re also like biometrics, in the facial recognition realm, used for individual identification. So we can take measurements of the cranium and aspects of the craniofacial shape and we can match them between
the unknown individual for which we have that
source of information whether it’s a radiograph, a CT scan, or a collection of measurements, photography is also very common, and then match that to
a record that’s on file for a potential match, and you can see on the screen that we have an image of the skull
and then a photograph and we use photogrammatographic
superimposition as a way to identifying
corresponding points and by the number of
those congruent points, we can determine whether or not that’s most likely that individual. So think of it the same
way as a DNA match is made, a K unknown to a Q to a question, or a fingerprint, where
you’re taking a known and an unknown print and you’re looking for concordance in the shapes. So then, I assume that
as I’ve been talking, in the backs of your minds, you’re hearing my concerns
for ethical issues, how we hold ourselves to account and how we hold the academic community and the larger industry to account in terms of doing good science. So, when Daniel invited me, he asked me to think
about my grand challenge, and without question, the grand challenge that came
to mind to me was ethical. How we deal with the ethics
of doing scientific research and making sure that
that work is being used for the best purposes. So, I thought then
today, I would go through and choose a few of the ethical issues that I think, together, create a large grand challenge and we can hopefully deeper dive into some of these as we
move into the discussion. So first and foremost, I think it’s critical that
we are critical of ourselves. That we ask, as scientists, what our responsibility is
for actually doing the work. When do we call halt and
say that, as scientists, we are developing things that may be misused, abused, and dangerous for people? When do we accept that
we only do the science and that is enough? Or we need to recognize
that we do the science and that science is
disseminated, and therefore, that trail back to us is always there. So, as the progenitors,
we have a responsibility to determine either
collectively, as a university, of what we accept and what we don’t accept in terms of breaking technological bounds with the risks that come with that, or what we believe to
be the ultimate gains by advancing science
and what infrastructure or what systems we have in place that will be the checks
and balances for us. So, as an example, and harkening back to what Joy spoke about yesterday, that we have in facial recognition then, the issue of identification
based on skin color, so that the successful
identification of darker faces suggests higher rates
of misidentification. But, what happens when
we get better at it? So, ultimately we wanna develop
better technologies, right? So we do much better as scientists, then, at improving the technology
and we can now, not only, identify lighter skins, but we identify darker skins
with equal probability. Where does that leave us? It then leaves us with another technology that can potentially be used to marginalize, disenfranchise, and cause harm to people of color. So there’s a push towards a scientist for improving what we
do to do better science, but in that act of doing better science, are we doing more bad than good? And where do we draw the line
in terms of what is the limit that we should place for responsibility on us as scientists and the responsibility falls then on policymakers or does it fall on technologists who distribute the software that enables this to happen? So there are many levels of
discussions that need to occur. That brings me to the
dangers of dissemination. So, we have these technologies, we produce them within
the cloistered environment of academia where it’s safe, it’s safe to be radical thinkers, but then those radical technologies are transmitted into the public, and who is the safeguard,
or the safety check, or the educator of the public. Well we educate our students and we hope that then they
go out and in their jobs, we’ve given them a good foundation to be able to transmit
quality information. But, we have to think about the role that these types of technologies
play within industry. So ultimately, when something
is distributed sufficiently, it’s sufficiently popular, you have lots of technology, lots of competing companies
producing softwares. Then, you have it distributed widely. Do we always have the best education? Are the people who are employing these technologies
always the best trained? You can have the optimal technology that makes very few errors itself, and then it has a very
easy application threshold. So that means an analyst is
gonna do a pretty good job at applying it, but then there’s always that
issue of interpretation. What weight do we put on the information that we get from it? So this is nothing to
do with the software, nothing to do with the algorithm, but it’s in what do we do with
that information afterwards? And there’s a lot of literature right now that looks at standards
for how law enforcement are trained and using
these kinds of software, and there’s not a lot of evidence or sharing in terms of
the types of training, what the approaches are to the training, and how these vary between
law enforcement offices, between states, between countries. So, the idea that these
softwares are being distributed, and then we have the public as well, who is maybe not as best
informed about what they do, creates a lot of confusion and chaos. So it’s critical that we
not only educate the people who are using this software, we need to educate those who are consuming that
information from the software, and then we also have
to have the public aware of what’s going on in
terms of the software use, what that actually means. Because ultimately, they
are serving as the jurors when law enforcement cases
are brought to court. And, as an example, Rashida
Tlaib, congress spokeswoman, was at a law enforcement office, and she made the comment
that’s quoted up here, and while her intentions are correct, that there is a bias in the process and the software and the way that it’s often used, she’s placing that issue
on the actual analyst without a larger understanding
of the whole mechanism. This suggests that, as she is a representative of the people, and yet she is maybe not as informed as we would like her to be, and that is incumbent upon us as academics to do a much better job to teach people. It’s incumbent upon the
industry to educate more. So, we need to make sure
that we are investing in not just production of tools, or safeguarding the use of the tools, but also in the education around them. This also brings me, as the mathematical person
to geek out for a moment, that we also need to ask mathematically or computationally or probabilistically, what it actually means to get a match? Sometimes you can get matches and it doesn’t mean they’re useful. So we have to weigh that risk against what the ultimate reward is. So, think about the
fact that this software does its absolute best job
on people of light skin, and we know that the
US is populated mostly by people who self-identify as white, who we would extrapolate
generally have light skin. So that means, it does really well on the majority of the population. So then if you make an identification of someone who is white, based
on the facial recognition, making that identification match, what does that actually mean? Probability-wise, the evidentiary
value means very little. So now we’ve created this technology that allows us to do these amazing things, but what it actually outputs doesn’t actually give us anything that improves a case, in terms of advancing social justice of the judicial system. So, there’s an important
balance that needs to be hit between doing technology for improvement, and does that improvement
actually improve things? So to be thinking constantly, critically. And then, speaking to
the issue of the court, we need to ask, are we at
risk of a new CSI effect? We need to be thinking about how jurors, or the public who are our future jurors, consume this information. So, as someone who has testified in court, in terms of DNA evidence, we know that there’s evidence
to suggest that jurors now, on account of exposure to
scientific technologies, have expectations of what will
be delivered in a court room. What they expect that law
enforcement should have done to demonstrate that
they did due diligence, that the case is believable, that there’s adequate evidence. So, if we are presenting this technologies without teaching the public that there are problems with them, then we can run into the
risk that they become an expectation of what is good science and what is a good court case. So we need to be always cognizant of how these are being
consumed in the public sphere in terms of perception. Often as a academics, we think of, or as scientists in particular,
we think of numbers. Statistically significant. Do we have high probabilities? Without also taking into account that perception often trumps numbers. How we think, or believe, or
what we feel we innately know is sometimes way more
important in convincing people than what a statistic does. And then lastly, we run also the risk of what I’ll call the
compound technology effect, where we have surveillance technologies that then get combined and we are regulating them independently, but what happens when we
start creating systems that pool all of these
technologies together. So, as an example, there’s
this project called The Face of Litter in China. It analyzes and collects
DNA profiles of litterers using an American software
program called Snapshot DNA from which they use forensic phenotyping, which is to take your DNA and to try to recreate what
you look like physically, just from a DNA sample, and they combine that with
demographic information, so who they expect lives
in a particular area and what they know about the habits of that particular area in Hong Kong. And, that means that they’re one step towards then being able to reconstruct that individual’s whole person a picture essentially of that person based on their DNA sample that they’re getting from
their litter, and then, they’ve taken those pictures and they’ve put them up on billboards. So now, anyone walking around has pictures to view of what this
potential litterer looks like, and then, if you scroll down to my slide, you’ll see that there’s
evidence where police officers are wearing facial recognition glasses to identify known criminals. So now we have pictures
of known litterers, translate that to criminals, and then we have someone who
has a database in their glasses of known criminals walking around waiting to identify them. So, we have this combination
of DNA technology and facial recognition technology that’s coming together to create this mass
system of surveillance. And while we talk about China
in sort of in the abstract, as some place far that
doing these extremes things in terms of surveillance, we are still producing the technologies that are going into supporting this, so we are ourselves still
invested in this process, we may not be doing it,
but we are not that far from having to grapple with these issues, and these are issues then that
we should be grappling with, with the recognition that
it could be us some day. So, we need to be thinking forward. And I will leave it there, thanks to you for your attention and to Daniel for the opportunity. (audience clapping) – Thank you Bridget, definitely raised some important challenges
for us to discuss in a moment here! But before we get there, I’d like to introduce our last panelist. Dr. Wendy Hui Kyong Chun is
Simon Fraser University’s Canada 150 Research Chair in new media in the school of communication. She’s studied both
Systems Design Engineering and English literature, which she combines and mutates in her current
work on digital media. She’s author of a number
of books including, “Control and Freedom,” “Power in Paranoia in
the Age of Fiber Optics,” and co-author of “Pattern Discrimination.” She’s been a visiting
professor at AI Now at NYU and currently serves on their
academic advisory board, and a visiting associate professor in the History of Science
department at Harvard of which she is an associate. Dr. Wendy Chun. (audience clapping) – Hello, everybody, it’s been a very intense two days, and it’s really lovely to be here, thank you Daniel and
Jennifer for the invitation, and HAI for bringing us all together. So today I’m gonna talk really
broadly about the question of race, rights, and recognition by thinking through one
of my current projects on discriminating data,
and as Daniel mentioned, this project emerges
from my new group at SFU called the Digital Democracies Group, which was founded by a
Canada 150 Research Chair. So this, for all of you who don’t know about the Canada 150
Research Chair Program, was basically Canada
deciding, now is the time to try to poach foreign
academics and get them to come to Canada. And so, what we’re doing
is setting up a really multidisciplinary group, so we’re bringing together
research in the humanities, the social sciences,
engineering, and data sciences, in order to address issues of equality and democratic dialogue. And we’re really unusually
interdisciplinary, so we’re working with a global network of network and data scientists, quantitative political scientists, media theorists, political theorists, artists, dramaturges, as
well as computer programmers. We’re also working across sectors, we’re partnering with
some researchers at Intel as well as non-profits, such as Social Science Research Council. And we’re working across these discipline and sectors as many of you
people have been talking about today because we have to. Because in order to take on
the really tough problems that face us, we need to be really unusual and a little unholy in our alliances, and so what we wanna do is
help counter the proliferation of equichambers, abusive language, discriminatory algorithms, and miss and disinformation, and I’m happy to talk to you about more about the lab and our other
projects during the Q and A. So, discriminating data
explores how algorithms currently encode legacies
of segregation, eugenics, and multiculturalism. So, it looks at the way in
which identity categories, such as race, gender, and class, mutate and persist through algorithms that are allegedly blind to them and are arguably designed
to blind us to them. So how do categories such as gender, sexuality, race, and class persist and mutate through as many others, including Oscar Gandy
many, many, years ago, have shown through proxies
that launder discrimination, and this is taken from
the Marshall Project, and they’ve shown how categories such as, doesn’t have a high school diploma, which is used to determine
the risk of recidivism and thus sentencing in parole in programs such as Compass inscribe race and class in an intersectional manner. What’s interesting is
their getting at that intersectionality of race and class. These categories are also
inscribed through latent factors, or principal components, that discover or naturalize
sexual difference, so here’s one finding from
the notorious Gaydar AI, which produce prototypical
gay and straight faces by analyzing snapshots
from an online dating site. There’s a ton to say about this, which I won’t say right now, but what’s fascinating is that they used machine learning
to discover and naturalize the sexual politics of facial hair. And these categories are also reinforced, as I’ll discuss in more detail, through network neighborhoods that are premised on and
that produce segregation. So, social networks are based
on the notion of homophily, the idea that birds of a
feather flock together, right. And so you’re clustered
into network neighborhoods based on how you deviate
from the norm like others. And so, echo chambers
aren’t a side effect, they’re the goal. So, discriminating data follows the following five-step program. The first is to expose and investigate how ignoring differences
amplifies discrimination both currently and historically. So how crossing out race
actually perpetuates racism. So how dreams of cyberspace and AI is solving the problem
of human discrimination actually perpetuates discrimination. And there you see the article
sort of introducing Compass in the Washington Post as something that would actually dissolve the problem of discrimination in courts, all right. Also, looking at how dreams
of machines as our slaves or servants, so self-driving
cars and chauffeurs, recommendation engines
as everybody’s butler drives fears of the coming singularity, and Jenny Reed has written a great book sort of documenting the
intersection of race, class, and gender, with robotics. And so, one thing I’ve
been thinking through is the dangers of hopeful ignorance, the repeated dangers of
this hopeful ignorance. Step two is to interrogate
the default assumptions and axioms that ground
algorithms and data structures. So homophily, again, underlies
collaborative filtering. So collaborative filtering
groups, users, and objects into neighborhoods based
on boundary defining likes and dislikes. And not surprisingly, U.S.
residential segregation is often given as evidence
that homophily really exists, right, and here you see an excerpt from Easley and Kleinburg’s
really excellent textbook on network science. But let’s ask ourselves,
under what conditions and why has this become a default, right? Because the ties between homophily and residential segregation
are deep and profound. So, homophily was actually
first coined as a term in 1954 by sociologists Lazarsfeld and Merton, and they also coined
the term herterophily, which is something that’s
been completely erased, but it’s also clear that
this is one of the most cited articles in the literature, but also clearly, one of the least read. Not only did they also cite heterophily, at the heart of this study, is actually a study of
two housing projects. One which was a white
only co-op in New Jersey, and another which was a biracial yet segregated housing project in Pittsburgh, and they
didn’t assume homophily to be axiomatic, but they rather asked what other dynamic processes through which similarity
or opposition of values shape the formation,
maintenance, and disruption of close friendships, and what’s important to remember is that they were working in the late ’40s. This was a time of great income equality. It was also a time when
there was 80% support within the U.S. public for public housing. It was also a time when public housing was predominantly white, and what they noted in their
studies of these projects is that homophily
actually wasn’t the norm, and what they argued,
though, was two circumstances in which you did see status homophily, which was race and gender. This was actually due to what
they called value homophily, or the sharing of values. And to do this, and to prove this, they concentrated on the over-selection of white liberals and white
illiberals, of friends, where friend was considered to be one of your three closest friends. Now, to make this point, as you see here, they actually threw out the responses of all the black
residents, on the argument that they were all mainly liberal. So there was no idea ever of putting the black and white
responses together, right. So, liberal meant that you
believed in integrated housing and you thought that the races got along in the housing project. Illiberals thought the opposite, and ambivalents thought
that housing projects should not be integrated, but the races did get
along in the project. So, a prior racial
segregation, grounds homophily, and in their subsequent analysis, a model of friendship formation, they also ignored the white ambivalence who actually constituted the largest category of white residents. So they threw out most of their data and they openly speculated. So here at the heart of
functionalist sociology is this wonderful
acknowledgment of fiction, in terms of thinking through and institutionalizing homophily, and what’s also interesting is that they never published the
data from this survey, because if you did as you see here, and we’ve gone through the archive, the over-selection of
illiberals for other illiberals is the difference of two people. And so, what’s important is that they also have this incredible other data. So, they also asked people
do you have acquaintances of the other race? Do you have friends of the other race? And based on those questions, the majority of people said yes. So the data was there to
support a very different vision, and it’s not just homophily
we need to rethink, but it’s also correlation. So the early 21st century
celebration of correlation isn’t new, right, and here
you see two quotations taken, one, from the beginning
of the 21st century, one, at the beginning of the 20th century, declaring how correlation
was going to change the face of science. And the piercing correlation coefficient, which is used in many of these algorithms to determine similarity
and linear regression, were actually developed to
prove the truth of eugenics. So they’re developed as tools
to understand human heredity, and so Francis Galton, who was the father of eugenics, developed linear regression, which he first called liner reversion, while studying two problems. One was the identification of criminals, and the other one was to understand the transmission of human heredity. So he wanted to understand how genius was or wasn’t transmitted
culturally, biologically, amongst his schoolmates in Oxbridge. And Galton was a biometrician
rather than a Mendelian, so he believed that traits
followed a normal curve rather than Mendelian genetics. So exceptions, such as genius
were statistical outliers, right, so they were located
at the ends of the curve, and so, Galton developed
linear regression, which is now used to determine the best fitting line
between scattered points in order to understand the
transmission of deviation. And so this is very different from how we understand
linear regression now, right. The point of linear regression
is to minimize deviation. Come up with a line that
minimizes standard deviation, but what Galton wanted to
do was try to understand how to maximize deviation. So he formulated the
notion of linear reversion by studying the deviation
in heights between people of exceptional
height and their offspring, and what he did was draw a line and try to figure out a slope that linked their deviations, and what he showed was that deviation, unless carefully bred,
decreased over time. So according to Galton, his graph showed that we were regressing
towards an ancestral mean, and so the point was to
constantly stir the system so that deviation would be produced. And it’s not just linear
regression, it’s also SVM. But to be clear, my point is not that any use of these
techniques are eugenicist. Clearly statistics and
proxies are important to understanding things, such
as global climate change, but my point rather, is that
if the world now seems closed, it’s because these methods focused on breeding a human future
based on an unchanging past, so in this rule, there was
no nurture, only nature. Intelligence, Pearson famously argued, couldn’t be learned,
it could only be bred. So the future could be calculated and had to be calculated, but it couldn’t be radically
different from the past. So if you think about
it, it’s really perverse, that a system that believed
that learning was impossible has become the basis for machine learning. Which leads me to step three, to apprehend the future
machine-learning algorithms put in place, focusing on when, why, and how the predictions work. So what world are we living in, that these predictions seem to make sense, and what sort of role
do they put in place? So again, in terms of
regression and correlation, how did a system that denied
the possibility of learning, again intelligence could
not be learned, only bred, become the basis for machine learning? And how did this system
that was determined to shut down disruption, right, so a system designed to
shut down the future, to make the future repeat the past, how did that become the
basis for disruption? Or maybe, the shutdown is
actually what’s so disruptive because if knowledge now
means knowing the future, it’s because in these models, the past and the future coincide. So, most basically as we all know, these algorithms are trained
on and tested against past data, which means
if the past is racist, they’ll make racist predictions, but also that if the past is racist, they won’t be verified as correct unless they make racist predictions. So, as Reed Hoffman pointed out yesterday, these programs can lock
in-scale inequality in a non-transparent way. Yes, so what should we do? Well maybe, we need to
be a little perverse. Maybe we need to be a
little perverse towards our attitudes towards these
discriminatory algorithms, and what if we used existing AI assistance to diagnose current inequalities? What if we corralled
discriminatory predictions as evidence of current and historical forms of discrimination? Right, so the example that
everyone knows, of course, is Amazon’s secret hiring tool, which they’ve stopped using because it knocked your
score down if you had woman anywhere in your CV. So they’ve stopped using it, but what if instead we
sort of thanked Amazon for meticulously documenting their discriminatory hiring practices? So in other words,
rather than these models automating the mistakes of the past, what if we used them to diagnose
and address past mistakes? So in other words, what if we used them and treated them like global climate change models? Right, so global climate
change models tell us the most probable future if we keep repeating our
certain actions, right, and when the global climate
change models shows us that things are going up by two degrees, we don’t then say let’s fix the model, we say let’s fix the world. So in other words, these
predictions of the future, not to close it, but rather to open it up so another future could emerge, right. So what if we also devised,
and this is the last point, what if we also devised
different kinds of algorithms in new ways to verify
machine-learning programs so we displace these disturbing eugenic and segregationist histories, right? And this is something that
was touched on really nicely earlier in the conference. So, heterophily, the notion
that opposites attract, what if those lines
represented heterophily? This is how electromagnetism works. How many of you know people
who are heterosexual? Apparently, you can have
heterophilis networks. It exists. Think of something like indifference. What if indifference was
actually the grounds that tie? Think of everything you
need to be indifferent to in order to sit in the seat
that you’re sitting in, right. What if indifference or infrastructure became a way of understanding it? So, basically, the goal is to reach out and to engage the rich
spectrum of human experience so we can build better models
and systems in the world. Thank you. (audience clapping) – Thank you, Wendy and thank
you, Bridget and Matt as well. I told you Bridget and Wendy were gonna expand our thinking today and I think they’ve delivered. So we have about a half an hour left, I have some questions here, but would love to hear your questions too. So, I’ll ask a couple and then, you know, we’ll do some back and forth here, and there’ll be some microphones going around the room as well. So, one thing that’s, whether we’re talking about the design or the deployment of AI
systems, creating the frameworks to regulate them, who is in the room and
at the table matters. So we talked about this panel
having different disciplines, different sectors coming together, this conference has attempted to do that in certain ways as well, Wendy, you described the diverse
group of academics in your digital democracy lab, also about the challenges and
problems of echo chambers, you called your group an unholy alliance, which I appreciated. Matt the coalition of civil rights groups and others who are not traditionally or haven’t traditionally
been focused on privacy and similar issues to come
together in the table, and that was really transformative. I’d like to dig a little
bit deeper into that and ask you to reflect
on how its been important in your work to have this
diverse set of perspectives. So what are those, you know, there are some of those
moments that you have really illuminated new perspectives
in your thinking, new approaches on how we
might be able to do legal or legislative or litigation work, and how do we ensure that this
is the norm going forward? Matt, we’ll start here. Go down the line.
– Sure, sure, yeah thank you. Being able to collaborate with
our racial justice partners has meant that we have been able to pass better policy basically. To give a particular example,
each year at the ACLU, we sit down and we do strategic planning at our affiliate in San Francisco and thinking about what
are we gonna be working on in the next year, what sort of annual goal do we wanna sort of push towards, and what are the strategies
we’re gonna need to get there, so we like to work on a variety of integrated advocacy strategies, whether it’s public records work, legislative work,
litigation in the courts, we don’t just choose the
tool that we wanna chose but we choose the one that is the best for getting towards the goal. At the beginning of this year and coming out of last
year’s legislative year, we’ve sat down deliberately with our racial justice partners including Malkia Cyril at
The Center for Media Justice, simply now Media Justice, and we talked about where
surveillance in communities was moving and what we saw together as some particular threats
that needed to be addressed. So out of that conversation came two really, sort
of, definitive things that set us on a course
to where we are now. That was one, the understanding that these body cameras that had been deployed in many communities in
California and elsewhere, were one, not delivering on
the promise of transparency and accountability, but two, were at risk of being turned into roving surveillance devices. And so Malkia and her deputy Steven, now the co-executive
director of Media Justice, they said this is something
we need to be focusing on, and having that conversation right there sort of narrowed the aperture towards this issue of body cameras, and that moved into legislative work which they were there
alongside us the whole time. Another thing that was really important for having the discussion early and often is to develop a narrative like why do we want this change
that we want to achieve in the world and sitting
down with Media Justice and other partners, we were able to break down, I think a traditional, a narrative that has traditionally been a barrier to passing civil rights reforms when it comes to surveillance, which is, on one side you have security and safety, which is typically owned
as this sort of narrative by law enforcement, and the other side, you have civil rights and civil liberties and that sort of balance
is not a productive one for conversations with policymakers, who are reasonably and
understandably concerned about safety as well as the other things. What we did by sitting
down with our partners and a broader sort of coalition of groups is we sought deliberately to re-own the public safety narrative. Not only to literally mention
the word public safety or the phrase, whenever
we could in our work, but to think about the ways, which history provides plenty, where unrestricted secretive surveillance doesn’t make people safer. And so, not only the sort of concept of what the legislation would be, but the messages were a direct result of simply convening
folks that we worked with and wanted to work with
about and talking about what we were concerned about and how we were gonna talk about it. – Bridget, you talked
about a diverse group of collaborators, whether that’s
an interdisciplinary group of academics and researchers, but also Boots on the Ground, how is that diversity of
voices and participants in the work that you do shape the work and led to challenges potentially as well? – Yeah so I actually have a project now that I think probably
is a really good example of the value of reaching across the aisle to a whole variety of stakeholders, so I mentioned that I work
with Boots on the Ground, groups in Arizona in particular, I have a very generous
partner who is in Pima County who I’ve worked with there
who also shares their data. So I have a project in which I have data on unidentified border
deaths, and then subsequently, they have been identified. So I have the ability to test the models and then actually verify them
against the known information and using that information and applying the machine-learning algorithm, I’m able to estimate with
very high probability, like in the 80s and 90 percents, where each of those
individuals originated. And while on the one hand that seems very negative and
counterintuitive because it is a form of surveillance, right, that we’re able to make those
types of determinations, it’s to the people who are
working Boots on the Ground every day trying to take those remains and get them sent back to the right place, to their families of where they came from. This kind of work is
critical to be able to say, no this person didn’t come from Mexico or this person came from
this state in Mexico, rather than just sending
a file to a consulate that stays on a shelf indefinitely. So, to be able to know what the needs are of the communities that you
serve and think critically about what you’re doing and broadly about what they need, to be able to bring those two perspectives together, I think is really important, because sometimes we get
really lost in what we think other people need from us, or what we think is the type
of research or good science that should be done and
we miss the potential to actually make a contribution thinking about what
the applied context is. I also will say as, so
as Matt was talking about the very practical, I’m gonna go back to the
very academic theoretical that interdisciplinarity
is talked a lot about and it’s very rarely practiced, and it is critical that we need
to think interdisciplinarily so that we are creating students, student opportunities
that celebrate the fact that we are good at a bunch of things, that we have multi or
intersectional perspectives, that we are able to talk
about not only what we do, but appreciate what someone else does, because how can we ever make judgments on the quality of work if we
don’t have an understanding of how our work interacts with others? And to understand that the work that they’re giving us
is properly contributive to the types of questions
that we are seeing as important to answer. So, we need to have systems in place as we’re teaching students
to think that way, we need to have systems that
allow for that to flourish, and then we need to also
hopefully in doing that, create a culture in which this is the way that we think globally, that we are not promoting
the fissuring of academia, or the partitioning of
policy versus academics, but that we’re thinking connectively, where we’re creating a network of people working towards social justice, and I think that’s what
so great about CCSRE is that it’s in locus here at Stanford that really celebrates
bringing in many people from different departments, celebrating those perspectives and allowing those voices to be heard, and those voices are all voicing, work towards a common social justice goal, and I think we’re very lucky
at Stanford to have that. – So, Bridget you’ve talked
about interdisciplinary, Matt, you’ve talked about trying to create a
shared narrative, Wendy, one of the big challenges
in doing this kind of work is language, right. How do we communicate across disciplines or just even understand, what
computer scientists are saying and scholars of literature are saying or critical theorists,
this often, you know, sort of over each others heads. In your unholy alliance, how have you been able to
communicate effectively to do collaborative work? – Well this is gonna sound
bizarre but I think what, in order to communicate
across disciplines, what you need to do is
actually go higher than these methodological issues and
ask yourselves, what matters? So this all started because Brown’s like most academic institutions sometimes it feels like
“Survivor” for academics. A new president comes in and decides, we’re gonna have these little groups that are faculty-led and they’re gonna come up
with these initiatives. So, I was put on the big-data initiative, which gives you a sense of when it was, with the head of CS and
someone in bio-stats, and we looked at each other and we said, we’re not gonna work together unless there’s a problem that
you can’t solve without me and I can’t solve without you. And the problem that we came up with is what’s the relationship between what’s real and what’s true? That if almost any correlation
can be shown to exist, how do we know which matter? And this actually came from
the person in bio-stats who’s working on AIDS research. And then we added a second question was, so what if we know it’s true, what actually inspires political action? And this is what the engagement with global climate change
but also asking ourselves what worked as simulations actually do. So I’ve often found, and then to give you one more current that we’re working on right now is the question of misinformation, right. a lot of people have shown
that accuracy is important, known verification is important but it’s not enough, right. Corrections don’t reach the same audience, people don’t care if a story is correct. They find it compelling, and to be true, irregardless of whether
or not it’s verifiable. And so what we’re doing
is working with somebody to build a model of authenticity. So authenticity is different from facts. It also has a very dramatic history, in the notion of the tragic hero, someone who lies a lot is
considered to be authentic. And so, what we’re thinking through is what would an authenticity measure, as opposed to an accuracy
measure look like? So using machine-learning for that but then also working with
people in these workshops, where they produce scripts, based on their experiences
with misinformation, which we then perform. We’re also turning them
into web extensions, but also trying to work
with people in theater and performance studies to come up with other forms of interactions and scripts. – So, we’re here at a panel on race rights and facial recognition and each of you talked about it in different
ways in your discussion. At this conference, we’ve
heard some conversation about race, Joy certainly
focused on it in her talk and other have discussed it as well, but I’d like you to help
us reflect a bit more on why you think it’s important to center race and
specifically racial justice in discussions about facial
recognition technology in particular but also AI in general. How have you worked to
do this in your work, challenges you’ve encountered and people who don’t really
wanna frame the conversation or the work in that way, thinking Matt, starting here, you’ve
done work in the Bay Area, having a conversation here
is very different than, about racial justice in a legal context is different than having it, say in other parts of the country. Whether you, sort of lead
with that argument or not is taken differently. How does that manifest in your work? – Yeah, so, it’s been important to center racial justice in our work
because of the sort of the historical arc of how
surveillance technology has been used in communities of color and against communities of color, against immigrants and against activists, as I was saying during my presentation. We’re to the point now
where we have plenty of historical examples of what happens when a new surveillance system is built and what will the regime, or the sort of government
in power, will do to use it? And who it will be used against? And so, the writing is on the wall, then we need to be paying
attention to this issue and more and more, the intersectional
conversations are happening in the civil rights communities, where we realize together that, we can’t be talking about privacy as a sort of high-minded
concept because one, that’s not persuasive to policymakers, that’s not persuasive to the members of all of our groups as talking about real people’s lives are. So, really it’s a, I think sometimes just more valuable to be talking about people’s lives and
addressing the history and the patterns that have occurred. And you’re right that
the Bay Area is different than a lot of other of other places, but we’re seeing this narrative and sort of conversation expand across the United States right now. Over a dozen cities in the
United States have passed laws. Not all of them have
facial recognition bans, but many of them have a requirement that the community has to be
a part of the conversations about surveillance technology, and so you have places
like Nashville, Tennessee, St. Louis, considering
and adopting legislation that requires that all communities members have an opportunity to say no
to surveillance technologies and to say what the safeguards should be. I was also, as you were
saying that question, thinking about a meeting
we had last December with the general council
of Amazon as a coalition, I flew up to Seattle and
local groups in Seattle working on racial justice issues came to that meeting and besides the sort of disappointment of not having this sort of productive conversation around racial justice
with Amazon, who I think and we think, has really
utterly refused to engage with the sort of impacts that
their technology can have on real human beings. What was striking, the
most striking moment in that meeting for me
was hearing a gentleman from a Japanese-American
advocacy organization sit across the table from
the general council of Amazon and talk about Japanese internment, and how the history of
what happened there, which I mean, amongst many other things, the government used a
very superficial feature, the Japaneseness of Americans, to round people up and put
them in internment camps without regards for real dangerousness or anything more than skin deep level, and that was just an amazing, and I thought, impressive reminder of why it’s so important
to be talking about this with as many people at
the table as possible because that sort of reminder
has not only stuck with me, it stuck with everyone in that room. I know it stuck with the
Amazon executives as well, and it’s a different perspective than what a community member in San Francisco or Oakland might bring to the table. – Wendy, you think about it
sort of give us a genealogy, challenge some of these
histories of development of methodologies, how does that
connect with racial justice? Why is it important to
center racial justice in conversations that we think about, that history of science
and what the future brings. – Because, what it brings out is the fact that race is central. It’s not something that people have added on to the discussion,
it’s actually been there and in terms of Japanese internment camps, one of the interesting things in terms of looking at homophily and studies of homophily was it wasn’t just this integrated, yet
segregated housing project. When they were trying to
think through it, they said, okay we’re interested in this because it’s social engineering, they were blatant about
this is social engineering, and one of the examples
that they drew from and tried to understand was
Japanese internment camps, as well as Indigenous reservations. So, it’s not, why it’s so important to engage questions of race, is that you realize that if
we’re in this situation now, it’s not an accident, and that
we need to engage communities and think through what
other things were possible. People in the Japanese internment camp did not experience things the ways in which the sociologists
who were studying them did. So it’s by engaging these communities and histories and making them the basis for different kinds of
models and understanding, that’s absolutely key. – And Bridget, we’re thinking
about race as a marker, along with age and stature. Why is it important to think of it as more than just a
specific variable or marker but think about and have conversations and do the work in
terms of racial justice? – ‘Cause it’s always
already all about race and so how we, who we chose to have as a partner is racialized. That then has effects
on races may not exist, but biologically, but races
have biological consequences. So to think about how race influences every aspect of our biology. So to understand how through
our biological shifts and biological patterns, we need to understand how race
operates within the world, and, I will say that I feel like I’ve been super negative
about facial recognition and science and I’m actually
a very positive scientist that I would say that doing innovative exciting work, is a
lifeblood of what we do and that’s important. And I will say that from my perspective, there can be very positive
outcomes of issues of race and its intersection
with facial recognition. We know that eye-witness testimony is terrible and having
the opportunity then in certain cases to be able to provide another source of documentation
may prove helpful. To be able to have information on where someone might have been if we’re dealing with a missing persons case,
a human trafficking case, that there are potential good uses of it. As I said earlier, we just
don’t know what we don’t know, and we’re in such an area of change and flux within the technology and within its uses that often the potential for the good needs to be regulated by the, or weighed against the immediate risk right now for its misuse. So I would hope someday
that we might reach a point at which policy and risk
versus value will balance out, where we can have opportunities where it can be used in places where it would actually do
the work of social justice. – So, in our last few minutes here, our last ten minutes or so, we’d like to open it up for some questions from the audience. I believe we have some microphones. How about up front here, David. – Thank you, it was a really fascinating and great discussion. I guess one of the categories
I was thinking about, not just around the difference
between truth and reality, for example, was what
counts as authoritative? Right, so there’s so much of what happens with this technology is that
there’s this presumption that technology produces
authoritative results, and there’s a question and
it’s an ethical conundrum, what are the correctives against that, not just the data, but the mechanisms that will allow to correct for what is
considered to be authoritative, so cultural literacy, you know, you said everyday people, Matt, right, like what are communities
that are being targeted around these technologies, like what are their
bull works against that? – So, I can talk just
quickly about the question, so clearly authenticity and authority come from the same root as well, so that’s part of trying to think through that authenticity route, but I think that the bigger question that you raise is verification. How are we going to
talk about verification if we don’t want verification of a model to be the fact that it’s results happen? So like for instance, global
climate change modelers, they don’t wanna be proved right, right. The idea isn’t to wait to see if their predictions are true, right. So there, you know, this is why you have hindcasting and different
forms of trying to understand, oh can it do a different data set, but I think that that whole question of what is verifiable versus what is outside of simply repetition is absolutely key and it’s
something that’s facing us. And also, one reason why we like the example of global
climate change models is because it’s unclear
they’re convincing either, given the debate that
we’re having in the U.S., right, so I mean, I think thinking through that and that relationship is really important. – And I will just add that this idea of one version I think, of
what I was hearing you say is that people trust machines
in some sort of sense, we’re just more likely to trust machines than we would have trusted an eye-witness or a human telling us the same thing, so in the field that not only can result in the wrong decisions being made when the systems themselves are incorrect, which is life and death
questions sometimes, but I do also think that it
results, that sort of bias towards trusting machines, I’ve also heard it called
automation bias, right. It also causes us to sometimes
reflexively think that surveillance technology
is going to be the answer to societal problems
that could be addressed through other policy mechanisms
or other discussions. And so, you’ll see this
every time there’s a tragedy, every time, unfortunately now
there’s a school shooting, the reflects of response frequently is going to be more
surveillance, more security, I mean this is just a tried and tested pattern unfortunately. But I think that is in some ways influenced by our sort of willingness to trust machinery and
processes, when in fact, they don’t address the
root causes of the problems that are harming people in our society. – All right, let’s go
back in the middle, yeah. – So, are there other
interdisciplinary examples where people are using machine-learning to challenge bias in non-linear, non-binary ways that are visual. For example, there’s a
project here with ID cards for the medical school, maybe that doesn’t have to be the standard kind of mugshot ID. Are you aware of any other projects like that that we’re doing? – So there’s a really
fascinating computer scientist based at the University of Washington who’s using Word2vec so
this can, this is textual, but in order to diagnose
and redress discrimination. So using Word2vec to see how certain words are placed close to each
other in certain corpuses, in order then make the case for certain kinds of both
implicit and explicit bias. – To just build off that, some of my own work
using text mining methods which are inherently
machine-learning as well, studying how academics
talk about issues of race. So, if we removed the word race, are there patterns in
which we speak about race and it’s an encoded structure that we don’t actually
have to say the word, but that we have a
language and the language that signals to people reading it, that we’re talking about race or that there’s this implicit bias in the way that we’re describing things, and in being able to
tease those patterns out, we can then do better ourselves and teach our students how these latent trends exist and the way people consume
these latent trends and how they’re effective
of particular sentiments or reactions in people
when they read our work. For me, coming from initially a biology, and we talked about this a bit, about how there are different languages, there’s the language of science,
there’s a language of law, there’s the language of humanities, and to be able to recognize that the way you’re presenting something could encode a different
message to someone else, and if we’re really trying to
embrace interdisciplinarity, we also have to be able to
talk in a common language and to appreciate how what we’re saying could potentially be, or not
be, taken as we intended it, so there are exciting
machine-learning ways to also get at that as well. – Others, why don’t we do here and here. – Maybe him first. – All right, let’s collect
a couple of questions and we’ll have our panelists answer them. Sure, we got a mic back here, go for it. – Thanks, just a quick one, I think this is mostly
for Wendy and Bridget but I’d be interested to
hear anyone’s thoughts on it. So, Wendy you mentioned this
kind of leap into fiction in that sociological article
you were talking about as something that could
potentially, at its best, sort of shed light on the made up-ness of the facts involved, right. And Bridget you closed talking about these AI-generated faces based
on DNA that are, you know, in a real sense fictional faces that are being used to guide policing in this litter project. And so I was just wondering
if you had any thoughts about, as someone who works on literal fictions, I work on 19th century literature, on if we can think about
AI as producing fictions, about machine-learning models as kind of generating fictions and whether that kind of ontological shift would have an impact on, kind of, what we think about the
goals of these models being? – All right let’s hold that one, and we’ll collect a couple here, how about up in the front. – Hi, so I have a question
mostly for Bridget which is like a lot of
critics of biometrics or facial recognition jump
to like a historical analogy with phrenology, and I think this is
because like phrenology did a lot to systematize
and worsen racial injustice, and also, because it
was kind of bad science like they thought they could
detect people’s personality from like these craniological measures, they had that time and they
were fooling themselves, and a lot of people think that like with adversarial examples
and not being able to generalize machine-learning has a lot of maybe similar flaws, so I’m wondering like, as
someone from your background, what do you think about
that historical analogy? – All right, can we get one more in there? Up front here. Michelle, so we’ve got
fiction, NAI, and phrenology. – So much, just a quick question, this is such a great panel, so exciting, I want to just pick up on
something I heard, first, congratulations on passing
that legislation, Matt, I think that’s kind of amazing, and all of you have
sort of have talked also about educating the
public or the industry, Bridget was pointing out
educating the public, but you also notice that you know, that there’s tech PR and lobbyists that have the ear of
influencers who are elected and not and I know you’ve
all sort of dealt with them, and so in the collaboration
with interdisciplinarity, of course, there’s also all
these competing interests and incentives and commercial
self-interest as well too, so I’m wondering, when
you were talking, Matt, about shaping the public
imagination, and you were, I’m really excited also to think about how the arts and humanities are
shaping perception then too. It’s sort of, I also heard
Reed Hoffman the other day and also have heard Eric Smith talk about how they don’t support this legislation, so I’m wondering as this plays out, if recruiting narratives
will be successful given the enormous amount of money in not regulating those kinds
of emerging technologies in the way that you were suggesting. – Thank you. Let’s hear about phrenology. – Yeah, let’s do phrenology.
Yeah, do that first. So, I will say first that I am a question, question, question person. So when I first got into anthropology and started thinking about ancestry and that issue of phrenology,
it of course came to mind, how am I doing something
different than phrenology? Why is this different, why
are all these critics who are, a lot of them who are anthropologists, biological anthropologists, wrong in when they say this is just an extension to phrenology. So actually, that was like the thrust of my dissertation project so I’m very passionate
about this, good question. – Good question. – I decided that within my
dissertation, for my dissertation I was going to turn
the mathematical models on their head and instead of using models for testing whether or not
we get the same answers, just with different data sets, in terms of, can we organize
people into geographic groups or is there, what we call,
biologically structure, so there’s some organization
in human biology that follows along some
cultural or geographic lines. Instead of introducing new data, ’cause that’s always
been like the solution like more data, more data,
more data, better models. What if we completely alter
how we think about the model, and take the classifier
out of the classification? And this is something that’s very similar to what’s also done in genetics. So, I stripped all the
data down so that it had no identifiers, and I wrote an algorithm that would look at the
data structure alone, those of you who are maybe
familiar with text analysis, there’s like this concept of bag of words where you sort of take everything away and you just look at the word patterns. I took all of those
reference information away, the metadata, created an algorithm that just created patterns based on the measurements alone, and it could have determined that there was no pattern whatsoever. So it would have been one
great big glump of everyone all amassed or it could
have assigned each person, based on their measurements,
to their own little category. So that instead of having
one group of 25,000, I would 25,000 mini groups. Instead, it actually grouped
people into evolutionarily and social-historically relevant groups. So then, it tells us that yes, there are aspects to biology
that are shaped by culture, that are shaped by our
evolutionary history, that are driven by climate, that are driven by social norms, rules against assortative
mating et cetera, that have changed how
our biology is expressed. So there is a way then
to get at that question of phrenology and still do it
in a mathematical way using, you know, cutting-edge technology that doesn’t necessarily
require you to take a position. I essentially let the data speak, and then use what I knew contextually, as an anthropologist, as someone who studied
history, and geography, and cultural anthropology
to make sense of that, and I think that also gets to your comment about authority and authenticity that numbers are numbers
until we bring context. So, there’s the opportunity then to bring that context to the numbers, to flush out the picture, to be able to determine whether or not it actually presents something
that’s authentic or real, and, as I’ve said in my talk, something can also be authentic and real and it doesn’t mean it matters, so just because you’ve
expended all this energy on defining something
doesn’t mean you should, and doesn’t mean that it
was worthwhile in the end or should be assigned some value just because it’s present
or absent or whatever. So again, context in that
regard is also important. – AI and fiction, AI as fiction perhaps. – That’s such a great question and I think that the relationship between fiction and technology
is such a large one, I mean. So, just a clear example is something like how did the internet
ever become cyberspace? Right, you know, read “Neuromancer” then look at “Gopher,” whoa. Two do not map onto each other, there was no jacking
and there was, you know, so I think that the
generative power of fiction is important, especially if
you look at something again like authenticity as
opposed to verifiability. Just telling a bunch of dead heads that you know there’s a
difference between jacking in and “Gopher” that’s not gonna do anything. The question was why and how
did it become felt as true? Like what was that capacity there? And I think there is
this generative moment that we need to engage and think through. – And I’ll just add that I think, so in terms of your question and in terms of literary studies, I think there’s great
value in studying fiction, because how people write about people and present the world tells us a lot about what we think and feel and believe and know and it helps us to come to that better understanding of that context. So sometimes the numbers
in the scientific articles don’t tell us as much as a description of the world in something that is fiction. Sometimes fiction can be
more true or of greater value in studying that in terms of
flushing out those pictures of the understanding of that context. – And finally, how can we challenge and reshape the narrative and the face of massive
PR, powerful PR machines? – Yeah, I would say that,
you know, right now, big tech lacks an affirmative
vision of how the world would be a better place
if facial recognition were pervasive in our society. And the fact that they lack that vision is not lost on legislators, and legislators when given the opportunity to vote against facial recognition and for what the voters want, are so far falling behind
the voters vision of things. I would also say in addition to that, that this sort of idea that platforms are not going to be
responsible for the ills that they cause in the world. That is something, I think, big techs sometimes clings to, still but it doesn’t get any
traction to policymakers now. People, policymakers, I mean you’re seeing this on
the news all the time, whether it’s Russian interference or some other thing
that the tech companies have had a role in, they’re rejecting, policymakers are rejecting this idea that a company like Amazon
should have the ability to throw its hands up when it sells an API to a law enforcement agency who goes and uses it to harm people. There’s not the disconnect that I think, tech would prefer to be there, so that is also playing a role, and then having pased
legislation in California, in Massachusetts, and here in
the state of California now, the status quo and this
sort of gold standard is moratorium or a ban. So the burden is now on
proponents of the technology to articulate that affirmative vision and to show how it actually
is good for communities and not bad for civil rights. – With that, I’d like to thank
you all for joining us today, (audience clapping) and thank you Wendy, Bridget, and Matt, for sharing your work.

About the Author: Earl Hamill

Leave a Reply

Your email address will not be published. Required fields are marked *