Mystery AI Hype Theater 3000

Linguists Versus 'AI' Speech Analysis (with Nicole Holliday), 2025.03.17

• Emily M. Bender and Alex Hanna • Episode 53

Share episode

Measuring your talk time? Counting your filler words? What about "analyzing" your "emotions"? Companies that push LLM technology to surveil and summarize video meetings are increasingly offering to (purportedly) analyze your participation and assign your speech some metrics, all in the name of "productivity". Sociolinguist Nicole Holliday joins Alex and Emily to take apart claims about these "AI" meeting feedback tools, and reveal them to be just sparkling bossware, with little insight into how we talk.

Nicole Holliday is Acting Associate Professor of Linguistics at the University of California-Berkeley.

Quick note: Our guest for this episode had some sound equipment issues, which unfortunately affected her audio quality.

Main course:

Read AI Review: This AI Reads Emotions During Video Calls

Zoom rebrands existing and introduces new generative AI features

Speech analysis startup releases AI tool that simulates difficult job interview conversation

Fresh AI Hell:

Amazon Echo will send all recordings to Amazon beginning March 28

Trump’s NIST no longer concerned with “safety” or “fairness”

Reporter Kevin Roose is feeling the bullshit

UW’s eScience institute pushing “AI” for information access

OpenAI whines about data being too expensive, with a side of Sinophobia


Check out future streams at on Twitch, Meanwhile, send us any AI Hell you see.

Our book, 'The AI Con,' comes out in May! Pre-order now.

Subscribe to our newsletter via Buttondown.

Follow us!

Emily

Alex

Music by Toby Menon.
Artwork by Naomi Pleasure-Park.
Production by Christie Taylor.

Alex Hanna:

Welcome, everyone, to Mystery AI Hype Theater 3000, where we seek catharsis in this age of AI hype. We find the worst of it and pop it with the sharpest needles we can find.

Emily M. Bender:

Along the way we learn to always read the footnotes, and each time we think we've reached peak AI hype, the summit of Bullshit Mountain, we discover there's worst to come. I'm Emily M. Bender, professor of Linguistics at the University of Washington.

Alex Hanna:

And I'm Alex Hanna, director of Research for the Distributed AI Research Institute. This is episode 53, which we're recording on March 17th, 2025. Today's focus is a hot new workplace trend. This is not just the usual surveillance versus LLM, but a fun new, terrible application of pseudoscience to measure your meeting performance against made up standards.

Emily M. Bender:

At this point, it's no longer new that your online meeting might be transcribed and summarized by an LLM. Zoom, and other platforms are practically throwing these features at employers. But now the lurking bots that power this transcription may also offer to quote,"analyze your emotions" or give other real time feedback on your performance and productivity in meetings, but again, based on what actual science exactly?

Alex Hanna:

As usual, using an LLM for this kind of metric guarantees harm anyone who's underrepresented in the training of that LLM, not to mention, offer your boss a new tool for monitoring and exploding all workers. And with us today to help unpack all that is Nicole Holliday, Acting Associate Professor of Linguistics at the University of California-Berkeley.

Emily M. Bender:

Woo-hoo! Nicole's research--

Nicole Holliday:

Hi.

Emily M. Bender:

Hi! Nicole's research focuses on the connection between linguistic variation and social identity, and more recently how speech technologies relate to both variation and inequality across social identities. Welcome Nicole.

Nicole Holliday:

I'm such a fan of the pod. Very excited to talk with you all today.

Emily M. Bender:

We are so thrilled to have you here to discuss this topic. We saw, we saw you posting about it. We saw that stuff, were like gotta get Nicole on the pod. This is amazing.

Alex Hanna:

Completely.

Emily M. Bender:

So I am going to bring us to our first artifact, um, from a very strange outfit called Unite AI. Um, and so this is a, an article posted, um, well updated October 19th, 2024, presumably posted close to then, uh, with the sticker "AI tools 101" and the headline "Read AI Review: This AI reads emotions during video calls." Mm. By Janine Heinrichs. So, um, I guess I'll just read a little bit of this to get us, uh, started."Have you ever left a video call wondering how your tone came across, or how others really felt about the conversation? Imagine if you could instantly understand what was said and the emotions behind it. I recently came across Read AI, an AI meeting assistant designed to analyze emotions in real time during video calls. Research shows that 93% of communication is nonverbal, meaning much of what we convey during meetings goes beyond words. Read AI taps into this hidden layer, offering insights into how participants are truly reacting and helping you fine tune your communication like never before."

Alex Hanna:

Oh my goodness. And if for folks who are not watching this, Nicole's got her hands just on her face just in horror.

Nicole Holliday:

This is a stronger claim than even Read AI itself makes. Like Read AI does not say that it can do this.

Emily M. Bender:

It's, yeah. So much magical thinking here."Imagine if you could instantly understand what was said and the emotions behind it." Well, first of all, um, incidentally usually if we're in conversations, we are understanding what was said. Um, and you know, to the extent that we are sensing the emotions of the people we're talking to, um, that's the best you can do, right? No AI system is gonna get in there and like, give you the truth of the matter.

Nicole Holliday:

No. And these systems, like they focus a lot on the speaker side of the equation. And, in as much as they focus on like what the listener is doing, it's much less information. And so this claim is just ridiculous based on how they work, because they're saying like, we're gonna help you understand how other people are interpreting you, but the system isn't even really looking at what other people are doing. It's only looking at you, the speaker. So it can't possibly know that.

Emily M. Bender:

Right, right. And as you were saying before the show, it's all fake. And it's like, in addition to being fake, it is like, we shouldn't even be trying to build something like this. If it doesn't work, it's bad. If it works, it's bad. It's like this is just, ugh. All right. So, uh, the author here, um, talks about discussing the pros and cons, what it is and who it's best for, and its key features. Um. I wanna get to their verdict here and then maybe the pros are kind of funny. So, "Read AI is a powerful tool for automating meeting summaries, realtime transcription and personalized coaching. It makes it easy to boost productivity and communication skills. However, limitations in picking up on cultural nuances, transcription accuracy, and free plan features can be a drawback."

Alex Hanna:

Yeah--

Nicole Holliday:

Cultural nuance is doing a lot of work right there.

Emily M. Bender:

Really, really is.

Alex Hanna:

It really is. There's also the like element of this as personalized coaching. The idea that like, this thing is gonna be a coach for your speaking, which itself is also very horrific. It's this kind of self, um, self-discipline, you know?

Emily M. Bender:

So looking at the pros and cons here. Um, so one of the pros is "Get customized coaching metrics and recommendations to become a better speaker. Um, what is that actually under the hood, Nicole?

Nicole Holliday:

Yeah, so I've, uh, I've got an experiment, the paper's under review, where I compare Read AI, and there's another one that we'll get to, the Zoom Revenue Accelerator, but, when you actually use this, it's a program that you kind of sign up for and it integrates with your Zoom. It joins your Zoom call as though it's another caller, and then it can give you some real time metrics. So the metrics you see in real time are things like a score for engagement and a score for sentiment. And these are like zero to 100 and they are green, orange or red, um, or green, yellow or red. And then it gives you a each rate, talking speed kind of thing. Um, and says if it's too fast or too slow. And then after the fact it gives you this kind of detailed coaching report. And in the coaching report it gives you a lot of alleged information. So it gives you an overall score of like the goodness of the meeting. These sentiment and engagement scores. It gives you one for charisma, bias, which they say is how positively or negatively you reacted when other people talked, talking pace, filler words, non-inclusive terms--which they say, "words that are potentially offensive or harmful"--interruptions, and implicit bias. And they define implicit bias as your visual sentiment towards the speaker measured out of 100. So when you're looking at it in real time, it's just giving you sentiment, engagement and speech rate, um, which it doesn't even measure very well. Um, and then at the end, it gives you this really detailed coaching report with all of this kind of like magical stuff, like charisma. I don't know what, what, what is it doing? Like how does it know that? Um, it doesn't, is the, the short answer. But, um, it really does promise like, we're gonna make you a good speaker. And in my experiment, I did this with 90 college students and I asked them at the end, once I kind of debriefed and explained what this system did, would you ever use a system like this? A full, like one fourth of them said that they would, because they wanted to be better speakers or sound more professional or something like that. Um, but this is like the, the systems are preying on people's insecurity, their linguistic insecurity, that there's something wrong with the way that they talk. And if you just tweak all of these mystery metrics that they don't define very well, or tell you how they're calculated, then you're gonna be a successful talker in business or professional settings or whatever it is.

Emily M. Bender:

Yeah. That's, that's infuriating. It's so, it's so infuriating and it's like, this is what we were saying before, right? It's, it's bad, it's fake and it's bad. So like--

Nicole Holliday:

And it's fake, and it's bad!

Emily M. Bender:

Yeah.

Alex Hanna:

It just, it's just the idea that you can quantify all this, like what kind of, like I, I guess they thought that you could take a charisma score like you have in Dungeons and Dragons and just say, we're gonna increase, you're gonna have 20 charisma right away or, and do the same. And then you're gonna have very low implicit bias. Abstract Tesseract says, "Your implicit bias, it's over 9,000!" And it just, the idea that you can like quantify these things is just absolutely mind numbing.

Nicole Holliday:

Well also that they're not context dependent, right? So like how you sound to another person is not only a function of like what it is that you are doing, it's a function of their previous linguistic experience and also what is appropriate, right? If we're talking about a real estate deal, that's very different than if it's me talking to one of my students versus, you know, just on a call with my best friend or anything like that. But these systems don't actually have that level of sensitivity. So really what they're missing from, my scholarly perspective is like sociolinguistic competence. When we encounter other humans, understanding meaning and style and you know, social things about them is part of what we've done when we've acquired a language. These systems have no sociolinguistic competence to speak of. They have a coarse metric, X is the right speed, X is the right number of fillers. And if you deviate from that, then that's a problem. But it doesn't mean that that is what matches your listeners' expectations.

Emily M. Bender:

Right. And it doesn't mean that like listeners' expectations or what you're trying to do, like sometimes you might be slowing down to make a point. Really hard for me to do. Or you might be like trying to slip a word in edgewise and that's why you're going fast. And another thing about this, which I think is a really generalizable like lesson about hype detection, is none of these systems are saying how they evaluated, how they defined and evaluated any of that. Right. So like--

Nicole Holliday:

No, no, there's absolutely no transparency. Um, they don't tell you what they're evaluating on the side of the listener, even if they were. So, and they say like, for bias, I gave you the definition and charisma, um, "how positively or negatively you reacted when others spoke." That's bias. Like, what do you mean how I reacted? Are you looking at my eyes? Like my face, like my mouth? Like the words, like the speed, like what is happening here that could possibly be generalizable to something like charisma.

Emily M. Bender:

Yeah. And back to your point about context, like if somebody's telling me a sad story, I'm going to react with things that should be, you know, perceived as negative. Right? That's, but that's not--

Nicole Holliday:

Yeah. The sentiment thing, going from red to green actually concerns me a lot for that exact reason, which is, there's this assumption that 100% good sentiment is what you're going for because of the way that it's sort of visualized. It's not, you're a psychopath if somebody's telling you a story about someone dying and you are like, 'Wow! 100% sentiment!'

Alex Hanna:

'I love it!'

Emily M. Bender:

'Good look!'

Alex Hanna:

I'm also thinking, I'm also thinking about situations when people are actively aggrieved. I mean, it's just like if you're in a majority white workplace and you are getting microaggressed and like, what I, I'm not going to stay there. Like, I have no poker face. I'm, I'm like, I will just like look really mad. And then like, what do you, and then it's like, then it's like, ding, ding, ding. You're not happy enough for this. Like, what the,

Nicole Holliday:

So my imagined use case for these systems is like, okay, I manage a team of realtors, let's just like go down this path. Right? Um. That when you are buying a house, I've never bought one, but I hear, um-- can't afford it. peak millennial-- um, that you have this kinda ongoing relationship with your realtor, right? So if you are having a follow up Zoom because they showed you a house yesterday that you just really hate, right? Um, relentlessly positive, that's actually not effective, like workmanship at their job. Right? They should be in solidarity with you to, in a pro-social way. So if you hate the house and they say, yeah, you know, that really wasn't the best one. Like I think we can do better for you. It's gonna get evaluated as negative sentiment.

Alex Hanna:

Right.

Nicole Holliday:

But it's actually appropriate.

Emily M. Bender:

Yeah. Ugh.

Alex Hanna:

That makes a lot of sense.

Emily M. Bender:

Right. So are there any pros and cons that either of you wanna lift up here while we're in this part of the article?

Alex Hanna:

They're so--

Nicole Holliday:

Oh "some vague", I love this."Some vague areas regarding GDPR compliance and sentiment analysis features do not adhere to the EU AI Act." Oh, this is a GDPR nightmare. And if there were any sort of AI regulation in the United States--cough, cough--um, it would be the same problem because first of all, there's all of this bias, and that's the other part of my research that we haven't even talked about. It has real issues with respect to race and people who speak English as a second language. Unsurprisingly. Um, so first of all it's an unequal impact thing, which is a violation of GDPR, but also it's a privacy, like a real privacy problem. Um, and when I've, I've given a few talks to, you know, linguistic colloquium audiences about this and people have said, well, wait a minute. If somebody, if some company is using Read AI and they're saying that my reactions are part of the feedback, did I consent to having my information recorded and stored by this third party company? No. I mean, I guess you did by having it join the call, but like it's analyzing your facial expressions. You are the customer, you didn't sign up for this.

Alex Hanna:

Right. And in terms of, in terms of the EU AI act, you know where the violation, I can think about where the violations are, especially around hiring and hire risk, but, um, what particular runs afoul there?'cause they're saying the sentiment analysis in particular.

Nicole Holliday:

Yeah, I think there's some things about emotion detection that GDPR is a little bit sensitive about, but also I think it's more like the, like storing people's likeness, doing like emotion detection on, on people that didn't consent to it is a problem for them.

Alex Hanna:

Yeah.

Emily M. Bender:

Yeah. So that sounds really well thought out, Nicole. The way they wrote this sentence tells me that the person who wrote this article actually does not understand this stuff. If you look at the sentence, "Some vague areas regarding GDPR compliance do not adhere to the EU AI Act." Like that, that "some vague areas" is, is half of the coordinated subject noun phrase, sorry, going all linguist on your Alex. Um, and, but GDPR is a different piece of law than the EU AI Act. Right, so--

Alex Hanna:

Yeah. One has to do with data and another has to do with the, with kind of modeling and EU AI Act has a little a little less to say about the data.

Emily M. Bender:

Yeah.

Alex Hanna:

And so you're just like, hmm, okay. I don't think you know what, what's happening here and why, I was like, I think the set of analysis, probably a few more things don't actually protect, like are also concernable.

Nicole Holliday:

The top, uh, con and the bottom con here are them like dancing in circles, trying not to say that it's racist, but they know that it is. So, "The sentiment analysis feature may miss specific cultural nuances, leading to misinterpretations in diverse cultural contexts." And then the bottom one says, "Occasional misrepresentations in nuanced tones and inaccuracies in transcription spelling." First of all, Emily, you and everybody else we know has written about automatic speech recognition errors for African American English speakers. That's what they mean. They mean that, first of all because there's an ASR as the first step of this, we know ASRS are just universally biased against speakers that they perceive as non-standard, that aren't reflected in their training data. It's just getting some of the words wrong. The other thing though is the, the specific cultural nuances, well some of that is just word level stuff, so, um, we haven't even talked about the non-inclusive language part. But one thing that Read AI flags as non-inclusive language is the second person, plural, use of 'you guys.' That is culturally specific. It is regionally determined. It is age. Like whether or not your audience will find that to be offensive is really particular. Right. And also, by the way, what if I'm talking to three guys?--saying 'you guys' and I really do meet you guys.

Emily M. Bender:

Yeah. So, so, so to your point about this, this author dancing around and not just coming out and saying, and not finding themselves able to say it's racist. Um, and this thing about, uh, diverse cultural context, Abstract Tesseract says, "Diverse cultural context. Uh, when they say diverse, who are they referring to and whose culture are they treating as the default?"

Nicole Holliday:

Mm-hmm. They never tell us, by the way.

Emily M. Bender:

Right.

Alex Hanna:

Yeah. I mean, with that, I mean, this is, there's a great line in "Thick" by Tressie uh, McMillan Cottom where she says, yes, when people say the verse, they mean Black. Like--

Nicole Holliday:

Yes they do.

Alex Hanna:

And they're like, and maybe you should say that, and again, like you said, you're saying cultural nuance. You're saying it's racist and we know. And we're just not gonna say that.

Nicole Holliday:

And my, my human, human research, like before I got into any of the speech tech or AI research, has looked at how African American English speakers might be misinterpreted as hostile in some setting settings. Because there are intonation differences that are, that just are not, like Black listeners don't necessarily hear them as problematic or hostile, but white listeners might because they have a different intonation system. And so that's like actually the more pernicious thing here, like the word stuff, word level stuff, maybe we can fix. But if it assumes that a particular intonation pattern is only one that's used by people when they're frustrated, it could be the case that it's used by white people disproportionately when they're frustrated. But it's not, when Black people use it, they, it's not a frustration tone.

Emily M. Bender:

Mm mm mm-hmm. Yeah. So the flip side of like failing to name racism and the, you know, the people who are being excluded here, I'm reminded of, uh, your, your rule, the Holliday rule.

Nicole Holliday:

Thank you. Yeah. So the Holliday rule is, 'it's all right to say they're white.' Um, because, I'm just so, and I, I am basing it on the Bender role. Um, but it's just like so ridiculous. Even, I've read papers, I teach papers to my graduate students where they're like, we had 80 Berkeley undergrads, and I'm like, uh, were they white? Because you didn't say, and if you didn't say, I assume that they're white, because if they were Black, you definitely would've said so.

Emily M. Bender:

For sure, for sure. Yeah. And this is, so the Holliday rule, um, is a much snappier aphorism that sort of calls back to the Bender rule, which is, 'always name the language you're working on, even if it's English.' It's not as, it's not as snappy. I think I would score lower on charisma.

Nicole Holliday:

Oh, it's because you didn't rhyme. You gotta rhyme.

Emily M. Bender:

Gotta rhyme. Yep.

Nicole Holliday:

It's alright to say they're white.

Alex Hanna:

Oh, okay. Yeah, it's, uh, uh, nope. Not gonna get you. Like, I can't build on that. There's, there's nothing that rhymes with English that's that snappy.

Emily M. Bender:

Yeah. Yeah. Have to, I have to keep workshopping it. All right. Is there anything else you wanna do in this artifact before we move on to the next one? Um.

Alex Hanna:

There, there's a few things in here, which I found. I mean, it, it goes into it. I mean, like, that's kind of the executive summary. Um, there's some very silly kind of reductivist in terms of the AI and machine learning technologies used. VO and it says--um, so scroll down a little bit more and it says, um. Oh gosh, wait, no, before that there's like, oh, there's a thing here in terms of, um, it's just not, it's not just for the corporate world, which we're ta--we've talked a little bit about."Educators can use Read AI to track student participation and engagement in online classes, which can be be tricky in a virtual classroom." No.

Emily M. Bender:

No.

Alex Hanna:

No.

Emily M. Bender:

Fire.

Alex Hanna:

No, I hate that. I hate--

Nicole Holliday:

No, I, yeah, the students. Okay. So it was interesting to get the sort of qualitative feedback from the students in the study that I did, because they said, oh, well, you know, I could use it to know if I'm using too many fillers if I have a class presentation. Like they were thinking about this kind of use of it, and I had to keep being like, well, who told you you use too many fillers? And he is like my, my parents, my uncle, my teachers. And that's just old people hating young people. Like there's no objective too many fillers, right? And so you're handing over the power for sounding professional or sounding academic to this system that has these built-in assumptions that aren't necessarily appropriate. The other thing that the student said is like this one, Read AI, gives you real time feedback. And so some of them, I didn't tell them what the software did before. I just had it join the meeting like it would if they were a client of mine or something. And then they asked about it after the fact and I debriefed 'em and they said, that was freaking me out. Like getting these numbers on the right that were moving and changing, like made it so hard for me to actually function because I was really being worried about how I was evaluated. So in a educational context, I think that's particularly bad. Like we want our students to be able to say something contentful and not only focus on the style. Like this is a AI equivalent of you get your paper back, your physical paper back in the old days from your teacher and they've just underlined every single like preposition at the end of a sentence and everything you misspelled and didn't actually even read it for content. That's what this does.

Alex Hanna:

Yeah. Yeah. Absolutely.

Emily M. Bender:

Yeah.

Alex Hanna:

Just like a nightmare. I'm just trying to think about what it would be. It'd be so just awful to be a student in this context and just having kind of a live grade, I'm like, no, that's not gonna encourage conversation. That's, they're gonna, they're gonna be afraid to speak.

Emily M. Bender:

Yeah.

Nicole Holliday:

And it's, who does it prey on even more? Right? Back to that linguistic insecurity thing. Yeah. If you are super comfortable in your online fancy class at Berkeley or you know, where I work or wherever, then you don't care about it evaluating you. But if you came from a community that speaks a marginalized variety of English and you're already like first gen, low income kid, and the system's like, nope, too many fillers, you're never speaking up in class.

Alex Hanna:

Yeah, absolutely. Absolutely.

Emily M. Bender:

So, Alex, there's something you wanted to do down here in the--

Alex Hanna:

Oh, I mean. Oh yeah. Wait."The AI and machine learning technologies in Read AI are pretty impressive." Uh, I just, this is just for, this is like a nineties, like, like I got a friend and he's pretty impressive, anyways."They combine natural language processing, NLP, and computer vision." Like, okay. Okay? And? You know, so just, it, it just--

Nicole Holliday:

I'm not an eye tracking expert, but I have heard that laptop based eye tracking is really, really unreliable. And so in as much as they're looking at where people's eyes are, uh, in the call, that's first of all the, the computer, like the systems are just not good. That, like, your laptop camera could be really bad. But also think about what we do in normal Zoom calls. If you're taking notes, you're gonna look down and type or write with a pen, or, you know, sometimes we take calls because there's other people in the room. We got dogs, we got kids, we got enh, right. So like the speaker is getting dinged because I am distracted today. That's what it means.

Alex Hanna:

Yeah. Yeah. A hundred percent. And it, yeah, just, and the description of the, of the technology, "The NLP technology handles the transcription and understanding of what's being said." Ugh. And then, "Meanwhile, computer vision picks up all of the non-verbal cues." Ugh. Like those--

Nicole Holliday:

It's really anthropomorphizing here too, right? Yeah. The vision, the understanding.

Alex Hanna:

It's just, it's really, I mean, I mean, I mean, this is, there's a lot here. So, I mean, uh, and then they've done a lot of the stuff in the pros and cons, but just oh, absolutely, like terrible stuff in this document.

Emily M. Bender:

And the stuff about privacy just was ridiculous. So, um, so, "Despite some concerns, Read AI takes privacy pretty seriously. All the data is encrypted and you have control over what's stored and for how long."

Nicole Holliday:

Was this written by AI?

Alex Hanna:

It may, it reads very LLM like, like it's, it may, it's it's it's pretty generic. Yeah. Like someone dumped like a, it, it feels like it, but. Yeah.

Emily M. Bender:

So "Regardless, you'll want to let your team know you're using Read AI for meetings to ensure everyone is comfortable." Let them know, to make sure they're comfortable. That's not, that's not how that works, right?

Alex Hanna:

No.

Emily M. Bender:

"Discuss the data you're okay collecting and how you'd use it and set up some ground rules so everyone is on the same page." Ground rules for what? The tech does what the tech does, right? Like Read AI is collecting a certain amount of data and it's not like the, the different companies using it get to like set up ground rules. Um, but then, "Ultimately, Read AI is a tool to help you communicate better, not to spy on each other." That really does, that really is giving LLM, isn't it? Um, so I wanna also point out that, um, when we're talking about you know, eye tracking and other visual cues, you also have to think about people who are neurodivergent, right? So Polerin in the chat says, um, "I generally don't watch the exact thing on the call. I'm usually doing something else to focus on them. Simple card games, et cetera, help my ADHD self not lose track of what's being said." Right? And then there's folks for whom eye contact is really difficult, and I'm not quite sure how that plays out over Zoom, but I'm sure it's in there somehow, you know?

Nicole Holliday:

Yeah. So this is the, like, one of the gut punches about my research too. Um, I was doing this in a laboratory setting, and so the non-inclusive language stuff didn't come up very much except for when sometimes people use you guys. Um, but the only other place that it flagged non-inclusive language was when neurodivergent people were talking about themselves.

Alex Hanna:

Oooof.

Nicole Holliday:

I had a quest--I did an interview with the students-- I know it was like, really bad. So I did an interview with the students, uh, and I asked 'em like where they're from, you know, what are, what's your major? The kind of stuff we ask. And I said, because I was worried about this. Do you consider yourself neurodivergent at all? If so, how? And if they said, um, I have ADHD, or I have autism spectrum disorder, or I have OCD, which people did, those were universally flagged as non-inclusive language. Saying 'ADHD' means that you're using non-inclusive language. So their overall scores were dinged because they said something about themselves that this system just saw as non-inclusive.

Alex Hanna:

Oh my God. Wow.

Nicole Holliday:

Yeah.

Alex Hanna:

Wow, wow, wow. Oh dear. Okay.

Emily M. Bender:

Alright. So, um, I think, I think we can probably, like, it doesn't get any better, folks, if you wanna read the whole artifact, it'll be in the show notes.

Alex Hanna:

Yeah, it's, it's pretty bad. I mean, you might wanna, you might want to, uh, have a whiskey or something if that's, if that's your, or some kind of--

Nicole Holliday:

There's something, yeah. There's something sort of bigger picture here that I think like maybe if you're not a linguist, you, you're kind of like, it's not apparent to you. But all of this is what Rosina Lippi-Green calls like the mysticization of language. Mm-hmm. And this is one of the things that she mentions as like upholding the idea of standard language ideology. That there is a right way to speak. And that, that speaking that way will give you access to levers of power. The mysticization of language, like, 'you are bad at language, you need to spend money on all of these tools to be good at it, here, let me sell you something so that you can be better about it. Oh, and let me help you enforce people that are below you on a social hierarchy by giving you the, like, the tools.' And so that's all this is, right? Like, you are not good at talking. Don't worry, for $12.99 a month, we'll fix it for you by giving you this, like, honestly doing cop shit on your meetings.

Emily M. Bender:

Yeah, exactly. Or $12.99 a month from your employer, and then the employer is gonna do this on you, right? And there's actually in the chat, there's a pretty good exchange about that. So when we're talking about like, you know, make sure your team is comfortable, Faster And Worse says,"If they're not comfortable, then what?" And then in quotes, "'We paid a lot of money for this subscription.'"

Alex Hanna:

Yeah.

Emily M. Bender:

Yeah. So we, paihad a lot of money to subject you to prescriptive language technologies and this is your job, so you gotta deal with it.

Nicole Holliday:

That's my cover term for this and the other thing, we'll talk about, the Zoom Revenue Accelerator. I, I took me a long time to like coin a term for them--

Emily M. Bender:

It's a great term.

Nicole Holliday:

Thank you. Socially prescriptive speech technologies. So they pre, they're so, they do social prescriptions, they tell you how you should be in a social setting.

Emily M. Bender:

Yeah. So what's this tech for? It's for doing that prescriptivism.

Alex Hanna:

Yeah. Yeah. Yeah. Automating prescriptivism, wow.

Emily M. Bender:

Should we switch that other artifact?

Alex Hanna:

Let's do it.

Emily M. Bender:

So this is a little bit older. Um, this is Kyle Wiggers writing in, uh, TechCrunch in September of 2023. And the headline is "Zoom rebrands existing--and intros new--generative AI features." Um, so I'll just read a little bit to get us started. Or do you wanna do the reading, Alex?

Alex Hanna:

Yeah, why don't we get into it? So, "To stay competitive in the crowded market for video conferencing, Zoom is updating and rebranding several of its AI powered features, including the generative AI assistant formerly known as ZoomIQ." First off, love that they're moving away from the notion of IQ, but not like it's any better."The news comes after controversy over changes to Zoom's terms of service, which implied that Zoom reserved the right to use customer's videos to train its AI tools and models. In response to the blowback, Zoom updated its policy to explicitly state that 'communications-like'--" That's a hyphenated term."--'customer data won't be used in training AI apps and services for Zoom or its outside partners.'" Okay. And then there's a call to ditch Zoom and Zoom wrote this bullshit press release. But getting down into the Zoom AI companion, "the rebranded Zoom AI, Zoom IQ--" Sorry. Too much--

Emily M. Bender:

Or, or as SJayLett says in the chat, "Presumably IQ is pronounced 'ick'."

Alex Hanna:

Yeah, I love that."--called the AI Companion, is powered by the same mix of technologies such as, uh, technologies as ZoomIQ, Zoom's in-house generative AI, along with AI models from vendors including Meta, OpenAI and Anthropic. But its reach is expanding to more corners of the Zoom ecosystem, including Zoom Whiteboard, zoom Team Chat, and Zoom Mail--" Which I'm assuming it's just email. And so the, the, the last two get into what it actually is."Perhaps the biggest news is that Zoom is gaining what's essentially a ChatGPT like bot via the AI Companion, and the spring of 2024, Zoom will get a conversational interface to out that will allow users to chat directly with the AI Companion and ask questions about prior meetings and chats as well as take actions on their behalfs. For example, users will be able to query the AI Companion for the status of projects, pulling on transcribed meetings, chats, whiteboards, emails, documents, and even third party apps. They'll be able to ask the AI Companion questions during a meeting to catch up on key points, create and file support tickets and draft responses to inquiries. And as is possible with Zoom iq, they'll be able to ha, they'll be able to have the AI Companion summarize meetings, automatically identifying action items, and surfacing next steps." Terrible.

Emily M. Bender:

Yeah, like here, let's just put a chaos agent in the middle of all of your communications in your company.

Nicole Holliday:

You when you open Zoom now, um, 'cause this is a little bit older, you do get AI Companion, it's like down at the bottom where share screen and everything else is. Um, there's, there's like another layer here, right? So AI Companion is consumer facing. Everybody has that. There's another sort of beefed up version of AI Companion that's called Revenue Accelerator.

Alex Hanna:

Oh, I love it. Absolutely.

Nicole Holliday:

They even named it evil. Yeah. Um, so for short, ZRA is Zoom Revenue Accelerator, and it does all of this, the meeting summary and whatever, and that first of all, that seems like it's not evil, right? Like, oh, great, it'll take notes for me. And, you know, it does meeting summary. I'm not sure that that's not evil.

Emily M. Bender:

That's quite evil, right? You think about, you know, whose language is less likely to be transcribed accurately, right? And then people start relying on the summary instead of listening to people in meetings. No, it's, it's evil, but it gets worse, right? Yeah.

Nicole Holliday:

Yeah. And even like non linguistically, right? Um, there, uh, there was an article in New York Magazine last week about Read AI and that author was basically saying like, you know, it kind of feels like all meetings are now emails. Then like, what are you, what's your incentive to go to a meeting if everything's gonna be noted and transcribed? And we know that like what you get from reading is not the same as what you get from interaction. So it's telling everybody like, it's okay if you're not paying attention or if you skipped that meeting. It's encouraging them to not participate and then by not participating, the quality of the meeting for everybody goes down. So just the transcription part is already a problem, but then the Revenue Accelerator goes beyond and does some of the other stuff like the Read AI that we were talking about. So it gives you all this coaching and the metrics are slightly different, but same vibes.

Emily M. Bender:

Hmm. I'm reminded of the, the episode we did have to look, look up which one it was. Zoom CEO has this dream of individual LLM avatars that attend your meetings for you. So that nobody has to go to the meetings.

Alex Hanna:

Yeah. And this is--and in the chat Timnit, Timnit Gebru is in the chat, and she says--

Nicole Holliday:

Ah, hi Timnit Gebru.

Alex Hanna:

She says, oh, so that I have meetings with the AI Companion, uh, or what's the purpose of this? Or so that my AI Companion talks to someone else's AI Companion. And it's really, there's a vision that, that Eric, uh, Yuan has this, just AI Companions talking to AI Companions, and I can be at the beach, which I think he says. And, and thanks Christie, our producer says, that's episode 38, "Deflating Zoom's Digital Twin," which we--

Emily M. Bender:

Thanks Christie, yeah. I was just, just looking.

Alex Hanna:

From July. Last July.

Emily M. Bender:

Yeah. Uh, so, so Nicole, you studied this one too, like that experiment I did. You looked at both of them. Um, and let me guess. They, neither of them was any good.

Nicole Holliday:

No, neither of them is any good. Um, they have slightly different kind of bias problems. So I dug really into the--the top line scores that they give you are sentiment and engagement. And Read gives you this like overall score, but Zoom doesn't. It just gives you scores for sentiment and engagement. Um, for Zoom, uh, base, for both of them, the sentiment scores and the engagement scores were lower for Black participants than Latino participants, Asian or white. Um, so there is this kind of pervasive racial bias. Um, Read AI prefers first language English speakers, um, for sentiment. Um, but the Zoom Revenue Accelerator downgrades first language English speakers. I've no idea why these systems don't agree on things.

Emily M. Bender:

Um, because it's all fake.

Nicole Holliday:

Because it's all fake. No, and you know how it, how I know it's fake. Like this is the, the smoking gun for me, the things that are supposed to be more objective. So you would think like we can count fillers, like we know how many fillers there are. They're a quantifiable thing. We can count how fast someone is talking, speech rate, right? It's like words per minute or syllables per minute or something like that. They don't agree on how many syllables per minute the same speaker is using for the same stretch of speech. But they both agree that everybody's talking too fast. So the Zoom--yeah, yeah, yeah. The Zoom Revenue Accelerator said that 89 of my 90 or 87 of my 88 participants were talking too fast. Um, and the uh, or sorry, no, it was 69 out of 88 were talking too fast. 87 out of 88 using too many fillers. Um, Read AI was a little bit more realistic. So it said that, um, 17% of the sample was using too many fillers and uh, it was like 65% were talking too fast. But like, what is too fast if everybody's doing it?

Emily M. Bender:

Yeah, exactly.

Nicole Holliday:

And here's the, here's the like real kicker, right? The scores--for the ZRA, the scores that ranged, uh, from uh, all in the too fast bin were 120 words per minute to 227 words per minute. 227 words per minute, most humans are gonna be like, whoa, that's real fast. 120 is much slower than anybody on this call has talked this entire time. Um, and then it gives like"fast," "caution," and whatever. The categories are entirely overlapping. So if you were a person whose speech rate was 161, which I think is a little slower than I'm talking now, you could have gotten "fast" or "caution" or"good." It's not actually counting words by per minute, because that same number for words per minute could get any sort of evaluative label.

Emily M. Bender:

It's all fake.

Alex Hanna:

Wow.

Nicole Holliday:

It's all fake. And it says specifically, it's counting words per minute. So like where is this label coming from if there's no cutoff number for words per minute.

Emily M. Bender:

Yeah.

Alex Hanna:

That's wild. That's wild. Just like, I mean, it's just like, it's, it's all fake and just terrible engineering, it sounds like.

Emily M. Bender:

Yeah. Yeah. I mean, you, you might be able to imagine some scenario where if it was like individual opt-in, I'm running this, no one's running it on me. I would like to know how many words per minute I'm speaking.

Alex Hanna:

Mm-hmm.

Emily M. Bender:

That might be something that's useful in some contexts, but I would like to know that the thing is accurate and I would like to know how it was evaluated, how it was benchmarked, and like what kinds of speakers under what circumstances, like, so that I could actually rely on it. Um, and--

Nicole Holliday:

Also people converge in speech rate, and when people converge in speech rate, this enhances intelligibility. So even if you have in your mind, like, I wanna speak at 160 words per minute, if the person you're talking to is at 120, you should move to 120. If they're at 200, you should more move towards 200. Right, because that's actually gonna make both of you have a better conversation. So the idea that this isn't something that's being negotiated in real time, that's absolutely built into this system, is that's not how language works. Like that's actually my big point about all of this. None of this is how language works for humans,

Alex Hanna:

Right. Yeah. Someone took, yeah, someone took this number and they're like, this sounds okay. And Christie points out that 120 is like scripted NPR piece rate. And, and kind of what, what she was using when she was doing science explainy things. And I mean the same with like audio books tend to be, you know, I'd, I'd imagine somewhat around that rate. And it's like, that's like, that's not a meeting, that's not, that is, that is broadcast, right.

Emily M. Bender:

Exactly. And that's not a situation where you can easily sort of jump in and, and add something someone said. And, but I guess I'm sitting here thinking of like, where could I productively use a word rate counting thing on myself? And um, so my dad's hearing situation is, he's got some sort of a processing thing going on where if I talk too fast or if I talk my normal speed, he can't understand. So he is always asking me to slow down. So I could imagine like, it might save him some grief if I had a little app that I could keep an eye on and I could slow down before he had to ask me.

Nicole Holliday:

But Emily, you already know that. Like you already know that you're not supposed to talk as fast to him as other people. And so do we. When we encounter elderly people or children, almost all of us slow our speech rate. We slow our speech rate when we encounter people who are English language learners. But when we're with people that we know that we're comfortable with, we're doing routine eyes interactions like, Hey, how's it going? Da da da. We talk really fast. If you have a sort of routinized, uh, interaction with someone you talk to all the same all the time, that is demographically the same as you and you talk at 120, they're gonna be so bored.

Emily M. Bender:

Could you just get on with it, please?

Nicole Holliday:

Yeah.

Alex Hanna:

Oh, geez.

Emily M. Bender:

Yeah.

Alex Hanna:

And if I'm getting, like, even like if I'm getting heated at my mom, like, first off, I'm gonna be switching between English and Arabic, but I'm gonna be going very fast.

Nicole Holliday:

Oh, we haven't even talked about the fact that this cannot handle code switching.

Alex Hanna:

Oh, oh, absolutely. Absolutely. Absolutely. Yeah.

Emily M. Bender:

Yeah.

Alex Hanna:

Geez.

Emily M. Bender:

Oof. All right. Do we have anything else in this TechCrunch piece that we wanna get into here? Um, let's see. Uh, the, the, the, um, this, even though Zoom, the company is moving away from the IQ, this journalist sure wasn't. They seem to be bringing up at every opportunity. Zoom IQ, Zoom IQ. Um.

Nicole Holliday:

There's something a little, like, the most nefarious thing that I think that exists in any of this, uh, the Read AI or the Zoom, is the documentation for the Zoom Revenue Accelerator is really poor. And it also does not describe how any of these things are calculated. But there's a webinar, you can find it on YouTube, um, that's like 'introducing the Zoom Revenue Accelerator' and it gives you, you know, all of the things that it's supposed to do. And it shows all these employees on a call ranked by sort of their goodness, uh, according to the ZRA in the call, by name, um, like, you know, up and down ranked. And there's an asterisk on the bottom of the documentation that says, "Zoom Revenue Accelerator is not to be used for employment--hiring, firing, or other comparable employment decisions." In your webinar, you are ranking employees by goodness. You are advertising it to do exactly that. And so it's just like a legal disclaimer. But what it's saying is like, hey, and the Zoom Revenue Accelerator, this is important, isn't something that the employee can see. Read AI, you can see. Your manager sees the ZRA. So if it's giving you a bad score for sentiment or engagement or saying you're talking too fast or whatever, you're never gonna know. One day you're gonna get fired and they're gonna say, well, you're not good at talking, according to the ZRA.

Alex Hanna:

Ah, geez. Nightmare.

Nicole Holliday:

And given that it's biased, this allows people to run absolutely afoul of EEOC, like equal employment kind of regulation. Because it's that the mystery, right? There's nobody to blame for the bias.

Emily M. Bender:

Exactly. Oh, we, well, we're just following what the computer says. We're treating everybody equally. If the computer says they're bad, then they're bad.

Alex Hanna:

Yeah. ugh.

Emily M. Bender:

Yeah. Um, what that's, I mean, it's bias washing, right? There's another word for that. But basically the idea that, that the computer is objective and nevermind that these technologies were designed on data sets that are not representative, we can forget that when it's time to use them. Yeah.

Alex Hanna:

Good lord. Uh, some, it's just very interesting too, because at the end of the article um, we get into the financial aspects of it, which like, uh, which with the journalist Kyle Wiggers, um, who is usually pretty good, uh, has uh, has some notes."It comes for a pivotal moment for the tech giant, which faced it's first quarterly loss of $108 million since 2018." Um, and then, and then it laid off talking about all these layoffs. I mean, this is kind of old and I imagine it's a bit, um, 'cause, and I was actually just checking Zoom's, um, uh, uh, stock price, I mean, had a huge, uh, parabola around the pandemic. And this has completely come down to pre pandemic rates. So just lost just a bunch of value. So, like many companies just trying to shore up some of that, uh, some of that valuation stock price through these magical thinking features.

Nicole Holliday:

That's it, right? Like they are saying, you're just not good at talking. Let us fix it, pay us--and by the way, the Zoom Revenue Accelerator, very expensive compared to Read AI. Um, I think I got like one, just for one person and it's $80 a month.

Alex Hanna:

Oh geez.

Nicole Holliday:

So I have no idea what these companies are paying. Thousands and thousands of dollars to use a system that doesn't actually give them more information about how good people are at their jobs.

Emily M. Bender:

Right. But pretends to, which can be convenient sometimes. Right. Yeah. Oof. All right, well I guess we've got a couple minutes to look at this one last one, which is from GeekWire. And this was sort of like disappointingly rah rah from-- like GeekWire I think often does pretty good coverage. But this, I was not impressed. So this is from May 1st, 2023 by Nate Beck, and the headline is, "Speech analysis startup releases AI tool that simulates difficult job interview conversations." And this also is like. Uh, there's a trope where it's like, where we're actually creating some surveillance tech, but we are going to wrap it up as if it were a good thing to help the, you know, poor individual person who needs a better chance at getting a job or whatever. Yet again, you know, doing the thing of, you know, uh, stimulating and simulating linguistic insecurity.

Nicole Holliday:

Yeah. This is, this is interesting 'cause it's a, a different technology that I haven't actually studied. But comparing the marketing of Read AI and Zoom Revenue Accelerator is really interesting because ZRA is for corporate clients and so the whole marketing is like, look at how much more money we can get you. We can make your employees more effective, et cetera. Read is supposed-- it's a little progressive and you can tell because they're like, don't use non-inclusive language, don't be biased. Right. We're gonna make you a better human by like making you talk, you know, in like a progressive way. This one is like, don't like what you're saying? Don't worry, we're gonna like fix it so that you have economic opportunities.

Emily M. Bender:

Right. Right. Exactly. This, this is for you. So this is, um, the, "The new tech uses generative AI to produce realtime follow up questions based on a user's responses to questions for a mock conversation. Yoodli also introduced a personal speech coach that can be used during live discourse, providing immediate contextual feedback and analysis." So this is the same type of like, you know, we're gonna judge your, your speech based on, you know, hidden fake metrics based on biased training data and make you feel even more insecure. Right. But, um--

Alex Hanna:

Geez.

Nicole Holliday:

And in a high stakes situation too. And, and what happens, right? Like, is anybody ever going to say like, I followed, Yoodli's suggestions to a T and then I didn't get the job? Like, you can't ever know that because there are too many variables when you're dealing with conversations.

Alex Hanna:

Yeah.

Nicole Holliday:

You can get, you can achieve its ideal number of fillers and speech rate and, you know, interruption speed and all of this other kind of stuff, and that person could still just not like you. Or maybe you're not qualified for the job, or not good at interviewing,

Emily M. Bender:

You'd also be actually a really awkward interviewee if you were like, completely focused on speaking rate rather than doing, you know, relating to the context as you keep telling.

Nicole Holliday:

That's what those students were saying to me, like when they saw the Read AI feedback, they, you know, some of them were like cognitive science students and like, 'the cognitive load is too high. Like I can't focus on giving a good response to the, the answer when I have to be like, I'm using five fillers per minute and I'm speaking at 150.'

Alex Hanna:

God. Just a nightmare scenario. I mean, to think that like you can coach this out and then have to pay attention to this and do all this thing, and I mean, uh, I mean, I can see that this is like, this, this kind of thing might appeal to like a very particular sort of individual, but God, like for the, for like most people. Good Lord. I don't want these, I don't want these things metricized.

Nicole Holliday:

This sentence here, that's, um, in the middle that you've got right up right now. So the, Puri did a job, or Puri did a job mock job interview, right?"The tool found that they used the word filler word 'um' 10 times--" In what time period? I guess in a minute? Um, "His pacing was a bit fast, speaking at 180 words per minute versus the recommended 170." The difference between 170 and 180, depending on what words you're using, is literally nothing. Um, but it's gonna make the guy feel bad for using 10, you know, words too many in a minute, I guess. And 10 ums is also not a, like, what does it mean, 10 ums per minute? So in my study, looking at the Zoom Revenue Accelerator, any number of fillers, which it never tells you what the fillers are by the way, I guess um is probably one of them. Any number between 5 and 25 per minute was too many. 5 and 25--25 you're gonna hear as a lot, probably. 5, absolutely not. So it's just like zero is what it thinks is right, but zero is not right. We have a lot of psycholinguistic literature on this. And 10 ums, especially if someone's talking about something really complicated like you might be doing in a job interview, is a totally reasonable number of ums. If you use no ums, you also sound really weird.

Emily M. Bender:

Yeah. You sound like a robot and you are not giving the listener the cues that the ums actually give them. So I have to say, I, I get to teach, uh introduction to syntax for computational linguistics every fall out of the Sag/ Wasow / Bender textbook and chapter nine in there is about psycholinguistics. And so there's one lecture every fall where I'm talking about some research on filled pauses. So filler words and how they're distributed syntactically. And I hate giving that lecture because it totally sensitizes me to my own use of filler words that mostly I don't think about. And I could just imagine that anything that's like drawing your attention to the filler words, given the discourse around them, is gonna make you insecure, and it's gonna make it just hard to talk, because I go through that every November.

Nicole Holliday:

Do you get a lot--this is, um, so I sent a version of this research to somewhere and I got an R and R, so I got feedback. And the person was on board with my criticisms, but they said, 'Yeah, but people really do use too many fillers. Like that's gotta be a good use.' Um, do you get pushback from this student? People hate filler-- they've been like really beaten outta people.

Alex Hanna:

That's really wild.

Nicole Holliday:

Students in my study, they were like, oh, I say 'like' too much. I say 'um' too much. They say, it makes me sound stupid. And I was like, no, it doesn't. If people are giving you a, a hard time about your number of fillers, they hate you for some other reason that's probably not linguistic.

Emily M. Bender:

Yeah, absolutely.

Nicole Holliday:

And it's probably ageism or sexism or racism or something else.

Emily M. Bender:

Yeah, no, I, the way the lecture that I do is sort of about how, if you've got earlier in the sentence, you're more likely to get false starts and filled pauses versus later in the sentence. And so it's evidence for sort of incremental processing as we're speaking. Like that's, that's the point of the study. And the flip side is that tells your listener, oh, long, complicated NP is coming, probably. Right. It's, it's useful information. I'm gonna be ready to process a more complex noun phrase because I've just heard an um, or repeated the, and so that's the angle that we're talking about it on. And then I also just sort of say, hey, this makes me feel, you know, self-conscious. And I think the students are just sympathetic about that.

Nicole Holliday:

But that's another level here too, right? So maybe not all ums are created equal. If they're a sentence initial um, that might not be received by your listener the same way as if it's later. But this system doesn't know that.

Emily M. Bender:

Exactly. Exactly. They mean different things depending on the intonation, depending on where they are on the sentence, depending on which filled pause you've used. Right. Um. See now? No, I've sensitized myself. Dang it. Okay. Um, so actually it is time for us to move to Fresh AI Hell, and we have this tradition of, of I give Alex a, um, um, improv prompt, Alex musical or non musical this time.

Alex Hanna:

Now I got, now I'm thinking about my ums. Uh, musical. Sure.

Emily M. Bender:

All right. You got a style for me?

Alex Hanna:

Uh.

Emily M. Bender:

A a a style where the, the tempo can change.

Alex Hanna:

Where the tempo can change.

Emily M. Bender:

Yeah.

Alex Hanna:

Oh, uh, then it's gotta be, um, I guess Irish jig.

Emily M. Bender:

Okay. So you are Fresh AI Hell demon if you like, or not if you like. In a meeting with one of these things coming and realizing that you have to get your speech rate higher.

Alex Hanna:

Oh, okay. Oh, interesting. Uh, let's think, I'm like Come ye lads to the promised land on Zoom Revenue Accelerator. We're going to get and see the rate at which we go and have our fate. We're going down to the keg and having us a bit of dreg. We're going now to the time. I can't think of anything else. Um, um, um, um, um.

Emily M. Bender:

Fantastic. Alex, you have outdone yourself.

Nicole Holliday:

It was good.

Alex Hanna:

Thank you.

Emily M. Bender:

Okay. That, um, um, um, um, um, gets us into Fresh AI Hell, starting with this very recent piece from Ars Technica. The sticker is 'Alexa, should I trust Amazon with my voice recordings?' And the headline is, "Everything you say to your Echo will be sent to Amazon starting on March 28th." Subhead, "Amazon is killing a privacy feature to booster Alexa Plus, the new subscription assistant" by Sharon Harding on March 14th.

Alex Hanna:

Good lord. Uh, terrible. I mean, you should be, you should have been destroying your Amazon Dots or Echoes or whatever yesterday, but Yeah, now it's the next best time.

Emily M. Bender:

Exactly. Um, so I guess that it used to be that things were, processing was happening on device until it was some query that had to be sent and now it's all going to the cloud. Um, including presumably whenever it's sitting there waiting for the wake word.

Alex Hanna:

Yeah. So now it's, I mean, it's probably specifically if they're doing it on device now, the models have just gotten so large, they're like, well, we have to send this to the big LLM in the sky.

Emily M. Bender:

Yeah. Ugh. And I wonder like if we're gonna start developing norms like at dinner parties, around people disclosing the presence or absence of this kind of technology in their house. Um, so, okay. Uh, Alex, you wanna do the honors on this one?

Alex Hanna:

Oh Lord, if I must, so this is from Wired, the journalist is Will Knight. This is from March 14th of this year."Under Trump, AI scientists are told to remove, quote, 'ideological bias', unquote, from powerful models. A directive comes from the National Institute of Standards and Technology, uh, which eliminates a mention of quote, 'AI safety' and 'AI fairness'." So we already, you know, are, you know, we, we've, talked about the kind of in, um, incommensurability of AI safety and AI fairness. Now they're just like, you know, both are bad and you should, you shouldn't talk about, uh, bias at all. Uh, you should, or rather reduce ideological bias, in this way being the bias against conservatives. Now the new thing says you need to "enable human flourishing" and, um, and "economic competitiveness". Now I am first off, annoyed at the, uh, the infection of, uh, of human flourishing becoming a term used by absolute ghouls, 'cause I actually quite like that terminology following Erik Olin Wright in "Envisioning Real Utopias." But now it's just like, you need to make these things positive for conservatives.

Emily M. Bender:

Yeah. Ugh.

Nicole Holliday:

Sorry, we, so I have, um, sorry. I've been applying for a grant with Jennifer Smith who studies this sort of stuff at Glasgow looking at automatic speech error, uh, ASR rates, um, for Scots and African American English. And we had to apply for a grant that's a joint UK research institution in the NEH. And I was like, how do we get around saying algorithmic bias? Because bias is, and we mean algorithmic bias. It's one of the words that's banned by the administration in grant applications now. And so I don't know how we talk about this ASR error rate difference.'Cause I can't say 'dialect diversity' and I can't say 'algorithmic bias'. And so this is just like an even deeper like shot at the bow that you're not allowed to, because you're not allowed to talk about it, you can't study it.

Emily M. Bender:

Yeah, oof. Um, and we have this, uh, in the, in the chat, um, uh, Thanks It Has Pockets says, "Bias doesn't exist if we don't define it, right?" Sarcasm, obviously. Um, which, yeah. Um.

Alex Hanna:

I just realized that the username is Thanks It Has Pockets.

Emily M. Bender:

Yeah.

Alex Hanna:

A+ username. Incredible.

Emily M. Bender:

'Thanks, it has pockets.' Um, alright. And then back on the point about active Alexas and disclosing that, um, SJayLett says, "Someone on Bluesky said, just say 'Alexa, fart' when entering someone's home."

Alex Hanna:

That's that, that'll be a problem if the homeowner is named Alexa. As an, as an Alex.

Emily M. Bender:

Yes. Yes. Well, presum, presumably this would be knowable um, when you, yeah. Um. Okay, uh, very quickly, here's some more nonsense from Kevin Roose. Um, in, uh, the New York Times, of course, March 14th. Um, the sticker is "The Shift", the headline is, "Powerful AI is coming. We are not ready." And then we have subhead, "Three arguments for taking progress towards artificial general intelligence or AGI more seriously, whether you're an optimist or a pessimist." And it's just complete nonsense, magical thinking. He's like, I live in San Francisco and I'm surprised that all the, all the tech bros here think that they're really building AGI, so gotta believe them because they would know.

Nicole Holliday:

You don't know! They have no idea what they're doing.

Emily M. Bender:

No.

Alex Hanna:

Yeah, I'm, and I'm also--

Nicole Holliday:

I too am in San Francisco.

Alex Hanna:

Yeah, I know. I, I am also there and I'm like, uh, I just, I avoid-- also the URL for this is, uh, "why I'm feeling the AGI dot html," I don't know if you saw this.

Emily M. Bender:

Missed that, missed that.

Alex Hanna:

So that might've been the original title of this. Um, and then, and then Kevin Roo-- so, you know, completely, they're taking a phrase from Ilya Sutskever, formerly at OpenAI, now at his own little superintelligence startup.

Emily M. Bender:

This is, you know, he's basically been practicing access journalism, got served up all the bullshit and just ate it.

Alex Hanna:

Mm-hmm.

Emily M. Bender:

Like this. So, okay. I gotta do the next one.'cause this is here at the University of Washington. Um, we have a actually quite lovely research center called the eScience Institute um, that's older than the AI craze, but unfortunately seems to have decided that they had to jump onto it. Um, so this is an email they sent around to us. Um, so the, their, their tagline is "Advancing data intensive discovery in all fields." That's the purpose of the eScience Institute. And they are advertising an "AI and science postdoc workshop with RAG Copilot for scientific software", um, as the title. RAG is Retrieval Augmented Generation and Copilot is presumably Microsoft Copilot. And the first sentence here is"Generative AI systems built upon large language models show great promise as tools that enable people to access information through natural conversation." No, they don't. No they don't. No, they don't. I've written papers on this and I am so upset that my university is sending this around.

Alex Hanna:

Hmm.

Emily M. Bender:

So.

Alex Hanna:

Yeah.

Nicole Holliday:

My university too, by the way. Um, I looked up a, um, AGI that's supposed to be great for academic research. It's called Liner, I think. And I was like, what is this being advertised to me? I looked it up, "partnership with UC-Berkeley," and I was like, good Lord, stop. No, please don't.

Emily M. Bender:

Ahhh. Um, all right, so this last one is kind of good news. Alex, you wanna do the honors?

Alex Hanna:

Sure. Is this from Arts Technica? The sticker is "Freedom to Learn," uh, and the, and the title is "Open AI declares ai race quote 'over' if training on copyrighted works isn't fair use." Um, and then the subhead, which is very funny, is "National security hinges on unfettered access to AI training data, OpenAI says." So going to the national security angle here. Uh, the journalist is Ashley, uh, Bellinger, uh, published March, uh, 13th. So, yeah, so "OpenAI is hoping that Donald Trump's AI action plan, due out this July, will settle copyright debates by declaring AI training fair use, paving the way for AI companies' unfettered access training data that OpenAI claims is critical to defeat China in the AI race." And this is so funny. And first off, okay, the, okay, an AI action plan is not gonna determine whether a court says it's fair use or not. So I don't know what kind of like entree you're trying to do there. Um, and then, and then I love how now the, the angle is, is economic competition in the China angle, because they knew that the economic productivity angle isn't really cutting out. Isn't really working for them.

Emily M. Bender:

And I'm laughing here that in the chat, Elizabeth With A Z says exactly what Christie, our producer said when she shared this link with us in our group chat."Oh no, don't threaten us with a good time."

Nicole Holliday:

Also like, sorry, anybody that's ever written anything, your time, effort, effort, compensation is worth nothing because national security.

Alex Hanna:

Yeah. And there's already some good, some good news on this. I think Thompson Reuters won a case against, uh, an organization called I think Ross Communication, in which the judge basically threw out the fair use defense. And so we got many, uh, such course, uh, cases still working their ways through the courts.

Emily M. Bender:

Yeah. So let's see if the courts can, can hold. Okay. Um, I need to get to my correct window here because we are at time. Um, and, uh, that's it for this week. Nicole. It has been such a pleasure. Nicole Holliday is an associate professor of linguistics at the University of California Berkeley. Thank you so much.

Nicole Holliday:

Thank you. It was so fun to talk with you all.

Alex Hanna:

It was such a pleasure. Thank you. Thank you so much. Our theme song is by Toby Menon, graphic design by Naomi Pleasure-Park. Production by Christie Taylor. And thanks as always to the Distributed AI Research Institute. If you'd like this show, you can support us in so many ways. Rate and review us on Apple Podcast and Spotify. Pre-order "The AI Con" at TheCon.AI, or wherever you get your fine books. Subscribe to the Mystery AI Hype Theater 3000 newsletter on ButtonDown or donate to DAIR at DAIR-Institute, oof, DAIR-Institute.org. That's D-A-- I'm speaking so fast right now, and it's on topic.

Nicole Holliday:

I'm not judging.

Alex Hanna:

That's D-A-I-R hyphen Institute dot O-R-G..

Emily M. Bender:

Find all our past episodes on Peertube, and wherever you get your podcasts. You can watch and comment on the show while it's happening live on our Twitch stream. That's Twitch.TV/DAIR_Institute. Again, that's D-A-I-R underscore Institute. I'm Emily M. Bender.

Alex Hanna:

And I'm Alex Hanna. Stay out of AI Hell y'all.

People on this episode

Head Shot

Alex Hanna

Co-host
Head Shot

Emily M. Bender

Co-host