Week in Edtech 5/1/2024: TIME EdTech 100, Chegg & Coursera Stocks Fall, Microsoft's Phi-3, Anthropic's New Mobile App, Penn Engineering's Master’s Degree In AI and More! Plus Special Guest, Louisa Rosenheck of Kahoot!

Join Alex Sarlin and guest host, Matthew Tower,  Founder of etch.club as they explore pivotal topics in this Week in Edtech episode:

🌟 The TIME Edtech 100: Showcasing the leaders shaping the future of education
🧠 Microsoft's groundbreaking release of Phi-3, the smallest AI model to date
📱 The exciting new mobile app from Anthropic
💻 Penn Engineering's pioneering online Master’s Degree in AI - a first in the Ivy League
📉 The turbulent times at Chegg following their latest earnings report and leadership changes, alongside Coursera's financial updates

Plus special guest, Louisa Rosenheck, Director of Learning Design at Kahoot!

Don’t forget to subscribe to Edtech Insiders for more updates and insights from the forefront of educational technology!

Recommended Resource:
Designing for Inclusion, Designing for All whitepaper by Kahoot!

Transcript

Alexander Sarlin: 0:04

Welcome to Season Eight of Edtech Insiders where we speak to educators, founders, investors, thought leaders and the industry experts who are shaping the global education technology industry. Every week, we bring you the week in edtech. important updates from the EdTech field, including news about core technologies and issues we know will influence the sector like artificial intelligence, extended reality, education, politics, and more. We also conduct in depth interviews with a wide variety of EdTech thought leaders, and bring you insights and conversations from ed tech conferences all around the world. Remember to subscribe, follow and tell your ed tech friends about the podcast and to check out the Edtech Insiders substack newsletter. Thanks for being part of the Edtech Insiders community enjoy the show. Welcome to This Week in Edtech. It's the week of May 3 2024. We're here that, you know late spring is sort of at the end of the college semester that college campuses are up in arms, and all sorts of things are happening the FAFSA is still you know, just getting back on. And school is almost out for all our K 12 students, we have a great special guest host today, the one and only Matt Tower.

Matthew Tower: 1:27

Hey, Alex, thank you for having me. Yeah, I think it's fair to say the students are restless,

Alexander Sarlin: 1:32

Students are restless, they're, they're voting with their feet, so to speak, even if they're doing it on campus. So we have a few really big stories today, any of them could probably take up this whole show. But we're going to do like a little bit of a around the world on four really interesting stories out there, to our AI based and to our really specific and tech base. Let's let's go back and forth on this this time. So let's kick off with one that I thought was really interesting from the AI world, which is basically there's been this debate about how companies who have lots of intellectual property should be dealing with the LLM companies, which are sort of trying to scrape them or are sort of have an insatiable for data. And this week, we saw open AI make a Financial Times licensing deal with the British newspaper, The Financial Times, you know, very storied journal, and also get sued by some American newspapers because they've been using their material without fully getting it licensed, or it's sort of their that's the claim of the newspapers who places like The Daily News and the Chicago Tribune. So this is, you know, not an ed tech story directly. But it is a story about how data gets into LLM, which is going to undergird so much of the tech world. What do you make of that? I know you think a lot about this kind of data, data sharing agreement kind of thing. Yeah. So

Matthew Tower: 2:54

I think there's been a interesting evolution over the past call it even like three years in this space where, you know, GPT, one and GPT, two, and even GPT, three, open AI kind of had carte blanche to take stuff from the internet and put it into its model. And it wasn't a big deal. You know, chat GPT was sort of, you know, it worked fine through GPT. Three, then that it really blew the doors off everybody with GPT four. And so all of a sudden, you had these major publishers saying, Well, how did they train this model? And, you know, either through the grapevine or you know, who knows how it became clear that they trained it on a lot of not necessarily all proprietary content, but anything they could grab on the internet, and have debatable means it's not worth getting into how they got it, but they got it right. And so now all the publishers are saying, well, you know, we would like to be compensated for our efforts to put that information on the internet. And open API's position has historically been, you know, cool, we'll just take the rest of the publicly available information. And so it's there's this catch 22, where the proprietary content providers, whether that's a news publisher, or an education publisher, and the education publishers actually think is maybe a little bit more interesting, because it's more canonical knowledge versus present day knowledge, believe that there's a certain value to their intellectual property. And the BATNA the best alternative that open AI posits is publicly available knowledge. And the delta between that those two is not huge. But it's just enough for a large language model to go from 80%. Good to 100%. So open AI and the rest of the large language model providers want to bridge that gap. And actually, most of the content providers do too, because right now, they can't really build on top of the large language models. Because of the hallucinations. The catch 22 is as soon as they give their data over, they can no longer offer a proprietary product. Offering on top of it. So the value to them of the proprietary product is higher than what open AI and their large language model providers will ever be willing to pay. So they're like, Well, should we hold out because there's business value and a proprietary product, or just take, you know, the lower alternative of getting some revenue by providing our content. And we've seen a couple of deals that read a deal with Google, kind of at the pricing model at 60 million bucks a year for all the information on Reddit. And now we're starting to see more publishers total wide, you know, the Financial Times I don't think released the price of what they're getting paid. But you're seeing more people sort of just say, Well, I guess we'll take whatever that is or not. Yeah. And

Alexander Sarlin: 5:42

you can sort of imagine the internal conversation if you're, you know, the Washington Post, for example. And you say, well, we could create a chatbot, that is only Washington post content, you can ask the questions, it'll cite everything it does. And it's, it's the whole, you know, Archives of Washington Post, you'd say, well, that's a pretty cool product. But what people really want to use that versus, you know, versus GPT, or Gemini that have, you know, all the information from everyone, hard to say, right, you're going up against big, big, giant product, so then the decision maybe becomes a little bit, there's less fear of missing out, I think, if you start to really go down that path and say, Will every different newspaper or places like Reddit, you know, have their own product on top of their own data, it doesn't fully make sense, even though they have huge amounts of data. I don't think it fully makes sense, because we've already seen these generalized large language models that are trained on absolutely everything. And it's hard to compete with that. And they're getting better all the time. So and they'll probably they probably be built on them anyways, just like this totally circular, they probably use the open AI API's to do it. So I think it's a really, really interesting point. And I'm glad you brought up the educational content publishers too, because, like you said, canonical, I mean, Reddit content. I remember we talked about the Reddit deal when it came out a little bit. It's like, Reddit content is fascinating. And I get why open AI would or Google would want to have the Reddit content, because it's user generated. It's new, it's topical, there's a lot of humor in there. There's a lot of I mean, there's everything in there, right? Even if it's not all factual, it's very, very highly opinionated, very highly user centered. But the educational content publishers have the opposite, right? They have the history, all the chemistry textbooks for the last 50 years are all the you know, world history prompts for every you know, AP that's ever happened. Like there's some really, really specific and interesting information that is in there. I remember when that first article came out showing how GPT four was trained, I guess, probably GPT, three was trained at the time. Coursera was up there, I think it was like within the top 10 of the sites that had given the most tokens to the model. And I knew why because Coursera had makes their sort of strategy is to make all of their content, scraped, not scrape bubble, but visible on the internet in separate pages lead up basically every lecture on Coursera has its own dedicated page, it's hard to find, but it's there for SEO. So it's perfect for open AI, they can scrape everything. And I remember being like, ah Coursera and other big publishers, you know, people with a huge amounts of content, really, maybe missed the boat here, because they they could have had that $60 million a year deal or some million dollar a year deal. And instead, everything is sort of already out there. So we'll see where that all ends. I

Matthew Tower: 8:27

think like, what is the value to Chegg have a proprietary, you know, LLM, right? Arguably, it's like several billion dollars, you know, in 2021, are at the peak of the EdTech market. So it's not a perfect analogy. You know, they were worth what, four or five $6 billion. And today, they're worth about a billion, largely because the market ascribes a threat to them from chat GBT. And so like, you know, if they had their own proprietary thing, they might still be close to that peak valuation. And yet for them to build their own LLM would have cost hundreds of millions of dollars. And that's not funding that they had available to them. That's a whole different company. It is literally a whole different company. And so they ended up being market takers, right? They, they it's like, yeah, it could be worth that amount in new business to them. But maybe they should have just said, Well, we're just gonna license our data to open AI for, you know, X 10s of millions of dollars, right. I think we're seeing the results of that and the public market right now very

Alexander Sarlin: 9:30

much so. And I mean, it was no coincidence, right? The Chegg stock dropped like a rock when the CEO said in an earnings report, just just sort of cat mentioned almost basically, in passing that there was like a slight, you know, dip in usage because of these new models. And I think it just completely spooked the entire investor community. We're mentioning Chegg And Coursera. We should probably mention that you know, this. One of the other big stories this week is that both of those companies saw majorly Leadership changes this week in conjunction with lower earnings predictions and, you know, earnings reports that sort of had lower outlooks and weaker guidance, as I say, than projected, and saw some turnover. And at the very highest level, we saw check ceo dan Rosensweig leave after he was, you know, that the company the entire time, and the Chief Revenue Officer, Lea Belsky and the chief operating officer Shavon, goalie from Coursera, both sort of leaving the company in conjunction with these wacky moment, scary times in conjunction, I think of this as a, you know, the bad things come in threes and think of the to you, you know, stock crash and Chip Pasek being shown the door, you know, a month ago or so, but you may not see those as all connected, but I see it as this sort of three things. And these are three of the not very many public ed tech companies out there. What do you make of it, Matt? This is this was pretty seismic news in the public edtech community.

Matthew Tower: 10:56

Yeah, I mean, I think what feels sort of like, unfair to me, as like a both a student of the market and somebody who's, you know, pretty personally, both financially and just like, emotionally invested in the tech market is like, what do the public markets want? I don't think they have an idea of what they want. You know, it's been pretty clear for years that Nathan Schultz was going to be the next CEO of trag. You know, he's been promoted multiple times, right? And call it the past five years, he's been with Dan, on the executive team for, you know, more than a decade, I was sort of hopeful that the announcement of Dan stepping down and Nathan taking over would provide some sort of a buoy to the company, it's like, okay, you know, returning over a new leaf, obviously, it sucks. Whenever somebody has to step down, you know, maybe get fired, you would have thought that the kind of most vested interests from the public markets would have made it clear that Nathan, they weren't in favor of Nathan, there are ways to do that, you know, through investor days through, you know, activist investors, etc. And yet, I didn't see any of that. So I think, what is alarming, frustrating, I don't know what the right word for it is, is like, I don't think the public markets know what they want from the ad tech space. You know, they know they're unhappy, as we can see from the stock prices, but I don't think they know what they want, you know, Coursera, same thing, like, the market was not happy that they, they got rid of two key senior executives, and they just sent the stock down. So that's, I think, what's tough to me, I don't know if that tracks with what you're seeing, I

Alexander Sarlin: 12:34

see it as there's been this sort of shaken the foundation of the concept. med tech, literally, I mean, I feel like there was a period not too long ago, and definitely sort of at its height in the middle of the pandemic, but I would say it was going throughout the, you know, the late 2010s, that education, technology, a field that had always been, you know, below the radar for many, many people and sort of considered a little bit of a backwater from the business community, there were starting to be some really big names in it. And some companies that were really breaking out some models that were getting a lot of attention. And I think the investor community writ large, I know that's not you know, they're not one body Exactly. But sometimes they act like one body, you know, I think they started to see ad tech as sort of a viable, new technology that was actually writing some of the waves of new technology, cloud computing, mobile, and of course, now AI. And I think the narrative got a little distorted, at least from my perspective, and I think yours as well, in this AI era, and in this post pandemic era, because some of the growth that they saw during the pandemic era slowed down, you know, pretty quickly after the pandemic, as as this education system sort of began to snap back in, in some ways. And probably more importantly, I think the the perception started to be that, you know, AI is the hot new thing. That's the new technology that AI is going to eat, maybe search AI is going to eat, maybe you know, a lot of you know FinTech in some ways, like it's coming to do a lot of changes to a lot of industries. And I think that feeling about edtech is AI is going to actually like bite out of edtech it's not going to raise Ed Tech's profile, it's not going to make more edtech superstar unicorn public companies, it's going to actually hurt the entire sector. And, and then, you know, when they see these quarterly earnings that are lower than expected, I think it's just the alarm bell of like that suspicion that the investor community had suddenly seems confirmed. And so that's why it feels like the stock price is just shoot down. I mean, some of these drops are some of the largest drops in history, literally, like the two years drop in the stock market is one of the largest drops in the history of the stock market. They were at $90 and they're trading at under FAR under $1. Now, I mean, this is a not like a small change. This is these are Huge, I think it's basically, for lack of a better metaphor sort of rats running from the ship feeling where they feel like the tech companies rather than, you know, taking advantage and being sort of parallel and growing with this AI technology are actually going to get, you know, drowned by it, and they want to flee. And I don't think that's fair at all. And then to your to your point, yes. I don't know what any of these companies could do to actually restore confidence right now, other than have just ridiculous earnings. I mean, right, that's the only it's outperform would be the only possible thing they can do that would convince investors that maybe the ship is getting righted. That's my read. Yeah.

Matthew Tower: 15:40

Well, and I think basically, the only a tech company that is performing well in the market is, of course, you know, they operate outside the confines of the traditional education system. And so I think the only thing that comes to mind for me is, is for the public markets, you sort of have to say, we are, you know, quote, unquote, disruptive. But basically, that means you're not constrained by most of the regulatory burden. So, you know, I think, the more kind of aligned with the traditional system you are, it seems like, and I don't, I don't know if I agree with this or not, I have to think about this more, but seems like the more the public markets are going to knock you and kind of say, like, just hard to grow, the way that the market demands, while also playing nicely with the current regulatory regimes and regulatory is broad words, right? So it's the same way like, the university is going to have a tough time buying from you if you say you're going to disrupt the university, right? So it's, you kind of have to pick one and roll with it. So it's a really tough time out there. It

Alexander Sarlin: 16:46

really is. And I think, you know, my recent sort of vibe, has been that the EdTech community writ large, I mean, we've seen some of these companies, exactly these companies Coursera to you at Chegg be in, you know, fierce competition with each other for various types of, of work for the OPM market, for example, for the some of the sort of online textbook kind of material, or online course material, you could say, but I think we're kind of in a moment where we, as a community have to sort of justify our existence, we have to really convince, I think the rest of the world, that there is viability in even products that do rely on as you say that, you know, traditional education system, whether that's k 12, or higher ed, it feels like the narrative is taken over that like, you know, this isn't really tech in the way that we've all wanted it to be. It's more education, and education, gets Education Service education services. Exactly.

Matthew Tower: 17:42

And I will say, to try and be a little bit more bullish on this one potential, salvation feels a little heavy of a term. But one potential positive avenue for these companies is the emergence of smaller language models, and the on device, ilk language models, where users are going to have a more personalized implementation of these language models. And so I know one of the topics you wanted to talk about today was Microsoft's release of I don't know if they classify it as small but an on device language model, if you want to set some context there. We talked about how a tech companies might be able to leverage

Alexander Sarlin: 18:20

those, right? Yeah, you're right on. I mean, Microsoft launched something called fi three. It's an open LLM, importantly, an open LLM that is designed specifically to fit on smartphones with limited resources. It's low latency, and it's meant to be actually, you know, work on a phone without needing to do basically just quickly and powerfully, and live on phone. And then we also saw anthropic this week launch their mobile app for their cloud, you know, aim these sort of business services as well. So, you know, it hasn't been that long. I always like to remind myself and everyone, but we're starting to see the MLMs follow us into our daily lives in a lot of different ways. And this feels like a jump for mobile. So what Yeah, what do you think it means red tech?

Matthew Tower: 19:06

I think it could be positive. And, you know, it was a week or two ago, also, Sam Altman said, large language models are dead. It's now about small stuff. And, you know, I think, my hope, and, you know, we'll see if this is actually the case is that the smaller and on device models will allow, basically, they'll do that personalization part for other applications, and allow the applications to tie in nicely with them. So, you know, the language model on your phone will know about you and have your like, characteristics that, you know, today you would be having to put into the context window when you do your prompting, I hope. Again, this is sort of the start of a steel man, not a perfect argument, that applications so a Chegg or a to you or Coursera, or, you know, startups will be able to leverage those on device characteristics to build a Education applications and you know, every other type of application, right. So hopefully this applies across the spectrum of on phone apps. But that should allow a much higher fidelity experience, both with a language model and with the actual educational application, while still allowing for other applications to thrive. You know, I'm not saying it's all gonna go to these small on device, language models, they're not going to handle every application use case, but they will provide wind in the sails of other applications.

Alexander Sarlin: 20:30

I mean, a non education example that jumps to mind for me, because I was just discussing this, with my wife this week is that, you know, at this point, we all have so many photos that we've just don't even know what to do with years and years and years, especially people like you and I who have young children have ridiculous numbers of photos on our devices, and we carry them around with us, some of them are in the cloud, whatnot, you can pretty easily imagine a very useful use case for LLM and you know, AI in general, but especially that sort of personalized LM where you say, I want a cute picture of my kid to send to grandma, and it goes, Okay, I know what you mean, by cute. I know all your pictures, you know, I know all the pictures on the cloud, I know what you've done in the past, I can sort of make sense of that using generative AI, and you know, just sort of data mining, and I can pull you pull out the photo you actually want or you could even say, Hey, I have too many photos, can you suggest, you know, 500 that I could probably delete and it will go Yeah, I can. And I know what that means for you. I know the kind of photos you don't want. And of course, we can then transfer that to the education use case. And there's a lot of different interesting ways to look at it, including just curiosity of knowing what to search or being able to sort of recommend based on where you are or time of day or behavior in apps, you know, things that you might want to learn or that you might be missing in your understanding of what you're doing at any given moment. There's some really, really interesting educational use cases that sort of could put the phone which is in our pockets all the time into our educational lives in a different way than it has a it's hard to imagine right now, I think, but you know, it's getting there.

Matthew Tower: 22:09

Well, and just to extend the analogy, like imagine a world where, you know, in WhatsApp, and you know, I know WhatsApp and Apple are both big tech companies. So we'll see if the analogy works. But in WhatsApp, you can ask your on device bot to you know, send a photo to Grandma once a week. Right? And, you know, have that be part of your ongoing conversation with that person, right. And for education, you know, there's been so many articles about like, do phones belong in the classroom? Can you bring your phone to school? Yadda yadda yadda? You know, you can start to imagine a world where you're on device, Assistant by whatever you want to call it is capable of discerning like, oh, yeah, Matt's at school, probably he should only be able to access these apps. And so the guardrails become even more intelligent, right? And the schools have a say the parents have a say, like, those types of intelligent interactions that like you and I can come up with on a piece of paper, but could never implement on a device today. I think we're, you know, not too far away from being able to

Alexander Sarlin: 23:11

the interactivity between apps between the apps and your regular daily life. There's a lot of potential here. You know, I can imagine your sounds a little scary, but you know, you're on a phone call, and you say, oh, you know, I wonder what the weather's gonna be like, when we go travel to Tucson next week. And and it goes, I got it. You're going to Tucson next week, you're wondering what the weather was going to be like, I'm going to figure out, you know, everything you need to know about this schedule accordingly. Maybe I'll help change your, your hotel plans or whatever you make that restaurant reservation indoors, like, there's some really interesting ways that sort of everything can work together, if you have a powerful AI inside your phone inside your pocket. There's one more big story that we got to talk about here. And it is a doozy. And I don't I know, we don't have a huge amount of time. So let's do sort of that somewhat lightning version of this. But the Time magazine came out with its first ever, top 100 edtech companies list last week. And Ben and I mentioned it briefly last week, because it is sort of been a little bit of an object of ridicule in edtech circles. But you as somebody who really follows the stuff very, very closely in terms of the ratings, who I think thinks a lot about systemic reviews of industry, I'm sure have a lot of thoughts. This was such a bizarre list. And I think we're all wrestling with what they're even trying to accomplish with a list like this. But I would love to hear your thoughts about the time when edtech 100. What did you make of it?

Matthew Tower: 24:41

Yeah, so I think I'll start off by saying, the vast majority of the companies on the list I admire individually. So like, you know, it's not necessarily quibbling with the quality of the companies on the list. It is the principle of trying to rank them that I find sort of laughable, right? So it's like to give an example. It's like, how are you going to compare a company like Emeritus, which holds the top spot and a company like Duolingo? Like they do fundamentally different things. Like there's a tiny, tiny, tiny bit of overlap, and maybe in like learning business English. But like other than that, I don't think there's any overlap and who they're selling to. So it's like, as a result, you question the methodology just on principle, right? And then you start to say, Well, what went into deciding this? Like, did they talk to the companies that they might, who was involved, it's just like, it leads you down this road. And, you know, I'm not a fan of university rankings either. But at least they're selling largely to the same student population on the selective end of the spectrum. But I just like, I don't know how you compare, you know, a K 12 infrastructure company, to a executive education company, just like that doesn't compute. For me, it's like, sort of a fun, like bar exercise. Like, I love to cook, you know, and say, like, Who do you like better? Corsaro Duolingo? And, like, hash it out together? Because it's like, it's funny. But I don't know how you can do that in an intellectually rigorous way. Yeah.

Alexander Sarlin: 26:11

And they talk about what they tried to do. So I love that take, it's sort of like the idea of trying to rank these companies against each other, when they do wildly different things in totally different spaces is pretty ridiculous. I completely agree. I also think the results are pretty ridiculous. I mean, the way they've measured this as they have these sort of two pillars, this impact and financial viability, they call it financial strength and industry impact. And it's basically 70% financial strength and 30%. Impact. This is all done by statistic. And I mean, this, it's just so difficult when you start to read this list to even begin to understand how they're thinking about it. I mean, you see companies like Udacity, getting very high spots on this relatively, but Udacity was just bought for pretty small, some not that long ago, and it's number 53. On here above things like Kahoot Duolingo, is not even on this list. It's not even on the list. Actually,

Matthew Tower: 27:08

it is that Duolingo English test specifically is on where is it? What Duolingo the company is not it's buried. I mean, and this is, again, bizarre. I don't have a quibble with like most of the companies on this, but it's like, you don't even get the company, right? How am I supposed to trust the rest?

Alexander Sarlin: 27:27

I mean, Instructure is here at number 216 dominates the market. I don't even see Duolingo on here, I'm going to the very bottom. But yeah, I mean, this is just so bizarre, it almost makes me like I mean, it makes me a little embarrassed for a time. But it also makes me a little nervous about, again, about the EdTech field, and people are using this, to try to understand what's going on this is global, the global attack, they are getting a very, very bizarre, very bizarre take on what this field looks like. And, you know, I mean, I think one way to look at it. And again, and I agree with you look, the companies on this list deserve praise. They're amazing companies. And I'm not trying to put anybody down by saying, you know, they shouldn't be at a high mark here. But the top three companies they list in the United States are yellow brick, which is you know, a sort of internship, they do partnerships with universities to do work based sort of online experiences. It's, um, it's a very cool company. But it's not that big Stryver Labs, which does VR education, and civitatis, which is sort of student success. I think if I were to, you know, just start listing ad tech companies that are, you know, major US ad tech companies in the space, none of these would be in my top 40. Like, not even close. And I mean, they're great companies, no offense to anybody listening who's works at it, and they do great things. But this is not the top three US ad tech companies by anybody's imagination. Yeah,

Matthew Tower: 28:56

again, it's I don't understand. Also VR, which is like a surgery VR, like Series B, or Series C trainer is number 200. And I'm like, that's such a funny place for it to be versus like Breck Dara, which is a, you know, seed stage work based learning platform in Australia. Again, like, I think both companies are very interesting, but like, practice areas 182, ASO's 200 like, well, I don't know, it's hard for me to assess it.

Alexander Sarlin: 29:27

It's just bizarre. I mean, I agree with you that probably this whole enterprise is a pretty ridiculous thing to do. It would make way more sense to just list companies, you know, like by number of users or by revenue or by stock price, or by for the public ones, or by I mean, not that any of them tells the whole story, but somehow this just feels absolutely nuts. I'll stop ranting about it. But it just is the most bizarre. I mean, many of the companies find it now. It's hard to even get your mind around and again congratulate I mean, the companies should absolutely still be proud to be on this list and you Do you see companies up there like AIDS learning and Quizlet and class, fantastic companies that are making, you know, the top 25 They should be proud of, of it, I think emeritus has a good claim to be the top edtech company in the world, they've done incredibly well, in a lot of different ways. But then, you know, then it just gets bonkers. You know, it's hard to compare these companies, but it's also like, Man, I mean, they talk about the sort of intellectual property or whatever they call it within the financial strength, they say it's the impact measure is the quality and impact of their product and service portfolio and the quantity and value of a company's IP portfolio. I mean, those are really hard things to measure. Well,

Matthew Tower: 30:42

and statistic is estimating this, I think, I assume, with a reasonable degree of confidence, I'm not 100% sure that they did not talk to each of these companies to gather this data. This is statistics, scraping the internet for what they can find, and then using that to deduce and I just think it's just not rigorous. Like, I think they probably ended up with a lot of false information, which, I don't know, I wouldn't publish this list. No,

Alexander Sarlin: 31:08

me neither. I think it does a very, very bad job of representing this field. And I mean, at the same time, I'm happy that time it cares about ITT Tech. I mean, yeah, that's great. But this just didn't feel like the right approach. I mean, you know, sometimes you see these articles that are like, let's ask three top VCs, what are the best edtech companies and they obviously name things that are happened to be in their portfolios, that's not the best way to get it either. But this just did not work. And so I don't know, I don't think we need a ranking, I'm not gonna even recommend ways that they could do better rankings, because I don't think we need rankings in ad tech. I don't think that could get better if it's anybody

Matthew Tower: 31:44

Oh, like, this is why market maps are sort of the least offensive way to show a bunch of companies because it's like, you're not saying anyone's better than anybody else, you're just saying they participate in the space, they have their own flaws, there's yada, yada, yada, but at least you're not trying to compare apples and oranges is sort of my feeling about market maps.

Alexander Sarlin: 32:04

Agreed. And this would have been a much more, I think it'd be much more useful as a market map, because then you can actually see which of these companies do what, because when you see them side by side companies, like you know, I won't even name anymore, but just companies that have literally nothing in common are not in the same space or not for the same age users. They are companies in here that our LMS is there companies that create courses, there's companies that deliver courses, as companies that work with higher ed, there's companies that do publishing, I mean, they're just everywhere, there's boot camps, there's startups, it's just,

Matthew Tower: 32:37

I mean, tal education, which, you know, is basically banned from doing edtech. And as now they sell, basically like steaks online in China. And that's actually not like, they have an English language component, because they sell everything in English, but literally, like they're an E commerce shop, they're number 14 on this list. So it's, yeah, I'm sort of at a loss. It's a thing. So

Alexander Sarlin: 33:01

I mean, I do recommend just for the sort of sheer awe of it, that anybody listening to this, go check out this list just to see what you think, again, as always, if your company on it, be proud, somebody found something amazing for what you're doing. I'm not trying to, but wow, it's worth looking at and then worth clicking away from and never looking at it again. Because it is ridiculous. I think we need to do better as a field in letting the rest of the world and the mainstream media, so to speak, you know, understand our space better than us. I think that's all the time we have. I think it's time to move to our guest for the day. What do you think, Matt, any last thoughts about any of the topics we talked about before we move to Louisa?

Matthew Tower: 33:44

No, I really appreciate you having me. And, you know, I think it was a fun week, you know, with public earnings and AI and, you know, a rankings list that will hopefully never talk about.

Alexander Sarlin: 33:53

Exactly. Ai continues to take off and innovate. Ed Tech continues to innovate and be totally undervalued for their innovation and be punished over and over again, in every possible financial way. It'll get better. Thanks so much for being here. Louisa rosenheck, the director of learning design at Kahoot, which has a shockingly low number on this list is coming up next. And she's absolutely amazing and just published an incredible paper about inclusive design. So here we go. For our deep dive this week, we have a very special guest, Louisa Rosenheck, who is the director of learning design at Kahoot!, one of the largest and most successful edtech platforms in the world, Louisa. Welcome to Tech insiders.

Louisa Rosenheck: 34:40

Thanks so great to be here again.

Alexander Sarlin: 34:41

It's so great to see her again. You know, we talked a couple of years ago when you're first starting at Kahoot. You had been at MIT, and Harvard doing all sorts of amazing work about games and game design and learning. And one of the projects that you just came out with it's super interesting is this paper about inclusive design for education technology and for educational games, tell us a little bit about what you've been doing a Kahoot. And what inclusive design is.

Louisa Rosenheck: 35:08

Sure. So when I joined Kahoot, one of the things I hoped to do was kind of expand how we think about playful learning, and what types of learning can happen on the platform. And you know, who has always had as one of its values inclusion. So this, this project was a chance to really put those things together. And I think the main point from both the new tool that we've developed and the white paper on inclusive design methods, I think the big takeaway that I'll start with is that when you design for inclusion, it doesn't mean creating a separate version of the tool for certain kinds of learners. It means, you know, when you use universal design for learning, it means you design something that is flexible and customizable, that everyone can see themselves and everyone can engage with. And it ends up meaning better learning for everyone. You

Alexander Sarlin: 35:58

mentioned five techniques in this paper that you've used, and that are sort of core to the inclusive design principle to be able to reach this noble and tell us a little bit about the five techniques and why you highlight them. Yeah,

Louisa Rosenheck: 36:10

so there are many ways to design inclusively, of course, and within each of these methods, there are many different ways to execute them. But the white paper focuses on designing for neurodiverse classrooms. But these methods, of course, can be used to center any kind of any students any voice that may be marginalized, and that you want to make sure to get their perspectives. So the methods that we used in our development of code, sparks are detailed here. And so really briefly, here's what they are. One of them is CO design. So I'm very happy recently to see a lot more discussion and conversation about poor design, there was a number of sessions at South by Southwest edu. So people are starting to adopt this code design really means including your learners in the process before you even have a concept of exactly what your product is. So we're playtesting is okay, I've created something and I want your feedback on it, who design is here our goal, here's what we want to do. But we want to position learners as designers to you know, give their input from the very beginning. So co design can often look like a group, a cohort, doing different design activities, hopefully across multiple sessions, really getting building a relationship with the researchers with it, or the UI UX designers and learning designers to work together to come up with what is going to be the best type of learning experience. Another method is empathy interviews, this is you know, it's not something new. But when you think about empathy, interviews for inclusion, you just really want to make sure that who you're talking to are the people who don't often get a say, and really try to understand their experience. So in our case, we did empathy interviews with some neurodivergent students, as well as a lot of sped teachers. So we got, you know, a couple of different perspectives on not only what it's like to teach a neurodiverse classroom, but what works and doesn't work with existing edtech tools so that we could kind of build on what's out there. A few more methods really quickly. Rapid Prototyping is, again, it's a great method in any kind of software design game design. But the idea here is not getting married to any idea too early, and trying them out with neurodivergent students with your target population, so that you don't come in with too many jumps, assumptions. And you can see, okay, which ones resonate, which ones work well, and then which ones do we move forward with? Another really important idea is representation on the team. So for us, that meant having neurodivergent team members internally, there's there's always room to do better in this area. But you know, the people that we worked with, both internally on our team, and as advisors, were so helpful in providing a perspective that the rest of us designers just may not have. And lastly, a self check. I always like to include this as a method. So you kind of you know, when you're designing, you lay out your goals and your principles, but it is very easy to kind of slip away from those at times. So periodically, just coming back and saying, Okay, what were our guiding principles, our North Star? And are we actually hitting those. So for us, that was things like making sure that we had a learning experience that gave students voice and choice and multiple ways to express themselves. And it was something that would result in lots of different student ideas. So as we were play testing and rapid prototyping our kind of Lo Fi prototypes, we would periodically come back to that and say, Okay, are we seeing this? Is this actually happening? And if not, we need to, you know, adjust and change course.

Alexander Sarlin: 39:42

Yeah, they're really great methods. So that's, you know, co design with learners. Conduct empathy, interviews, do rapid prototyping, have representation of the learner population you're trying to address on your team, and that sort of metacognitive piece, you know, stop and self check and make sure you're doing it in the way that you intend. When you set out because you know, as we all know, projects tend to go, the best laid plans right don't don't always come to fruition.

Louisa Rosenheck: 40:06

The big idea with these methods is you know what the end result you want a product and experience that's more inclusive. And why these methods are so important because you're, it's very hard to come up with a product that is inclusive of all learners if your process wasn't so inclusive. So baking, this inclusion, and you know, listening to student voice in the design process is going to result in a more inclusive product for all your customers, users, learners.

Alexander Sarlin: 40:35

100%, in the case study you really focus on here is, as you say, it's for neurodivergent learners, which is 15 to 20%, of the population estimated. So that's a huge group of learners. And you came up with this product called Kahoot. Sparks, which is, you know, sort of a different take on what Kahoot does, then their traditional multiple choice, you know, gamified, multiple choice products that they tend to lead with, tell us about what came out of this process. And what Kahoot Sparks is, yeah,

Louisa Rosenheck: 41:03

so first of all, I'll say that this tool is development of the tool, and the white paper came out because we were part of a Lego foundation accelerator program called play for all, it was a wonderful experience and a great opportunity to develop this, the accelerator was supporting a variety of different companies and organizations to create experiences that were more based on Play, and digital learning and inclusion for Neuro diverse learners. So we also learned a lot from our advisors there, and our fellow cohort members, other companies that were doing this as well. And some of those companies also have case studies in the white paper. So that's really interesting to check out. So for our project, code, Sparks, what we started to think about was, you know, looking at the classic Kahoot experience, it's very engaging, and I mean, teachers and students love it. But it's not always engaging for everyone, right? Teachers know that when you play a Kahoot, it does happen, that there are sometimes students who, you know, the competition is just not motivating for them, or they don't think so fast, and they feel like they'll never be on the podium. So those are students that we really kept in mind from the start the way a typical Kahoot is you have to be correct, and you have to be fast. Now, there are many other ways to use Kahoot. But that is, you know, the most common way that's that's what people will kind of have in their mind. So we wanted to create an experience that was like who's in that, you know, play to its strengths, that social, it's fun and quick and easy to understand and really flexible, that can have any different kind of content subject area, but we wanted something that could provide for an opportunity for more students self expression, you know, really with the idea that with neurodiverse students, which is it's all students, it's all classrooms, students minds work in different ways. So people don't all want to answer the question in the same way, or they don't all have the same idea. But they can all be good ideas. So we ended up creating what sparks which is all about creative ideation, and expressing your ideas. So in the experience, as we've designed it, there is sort of a creative challenge, for example, design a new toy for a five year old. And then before you get to that students submit little inspirations, little sparks to a spark prompt. So it might be something like, give me a word that's playful, or give me a fun shape or an emotion. So students submit those and then all of those ideas get shuffled up and handed back out. And then each participant gets a set of, you know, two or three inspirations that they then use to come up with the new toy idea, or whatever the concept is. So teachers can create their own prompts. But the idea is, there's a lot of user generated content in here, students are coming up with ideas sharing them, they can draw, they can type, whatever they're comfortable with. And there's this idea of collaboration. So they're re mixing each other's ideas, and it really leads to a lot of divergent thinking you get ideas that you might not have had otherwise. And it's open ended. So it has a lot of student voice, but it's also scaffolded. We know that for many students, including a lot of neurodivergent students, it can be overwhelming to be asked to come up with an idea with just this huge open space. So these small ideas sparks are meant to scaffold that and you know, give a little bit more direction. And we've seen teachers start to use this in a lot of different ways, you know, for SEL for as warmups or icebreakers as assessments and for you know, more academic content.

Alexander Sarlin: 44:35

It's a really interesting, I think people who have known traditional Kahoot can imagine how this is just such a different and exciting approach to you know, class wide social activities. It's really about ideation bouncing ideas off one another. It feels almost like a design thinking session, you know, that we do as adults and when we're trying to come up with really interesting new ideas and I wanted to focus on one aspect because it's so interesting to me. We you mentioned that students can submit in different ways they can write, they can draw, I think you mentioned in here, you can even like sing. There are some really interesting ones. And this is sort of core to the UDL principle of, you know, multiple means of representation. Can you tell us a little bit more about that? Because I think that feels like a really new aspect to this kind of work. Right?

Louisa Rosenheck: 45:17

Yes, yes, this is something that is so important. And it's something that it comes from the UDL framework, of course, you know, multiple means of expression, multiple means of representation of ideas. And it's also something that we saw right away in our CO design session. So before we conceived of good sparks, before we knew what the mechanics were going to be, we did these sessions with special ed classrooms. And one thing that we definitely saw was different students were drawn to different forms of expression. So we did you know, non digital activities, giving students different kinds of prompts. And they could draw write act things out saying anything they could think of, and we just really noticed that most students did not pick text writing. And there was a huge variety, really, you know, students want to express themselves in different ways. And they're more comfortable in different modalities. So that was something that we really wanted to build in. And I think we definitely see it, honestly, we see it with when we try sparks in classrooms and with adults, some people type want to write words, some people want to draw, and some people want to do both. That's how they feel that they can best explain their idea and represent their thinking. And so, you know, the big idea of inclusive ed tech and of sparks is that every learner should be able to access the learning, everyone should be able to shine and share their good ideas in their thinking. So having options, being able to choose how you're demonstrating that it's a really simple way, but really important.

Matthew Tower: 46:46

So I really like that all this process work led to the creation of a new product, I think often we see features kind of bolted on at the end, or we see it become a wholly separate thing that, you know, is, oh, this is intended for this audience, or this is intended for that audience. And so I think what's cool about Sparks is that it is for everyone. And I think I'd be curious to hear what the most sort of unexpected and or delightful features of the sparks product that came out of all that process work you did was and to give a little context, one of my favorite features on the iPhone that you know, is baked into the very core of the phone is grayscale mode, which is designed to be an accessibility feature. But for me, you know, I don't need it for accessibility, but I use it to help, you know, reduce my screen time to make the screen a little less interesting. But I love that it is really core to the way that the iPhone operates. I can use any app, you know, in grayscale mode or normal mode. So with that context, I'd love to hear some of the features that came out that were sort of either unexpected, but great or just patently delightful for both neurodivergent learners and or regular learners.

Louisa Rosenheck: 48:01

Sure, yes. So one thing I'll mention is at the end of a Spark Session, so the students have, you know, come up with ideas and drawn or written them, and then they show up, they all show up on the big screen that's being projected. And then, you know, one of the classic elements of Kahoot is the podium. But of course, our whole message with sparks and inclusion is that there's no one best answer, there's no winner, right? So we thought a lot about what do we do with these ideas, and we came up with a kind of sticker exercise. So after all the ideas are there, students can browse the ideas and put emojis on the different ideas, but it's not just a star, or this is my favorite, we've have given them labels and teachers can customize their own label. So it could be something like the most creative or, you know, the most based on science, or this is something I would really use. And we weren't sure how students would react to this, if it would make sense to kind of do this critical thinking and, you know, label them and think about what makes an idea Good. We weren't sure if it would feel like kind of a drag or if it would take a lot of explanation. But it turns out that people love it. It's one of the most fun parts is celebrating all of the ideas. And you know, it's fun to see the confetti flying all over the screen and the little animation, but also students, they really like looking at each other's ideas. And that's something that you don't get in a classic hoop because there's you know, typically one answer or you know, a limited set of answers. And that is the case in a lot of edtech. And to be honest in a lot of traditional schooling. So this is where designing for inclusion, where we know that people think in many different ways and we wanted to really allow for that. But it also just aligns with what we know about good learning in general and what learning science says about constructing ideas and communicating ideas as part of building an understanding and demonstrating that understanding. So the Celebrate phase, you know, it really underscores the learning. And students always really have fun with it.

Alexander Sarlin: 50:05

It's great. And I think, you know, as somebody with a long history in game based learning, I think, you know, the nuances and how tricky it is to do competition in a classroom in a way that actually, you know, benefits everybody. And I think this is a really interesting twist on the sort of classic Kahoot. Like, you're saying, you know, the podium, the very individual competition, and makes it much more of a collaborative, sort of, as you say, celebratory experience, it's a really different take is really, really interesting, right. And

Louisa Rosenheck: 50:33

it's a nice little mental compliment. You know, there's the classic hood. And now the platform can also offer building creativity and these other skills. So it's yeah, the idea is to have, you know, a variety of ways to engage again, thinking about UDL principles,

Alexander Sarlin: 50:49

for sure, definitely. And I mean, you're not going to take away classic Kahoot from any, any teacher or set of students, they are going to buy it. So right, absolutely. That's part of the recipe. So for companies who want to find out more about this process, and more about the product and the processes behind them, where can they go online to find your white paper or more about what your work has been doing?

Louisa Rosenheck: 51:11

Yeah, so we'll link to both the information about the new tool, as well as the white paper. And I think the main message that I really want to get out there to edtech companies and anybody creating learning experiences, is when you think about inclusion, again, like I said, you want a more inclusive product or experience, you have to think about the process, how are you going about including student voice, including, you know, the learners that may be traditionally left out? And in thinking about that process, it doesn't have to be too intimidating. It doesn't have to be like, Oh, I have to do all these methods, or I have to do a huge codesign initiative in order to get it right. No, there are lots of ways that you can start small, just, you know, start with a few empathy interviews here and there and build that into your process into your you know, your typical design process and then start building from there. So it really doesn't have to be a huge barrier for a team. I know a lot of times it feels like oh my gosh, this is gonna take so much more time and resources. And in some ways it does, but it doesn't have to you can start small start simple, start getting used to it. And I think all the teams I know that have started to do more of this. They really enjoy it and they see a lot of value for it. Yeah,

Alexander Sarlin: 52:25

that's a great message. You know, don't be intimidated by this process. Start small start simple is start building it into your processes and you'll see amazing results. Thank you so much, Louisa Rosenheck, Director of learning design at Kahoot!. Thanks for being with us here today at Edtech Insiders Week in Edtech. Thanks for listening to this episode of EdTech Insiders. If you liked the podcast, remember to rate it and share it with others in the edtech community. For those who want even more Edtech Insider subscribe to the free Edtech Insiders newsletter on substack.