Topics
Artificial Intelligence and Business Strategy
In collaboration with
BCG
In collaboration with
Tonia Sideri was a data scientist herself before taking on her role as head of Novo Nordisk’s AI and Analytics Center of Excellence. Now she’s putting her experience to use helping the Danish pharmaceutical company in its quest to develop medicines and delivery systems to treat diabetes and other chronic diseases, such as hemophilia, obesity, and growth disorders.
In a highly regulated industry where failures are costly, Tonia’s philosophy is to fail fast through what she calls “data-to-wisdom sprints.” These two-week hackathons enable her group to rapidly test the feasibility of new product ideas with input from their colleagues on the business side.
Tonia joins this episode of the Me, Myself, and AI podcast to discuss her team’s approach to hypothesis testing, the benefits of incorporating design thinking into building data and AI products, and why she believes empathy is the most important skill a data scientist can have.
Read more about our show and follow along with the series at https://sloanreview.mit.edu/aipodcast.
Subscribe to Me, Myself, and AI on Apple Podcasts, Spotify, or Google Podcasts.
Give your feedback in this two-question survey.
Transcript
Sam Ransbotham: You might not often hear terms like “empathy” and “design thinking” when talking about AI projects. But on today’s episode, find out how one pharma company’s AI center of excellence takes a holistic approach to technology projects.
Tonia Sideri: I’m Tonia Sideri from Novo Nordisk, and you’re listening to Me, Myself, and AI.
Sam Ransbotham: Welcome to Me, Myself, and AI, a podcast on artificial intelligence in business. Each episode, we introduce you to someone innovating with AI. I’m Sam Ransbotham, professor of analytics at Boston College. I’m also the AI and business strategy guest editor at MIT Sloan Management Review.
Shervin Khodabandeh: And I’m Shervin Khodabandeh, senior partner with BCG, and I colead BCG’s AI practice in North America. Together, MIT SMR and BCG have been researching and publishing on AI for six years, interviewing hundreds of practitioners and surveying thousands of companies on what it takes to build and to deploy and scale AI capabilities and really transform the way organizations operate.
Sam Ransbotham: Today, Shervin and I are joined by Tonia Sideri, head of Novo Nordisk’s AI center of excellence. Tonia, thanks for joining us. Welcome. Let’s get started. First, maybe, can you tell us what Novo Nordisk does?
Tonia Sideri: We are a global pharma company. We are headquartered here in Denmark, and we are focusing on producing drugs [and] supporting patients with chronic diseases such as diabetes, obesity, hemophilia, and growth disorders. We are a 100-year-old company but still growing a lot [and] still very committed to the original values of the company and to our social responsibilities. There are more than 34 million diabetes patients using our products, and we produce more than 50% of the world’s insulin supply.
Sam Ransbotham: Currently, you lead the AI center of excellence. So, what is an AI center of excellence? What is your role there? What does that mean?
Tonia Sideri: An AI center of excellence can have different flavors in different companies, but what we do … we are a central team located in the company’s Global IT. We are a group of data scientists, machine learning engineers, and software developers working via a hub-and-spoke model across the company. So we want to minimize our distance from ourselves and our experts in the company — our data and domain experts — by working in cross-functional teams, product teams, across the company.
And we also want to increase the speed from where we go from a POC [proof of concept] of machine learning model to production. And that’s why we have analytics partners working across the company, and we also have an MLOps [machine learning operations] product team focusing on creating microservices across the whole machine learning model life cycle.
We want to take all the petabytes of data we consume as a company, all the way from our molecule identification to our clinical trials, to our commercial execution and production and shipping of the products, and take them from database, from flat files, from cloud storage and convert them to something that is ultimately useful for the company and ultimately supports patients’ lives. And that’s what we are here for: We want to bring this data to life.
We are around one and a half years old as a team, and we already have projects across the company. We are working with our R&D, for example, using knowledge graphs to identify molecules for insulin resistance; we have deployed different marketing mix model links and sales uplift recommendations models across our different commercial regions; and last but not least, we have recently deployed a deep learning machine learning model that uses vision inspection in our inspection lines — and that’s very important, because it’s an optimization on an existing process. However, it gave us a lot of skills of how to have live machine learning models in a very regulated setup, which is a GMP setup, [meaning] good manufacturing practices.
Sam Ransbotham: How does that work? Tell us more about that. That seems quite interesting.
Tonia Sideri: We were already using visual inspection the last 20 years from a rule-based approach that we have optimized, and now we have used different deep learning models to improve that. And of course, with deep learning, we are increasing the accuracy and the efficiency of the visual inspection process and thereby increasing quality and reducing the amount of good product going to waste due to particles being wrongly identified as defective.
So we save product and we optimize our products that way in a more efficient way, and we also produce less waste of good cartridges going to waste. But most importantly, what we get out of this project is the necessary capability of how to do machine learning in very regulated spaces —for example, like manufacturing of pharma.
Shervin Khodabandeh: Tonia, you’ve been a big advocate of design thinking in building data products, AI products. Tell us more about what that means and why it’s important.
Tonia Sideri: Yes. I think it started, first of all by … I used to be a data scientist myself. So sometimes I found myself working on projects that I could see … should have been killed earlier. So my interest in this is how to speed up our time for failure, and that’s why, when we started the area — and that was one and a half years ago — we really committed to actually start our projects by what we call a data-to-wisdom sprint.
[It’s] basically a hackathon [where] we work together with our business colleagues over a period of two weeks to really try to see what we can find from the data based on specific hypotheses. And at the end of these two weeks, we ask ourselves, is there any signal in the noise? Are the data good enough? Do we have the necessary technology to scale it further? And is there any business value out of this?
And if the answer is yes, then we go to the next step, where we do a POC [proof of concept], then [move on] to [the] implementation phase and, of course, operations. But if the answer is no, then within two weeks — very quickly — we should be able to kill it. And these two weeks we really use, with the help of agile coaches, also some design thinking techniques. But for me, it’s the outcome of the design thinking — how to use design thinking as a way to work cross-functionally and as a way to fail fast.
Shervin Khodabandeh: That’s great. No wisdom, you’re killed.
Tonia Sideri: Exactly.
Shervin Khodabandeh: Sort of like natural selection, right? Joking aside, I think this is a great idea because, Sam, how many times [do] we either see in our data, when we survey these thousands of companies, or in our conversations with executives where they are doing hundreds of POCs and pilots but there is just literally no value, and there is truly what I call AI fatigue across the organization because it’s like the whole organization has become this graduate school lab of, like, “Let’s try this; let’s try that.” So I love the idea of, just kill the ones that aren’t working so you focus on a handful that are valuable.
Tonia Sideri: Exactly. And for me, [from] those that are not working, we also have gotten a lot of learnings, because usually the reason that they’re not working is related to data. So at least we stress-test the data for two weeks based on what we want to achieve, and then we get some learnings: If we want to do this model in the future, what do we need to fix in our data to get there?
Sam Ransbotham: Ooh, that’s fabulous, because that’s actually tying back and learning from what you … I mean, it’s one thing to just cut a project off and say, “All right, well, we’re not going to keep dumping money into that if it’s not going to work,” but then there’s something else to … if you keep starting projects just like that over and over again, there needs to be some learning that those are going to fail or what you can do to improve those in the future. What kind of numbers are we talking about here? How much wisdom is there? Is there 2% wisdom, 20% wisdom, 97% wisdom?
Tonia Sideri: I think it’s very dangerous to try to quantify something like this, right? But one is the data wisdom, and the other, of course, is the change management wisdom, because we work together through this hackathon with our business experts, so even if something fails, they understand the way of working, and also we get a glimpse of their reality and they get a glimpse of what can be possible. And I think this wisdom is even more difficult to quantify because it will have a — hopefully — more of a wave impact effect in the future across the company.
Shervin Khodabandeh: If you look at the total opposite paradigm for what you’re talking about, it’s the old-school waterfall way of building these gigantic tech pieces, right? It was like tech development 20 years ago, where I remember we did a project and we looked at 100 companies building these massive tech products, and I think it was like 80% of these companies were building features and functionality that either nobody needed or could not be used with the rest of the technology, but they would only find this out like 18 months after development had started.
I guess it’s a totally new way, but sadly, there are still many organizations that are operating with that old paradigm, and they spend months in business-requirements gathering and planning and all that. And I think what you’re saying is, let’s get a good idea. Let’s start testing. If it’s got something there, then we double down and we make it big. But if it doesn’t, then we’ve learned something. And if that project, that idea, was important, then we could fix it. And I really, really like also your point around, it’s not just the technical part, it’s also the change management, and what it takes for it to work. It’s really, really good.
Tonia Sideri: Exactly. And by saying … that in advance, then we have no risk of failure, because it is how we work. We have two weeks, so it’s not going be our reputation on the line if the project doesn’t continue.
And having gated steps also, after even the MVP [minimum viable product] phase — [we] also [have] the ability to kill something there. And I think that helps, and also the budget [makes a difference]. The reason that a lot of companies have these long projects is because they have long budgets allocated to this. But in our case, we also assess if there’s any willingness to pay from our business side. Is what we do useful enough that our business is willing to invest in it?
Shervin Khodabandeh: Set the expectations upfront. Sam, imagine your — you know, Sam’s a college professor — your students come and say, “Professor, I’m warning you ahead of time: I will fail in two weeks.”
Sam Ransbotham: No, no. Actually, it is the opposite, Shervin. I go in and say, “Ninety percent of you are going to fail.” No, I don’t think that would go over very well.
Tonia, how do you transfer these learnings back? You mentioned that you do that. Is there a process for that? How do you codify, how do you make these things explicit and not just lore?
Tonia Sideri: That’s a good question. While we grow, we still have to find out what’s the right level of quantification that is not bureaucratic as well. But what we do is, first of all, during these two weeks, we have two demos across the organization, and especially with the business unit that we are working on. So at least that’s the change management part from a broader perspective, not only from the people [who] are working in the product team.
And then, regarding the data improvements or technology improvements, then we bring them back to our data governance [teams] or to the data owners or to our technology organization.
Sam Ransbotham: OK. That makes sense. One of the things you talked about — and something that Shervin and I, I think, are seeing overall — is that there’s a, let’s say, an increase in the maturity that we’re seeing. I don’t know, Shervin; maybe I’m reading too much into offhand comments that people are making. But I’m just seeing much more process getting put in place around what used to be very ad hoc, and maybe you’re a couple of steps ahead of this, looking at your building-block approaches to making different services consumable.
Can you explain how that works and how you’re developing these building blocks, and how other people are using them?
Tonia Sideri: Yes. So, of course, these building blocks and the idea of providing MLOps services or, in general, data services comes very much from this data mesh approach that now is the new hype, but especially for the MLOps work, what I can speak about is, based on our learning of how long it took to get a machine learning model validated, now we are creating microservices, wrapping existing services, either open source or from our cloud vendors — all the way from how we do model versioning, model monitoring, model validation, ground truth, storage validation — and then validating these services as qualified systems from a pharma setting. And in that way, we reduce the time to market from when we need to validate a GxP [good pharma process] model, because then we don’t expect any data scientists in the organization to build their own cloud solutions — to be both a data engineer, a software developer, and a validation expert — to bring the model into production, because by using these prequalified validation services, they can just focus on data science and use them as components. And we’re just building the first service based on our learning from this visual inspection model.
Shervin Khodabandeh: This is such a great point. If you look at a typical data scientist in a company, there will be such a wide variation in how much of their time’s actually [spent on] what you call extracting wisdom, or patterns or building models and testing, versus all the other stuff that’s prep work and setting up the environment and feature engineering and things that somebody else has already done, but in another part of the organization.
I want to ask you, Tonia, about talent. I mean, you’re talking about a way of working that is driven by design thinking, fail fast, highly interconnected with the business. What is the profile of the right skill sets from a data scientist/engineering perspective that’s going to be successful in that environment?
Tonia Sideri: That’s a good question. I think the technical skills, of course, should be a given there, and I can also see the market over time is getting more and more mature, so it’s easy to find those. But what is more difficult is these other, softer skills that make you a good value translator and a collaborator.
And for me, the most important skill of a data scientist is actually empathy — something we don’t expect from people from a technical field usually. It’s the ability to go into the businessperson’s mind and ask themselves, “If I was a marketer, if I was a production operator and I had to do the job every day, and I had the problems that I have, how would I use the data for something that would be useful for me?”
Being able to make this mental leap needs a lot of understanding of what is the reality of the other person and the ability also to communicate. So empathy and, of course, curiosity about the application of your machine learning models and the other person. And [those are] very difficult skills to quantify or interview for. It’s more of a cultural or a character trait.
Sam Ransbotham: It’s interesting, Shervin: We’re seeing maybe this first indication [that] it’s getting easier to find these technical skills. I think that’s an interesting transition.
Shervin Khodabandeh: Yep. That’s become more of a — as, Tonia, you’re saying — the table stakes that you need just to get started, but the real value is the softer skills and empathy. It ties well, Sam, to what we’re seeing as well, which is, when we look at the evolution of companies that are investing in AI, and we see that technology and data is only going to get them so far, but that big leap is all around organizational learning, interactivity with the business, process change …
Tonia Sideri: At least, to be fair about data scientists, there’s still a lot of shortage for machine learning engineers or data engineers or software developers, but for data science, because it becomes more mature as a field technically, it’s all the other skills that can differentiate somebody.
Sam Ransbotham: Tonia, what are you excited about next? What’s coming with artificial intelligence? I mean, we’re focusing on AI and machine learning. What are you excited about? What’s coming down the pike?
Tonia Sideri: I’m actually excited [about] data. No, it’s not so AI-related, but I think it’s relevant to a new trend that now it’s data-based; like, in order to fix our artificial intelligence and optimize, let’s optimize our data first. We also are actually investing more in the data mesh concept now — so, for example, treating data as a product, meaning that every time we want to make a new, let’s say, marketing mix model, we don’t have to go through the whole ETL [extract, transform, and load].
Shervin Khodabandeh: I once did a study 10 years ago, small group, maybe a couple hundred people in one company, but like 80% of their data scientists’ time was spent on ETL, and yet they had a data engineering group.
And the irony of it was — you’re talking about marketing mix optimization; this was actually for the marketing department — you’ve got data scientists next to each other in two cubicles working on something, using exactly the same data pipeline, but building it from scratch, both of them not even knowing that they’re using the same foundational features and … yeah, that’s a big deal.
Sam Ransbotham: Tonia, I know that you’re excited about that, because you talk about that in terms of tech indulgence; it seems very related there. That “Ikea effect,” perhaps?
Tonia Sideri: Yes, the tech indulgence. Yes. For me, that’s actually the worst sin that we make as technical people because the Ikea effect is the ability, I think, to give a higher value to something that you build yourself. And sometimes we tend to stay in a project because we built it ourselves or because we think it’s so cool to try the new machine learning algorithm. And for me, this tech indulgence is the biggest danger you can have, and that’s why it’s important to avoid this risk by working closer with the business and actually working with product teams, from a hackathon all the way to an operational product team.
Shervin Khodabandeh: I love that term, tech indulgence.
Sam Ransbotham: Tonia, we have a segment where we ask you a series of rapid-fire questions. So just answer the first thing that comes to your mind. What’s your proudest AI moment?
Tonia Sideri: I think this visual inspection problem we mentioned, not only for the business impact but especially for the capability providers — how to use machine learning in a GxP setting — and how quickly we worked together as a team with our business experts, with our manufacturing experts, to make this possible, and how quickly it actually got … validated.
Sam Ransbotham: I thought that might be your example because of how animated you were when you were talking about that. We can see this in video, but I think it probably comes across in your voice, too. What worries you about AI?
Tonia Sideri: As probably everybody on the show says, how it can be used also as a way to replicate our own biases. But on the other hand, I think technology also has the ability to decode these biases, because maybe it’s easier to remove these biases from technology than with people in the first place. So it’s a double-edged sword, but it worries me that we can replicate our own biases.
Sam Ransbotham: Bias is a common concern for everyone. What is your favorite activity that involves no technology?
Tonia Sideri: Reading books, definitely, and I try actually not to use even my Kindle for that, to read physical, 3D books. I can really recommend … I just finished Ishiguro’s book Klara and the Sun, about actually an AI robot that lives in a family and starts getting feelings about this family. I can really recommend that.
Sam Ransbotham: Well, that sounds great. Actually, I need a new book.
Shervin Khodabandeh: I love that. My 12-year-old boy grew up in the age of Kindle and screens and reading books, and so the first time he got an old-school book from the library, he’s like, “Dad, these books smell wonderful; what is this smell?” I was like, yeah, it’s an amazing smell that even a child of today’s day and age can appreciate.
Sam Ransbotham: What was the first career you wanted as a child? What did you want to be when you grew up?
Tonia Sideri: It’s very weird, but I wanted to be a garbage collector, [to] the surprise of my mother.
Shervin Khodabandeh: Me too! Me too!
Tonia Sideri: Really? That’s a very rare chance to find a fellow …
Shervin Khodabandeh: Yes. Fellow garbage collector enthusiasts.
Tonia Sideri: But I tend to think it’s somehow related [to our topic], right? I mean, you take something and you convert it to something else, and we collect data and we convert them to something else.
Sam Ransbotham: Yeah. I’m sure there’s some garbage analogy in there, too, with data that’s perfect. What’s your greatest wish for AI in the future?
Tonia Sideri: I will say “to be really democratized,” but I don’t really believe that it will get democratized anytime soon, because it needs so much conceptual understanding to really get democratized that I don’t think we’re going to get there. But that’s my real wish: that everybody has the tools, but more also know how to use them.
Sam Ransbotham: So by “democratize,” you mean everyone has access to those tools?
Tonia Sideri: Yes, and I think already there are so many platforms there that can help to have this low-code AI, but it’s more [that someone] has access to the tools [and is] able to use them. So [someone] has the right level of necessary knowledge to be able to use them and be independent in using them. And I think for that, it will take a lot of time, because it’s not a tool thing. It’s more, again, a change management — an educational — thing.
Sam Ransbotham: Tonia, great meeting you. I think that a lot of what Novo Nordisk has done with systematizing and developing processes around machine learning and AI are things that a lot of organizations could learn from. We’ve really enjoyed talking to you. Thank you.
Shervin Khodabandeh: Yeah, it’s been really a pleasure. Thank you.
Tonia Sideri: Thank you.
Sam Ransbotham: Please join us next time when we talk with Jack Berkowitz, chief data officer at ADP.
Allison Ryder: Thanks for listening to Me, Myself, and AI. We believe, like you, that the conversation about AI implementation doesn’t start and stop with this podcast. That’s why we’ve created a group on LinkedIn specifically for listeners like you. It’s called AI for Leaders, and if you join us, you can chat with show creators and hosts, ask your own questions, share your insights, and gain access to valuable resources about AI implementation from MIT SMR and BCG. You can access it by visiting mitsmr.com/AIforLeaders. We’ll put that link in the show notes, and we hope to see you there.
