In this session, we’ll explore how Infineon’s PSOC Edge is revolutionizing natural language processing (NLP) directly within ultra-low power embedded systems. With cutting-edge advancements in on-device AI acceleration, optimized memory hierarchies, and secure compute domains, PSOC Edge is enabling voice and language understanding at endpoints that previously relied on cloud inference. This breakthrough technology is driving real-time NLP with minimal latency and power consumption, all while ensuring data privacy at the edge.

Joining us today is Omar Cruz from Infineon Technologies, who will unpack the technical innovations and system-side strategies behind this new generation of edge intelligence devices. Together, we’ll dive into how hardware-software co-design is enabling smarter, faster, and more secure speech interfaces for edge devices.

[FULL TRANSCRIPT BELOW]

SWF: Welcome to another episode of AI with Sally, an EE Times podcast that brings you inside my conversations with luminaries of the AI chip industry. Today’s episode is brought to you by our sponsor, Infineon Technologies. In this episode, we’re going to talk about how Infineon’s PSOC Edge is enabling natural language processing directly in ultra low power embedded systems. With advances in on device AI acceleration, optimized memory hierarchies, and secure compute domains, PSOC Edge brings voice and language understanding to endpoints that previously relied on cloud inference. We’ll discuss how Infineon’s hardware software co-design enables real time NLP with minimal latency and power consumption, all while maintaining data privacy at the edge. Joining us today is Infineon’s Omar Cruz. He’ll unpack the technical innovations, the system side strategies driving this new generation of edge intelligence devices. Here’s my conversation with Omar.

Okay, Omar. Welcome to the show.

OC: Thank you, Sally.

SWF: So Infineon has taken this PSOC Edge product into production. This is Infineon’s microcontroller family with hardware acceleration for edge AI. Tell us first about which markets are driving the need for hardware acceleration for AI in microcontrollers today.

OC: Yeah, definitely. Great question. What I keep telling people is that with now Edge AI, we are looking at new markets, right? That part of the traction that we’ve had with PSOC Edge is around the specific or different types of target markets, right? We are seeing a lot of traction on Edge AI around HMI applications. From the HMI applications, you can think about appliances. You can think about even in industrial space like device usability, and also factory automation, right? Those type of applications. Now they’re looking to the Edge AI capabilities, taking things back to smart home, thermostats, smart speakers, door locks. Of course wearables, they have taken already a lean into the Edge AI, but they now have gone even further. So think about fitness watches, augmented reality, mixed reality, virtual reality glasses, and all the accessories as a wearable. And then robotics, whether we’re talking about consumer robotics or whether we’re talking about industrial robotics, like the ones that you can see in industrial factory lines, or the ones you can see at home, like the vacuum cleaners, right? All of those also, they are leveraging the Edge AI capabilities that now we are able to bring. And then definitely last but not least is around monitoring and vision. So IP cameras, battery operated smart doorbells, as well as security cameras is something that we are putting a lot of emphasis. That is something that we are seeing a lot of traction and a lot of developers interested in to this particular set of applications.

SWF: So we talk about on-device AI quite a lot on this podcast. But to recap, what do you think are the biggest challenges with AI at the edge from the developers viewpoint today?

OC: I think from the developer’s point of view, part of the challenge is, am I going to be able to drive machine learning and applications without microcontroller? Because you need a lot of performance, right? And that performance also comes in the sense of a lot of operations being run. The other part is a little it counterintuitive, right? But also, you need low power. We are talking about consumer type of applications. We are talking about energy efficient needs. That energy efficiency is an absolute requirement, and an absolute challenge for these type of approaches, right, where you need high performance to run machine learning, but at the same time you need low power, right? It might be able, or might be hard for a developer to get a hardware that allows you to do that. The other thing that comes also of course when you talk about Edge AI is latency in real time response. And it kind of goes along the lines of high performance. But nevertheless, having that latency in real time response comes with the fact that you perhaps don’t have the time to go into the cloud, do the processing in the cloud, and then come back. That’s going to take some time. So having the actual processing happen at the edge is something that developers are looking into. The other aspect is security. Going ahead, again, taking this to the cloud, it might be susceptible to malicious attacks. But if you keep things at the edge, you might mitigate those malicious attempts of attack. On top of that, you would need to have some level of security or a high level of security in your end device nevertheless, because everybody’s might be able to just get into your end device and just start the attack from there. So high security is also a key challenge and a key requirement that we’re seeing from different developers. Last but definitely not least is the integration. When you are thinking about machine learning, you might need to have multiple models be running at the same time. You might need to have voice activation mechanisms. You might need to have audio type of capabilities, graphics, and then radar or multi-sense type of features or requirements. And this is something that we are seeing as key challenges, and this is something that we are addressing with PSOC Edge.

SWF: Specifically for things like speech and NLP that we’re going to be talking about a little bit today, the latency is super critical, right? We need to make it feel natural when you’re talking to the device, right?

OC: Yes. Yes, that’s a key aspect. I don’t know if you have your smart home device at home.

SWF: I have Alexa, yes.

OC: You have Alexa. Just imagine, like I have a 5-year-old. So that 5-year-old, he likes to talk to Alexa. It’s challenging. And latency doesn’t help, right? So having those type of challenges, we can see it in natural language processing. And now having an edge device that can just process everything at the edge is something that is, you know, I think is groundbreaking.

SWF: Absolutely. From Infineon’s viewpoint, or from your viewpoint, what are the major hurdles that you have to get over in terms of scaling this PSOC Edge microcontroller in the market?

OC: Yeah. We are changing the paradigms, right? Normally, when you think about embedded processing, think about microcontrollers. Think about microprocessor. You have a control, a core, that normally does control type operations, right? And you rely on the workforce or what you can do with that particular core. Now we are introducing hardware accelerators that are neural processing units, right? This is a different type of handling different use cases in these areas. So first of all coming up with a software or a set of software tools that allowed you to easily integrate those neural processing units into your workflow, into your development workflow, it’s gotta be a challenge. Then comes of course the technical knowledge that you need to have, comes the support that a vendor such as Infineon needs to provide. The overall, the adoption, right, because I think this is going to be the future, but you need to enable that, and this is something that we are putting a lot of emphasis in in PSOC Edge in Infineon, and this is something that we are really excited to be here in this position today.

SWF: So you mentioned software and toolchain, which is critical. We’re going to come to that a little bit later in the conversation, because first, obviously, I want to come to my favorite topic, which is hardware. The PSOC Edge family I know has two different AI accelerators on chip. I was hoping you could tell us a bit more about these hardware blocks. Why do we need two different AI accelerators on chip, and what’s the difference between them, really?

OC: Yes. I mean, and when you start thinking about your use cases and applications, let’s stick to the voice type of use case. You have your always on device that might be battery operated. First of all, you may just do [wait word] detection and keyword spotting, and then perhaps there’s going to be a quick interaction and nothing’s going to happen. But then you may do a natural language processing type of use cases. When you wake up your device, you start talking to that device in a natural way, and then the device comes back at you. Those are two different sets of use cases. The first one is the keyword spotting, like a pretty low complex type of use case, and the second one is a little bit more complex, a little bit more of a natural language processing that requires more processing capabilities. Having said that, we have come up with two different types of accelerators, or two different types of neural processing units. One, which is the one that’s going to address the high processing, the complex side of use cases like natural language processing, and that’s what [Ethos] U55 tied with our Cortex M55 and our Helium DSP architecture is going to be able to bring. But we also have come up with a low complexity MPU accelerator that we have developed in house and that we have called [NN Light]. So [NN Light] is an Infineon proprietary accelerator, is assigned to optimize the energy implementation. It’s designed to optimize models for wait word detection or anomaly detection or low resolution compute vision and sensing. It will keep things quite efficient when it comes to the energy perspective, and if you’re required to use transitioning to the high performance natural language processing vision [?10:12] when Ethos U55 has more processing capabilities will be really [heavy]. So that’s different. That’s a unique approach that we have taken at Infineon, and that’s an approach that allows us to become quite highly energy efficient in the embedded designs.

SWF: So I said the PSOC Edge family has two accelerators, but there’s actually four subfamilies, and not all of them have both. Maybe you could just talk us through kind of briefly, what is the difference between the four subfamilies? Which ones have both and which don’t.

OC: Yeah, definitely. So first of all, even though it’s our first family of microconverters, we have already come up with four different variants of PSOC Edge. The four of them are going to have the [NN Light] NPU accelerator, because we believe that’s going to be a quite efficient use case for battery operated type of applications, for different types of use cases that you may have at your design. So E81 and E82 are going to have the [NN Light] accelerator. E81 is going to be kind of like the baseline of our microcontroller family. If you need to add the graphics, that would be E82 is going to be there. Some of the designs do not require graphics, and we understand that. You don’t need to pay for having graphics components if you are not going to use them. But if you do require them, the E82 is going to be there. At the same time, you may not require hardware advanced AI acceleration or advanced NPU capabilities. So that’s why we have come up with the E81, E82. But if you do require advanced acceleration, NPU capabilities, we have the E83 and the E84 lines. The difference between the E83 and E84 is going to be similar to the differences to the E81 and E82, meaning E83 is going to have advanced machine learning capabilities, but it’s not going to have the 2.5B graphics processing unit. And then the E84 is going to have advanced machine learning capabilities, is going to have the 2.5D graphics processing unit, and on top of that we are adding an extra megabyte of S-RAM for those visioning graphics applications combining to one microcontroller.

SWF: Got it. So for [NN Light] then, your in-house accelerator, this is aimed always on applications. Can you tell us a bit more about how you manage the power consumption with that in mind? Is it fancy sleep modes, voltage and frequency scaling? Tell us about how you manage kind of wake-up events as well. I’m curious about how much power that block actually consumes as well.

OC: Yeah, we have kept things quite highly energy efficient. Of course in the architecture, when you look into the architecture of PSOC Edge, you’re going to find a lot of different power modes that you can leverage. So we are going to have hibernate mode. We are going to have a deep sleep mode. We’re going to have a sleep mode, and we’re going to have a running mode. Within the running mode, you’re going to find different frequencies that you can set depending on your requirements, depending on your needs. But if we can just stick to the wait word detection flow, if we can stick to that part is, we relied on the [NN Light]. We rely on the low power domain that we are featuring in PSOC Edge. We are talking about milliwatts, the low milliwatts, right? Single digit milliwatts to perform that wait word detection, to go with keywords pointing with acoustic activity detection that can sense and rely on the [NN Light] and rely on the analog and digital mechanisms that we have in the PSOC Edge architecture. So quite efficient when it comes to that acoustic event that is going to trigger the next stage in your flow. And if required, you can just turn on the high performance domain, the Cortex M55 that is going to be sleeping at that point, and they do absolutely require higher performance if you have a use case that requires more processing, then you can turn on the Cortex M55, and you can of course turn on the Ethos U55 neural processing unit.

SWF: Do you have a figure for us actually on power consumption for the different blocks? So what’s the minimum power consumption I can use to do an inference with with [NN Light] in this always on mode?

OC: Yeah. We have single digits of milliwatts, and it’s going to depend on the use case. And then when you take things into, let’s put an example of natural language processing, then you’re going to still stick to the milliwatts, perhaps hundreds of milliwatts, but we are talking about natural language processing. So of course there are different stages. Of course there are different use cases. Of course it depends on the optimization of the end developer designs, but this is something that we are seeing just as the high level numbers that we are seeing from our different blocks.

SWF: So it’s up to the developer then to decide which part of the AI workloads are run where, or with more than one accelerator on chip, how do you partition that workload effectively?

OC: Definitely the developer has a lot of say into that, and we provide the tools and the mechanisms for the developer to make it easy. We also provide our own AI tools that will allow you, or we come up with an optimization of these type of use cases and these type of scenarios. And we’ll talk about more when it comes the time to talk about software. But we can take a specific use case and then our machine learning software is going to optimize the workload, and it will allow you to come up with some basic recommendations from the code perspective. Of course the developer can always decide. We have a pretty user friendly interface from [Modus Toolbox and Deep Graph], and then the developer will be able to just decide what will be the best workflow that will work for him.

SWF: Yeah. So I can imagine a few use cases where it’s like you switch on [NN Light] and then you switch on the Ethos. Like maybe you’re doing voice detection through to understanding the command or something. But I can imagine more use cases, especially with NLP where it’s like, maybe I want to use Edge AI to wake up and do a basic query. But then maybe I want to fall back onto Cloud AI for something more, a more complicated type of conversation. Maybe I can use the device’s internet connection to access the cloud. Is that going to be possible PSOC Edge, or how would I implement that?

OC: It is possible. It is definitely something that the developer can leverage. Actually, what we are also introducing PSOC Edge is now the feasibility to run sophisticated language models. And we’re talking about tens of millions of parameters, on device, with the high energy efficiency setup that I have talked about. And this means like everyday devices, from ovens to watches, can have a natural language capability without relying on the cloud. So users can benefit from this, what we call the zero data egress, that means that no private data is going to go into the cloud. They can leverage that, because everything can stay at the edge. They can leverage the consistent low latency that we have also kind of already talked about. And then of course having a functionality, even if you are offline. Of course the [neno] line allows you to come up with different setups, with different scenarios. It allows you to even come up with over the air updates for your firmware and whatnot. But we are introducing a new paradigm, a new level of processing where you are being actually able to have a natural language processing without relying on the internet connectivity.

SWF: In terms of scalability, is there a possibility, maybe I have my AI application in mind, but then as it develops, maybe I have features or something. Maybe I start to in the [NN Light], and then maybe I realize I need to use the ARM Ethos accelerator. Do I need to change my code to do that? Or how scalable is my AI application within this same hardware?

18:40

OC: It is quite scalable. And one of the really nice things that we have come up with on PSOC Edge is that full scalability that goes not only from the being compatible perspective, from the hardware compatible perspective, but also from the software compatible perspective. So you’re going to have your system rely on [NN Light] today, for example, having an [Me 81] type of microcontroller is going to do the keyword detection, is going to be doing the wait word detection. But you say right now, this is all I need. But if in the future, if you want to rely on this natural language processing, what we have defined as edge language models, you can just rely on the E83, add pieces on top of your code, with the help of Infineon of course, and then just have a seamless transition from E81 to E83, not only from the software perspective but also from the hardware perspective and it’s plain compatibility options that we are providing with PSOC Edge.

SWF: Got it. So overall, hardware-wise, most of the big microcontroller players have AI enabled microcontrollers, and most have either their own in-house AI accelerator or some are using ARM too. How do you distinguish the PSOC Edge family from some of the other microcontrollers that people might be considering?

OC: Yes, a really good question. One of the key aspects that we have is that high energy efficiency. So we have the ability to have a high performance domain with a Cortex M55, with Helium DSP, with EDSU 55. But we also have that ability to provide that low power domain with Cortex N33 running at the lower frequency there, and tied to our Infineon [NN Light]. So that will take things through in terms of energy efficiency, as I’ve already described. On top of that, we have also come up with the highest level of security for a microcontroller. And because of our secured enclave architecture, this has allowed us to achieve the PSA level 4 of integrated secure enclave certification, from the platform security architecture consortium, and this is the highest level certification being achieved by a microcontroller, which in today’s world, security is quite important. On top of that, we have a unique HMI integration, and that’s part of the pedigree of PSOC overall as a family. And that HMI integration now has evolved to have multiple analog offerings, multiple DL offerings, having graphics, having of course the different type of cores that I have already described. So that HMI integration is unique. And last, but again definitely not least is just what we have come up from the software side, like the state of the arts software tools with [Modus] toolbox, a lot of emphasis on [Modus] toolbox, but now we have also have the support from DEEPCRAFT, which is game changing software that we are implementing, that we are pairing up with PSOC Edge, and this is also quite unique in the industry.

SWF: Yeah. I think it’s time we touched on software, like you’re saying. Software toolchain for Edge AI, and this kind of constant area of strife or constant area for improvement for a lot of chip companies. So tell us kind of from the beginning, how did Infineon approach building this Edge AI toolchain, and tell us a bit more about Deep Craft as well.

OC: Yes, definitely. So two and a half years ago, Infineon acquired a company that used to be called [Imagimo]. Now we have rebranded that [Imagimo] to DEEPCRAFT, and we have put a lot of investment into what we can develop or we can bring to the table with DEEPCRAFT, right? So we saw that particular platform that the [Imagimo] team had back in the day, and we realized that that was going to be a pretty good complement to what we were thinking of bringing from the hardware perspective. So aligning PSOC Edge, [Modus] Toolbox, and DEEPCRAFT will put us in a special place. And that’s the reason why I’m so happy to be here. Another reason why when we take PSOC Edge into production, we also came up with a lot of different products from DEEPCRAFT as well, because they are complementing our offerings from the hardware perspective, but also now from the software point of view. And that alignment between DEEPCRAFT and [Modus] Toolbox is also unique, because from the [Modus] Toolbox perspective, you have what the developer typically sees. We have the drivers. We have code examples. We have the middleware. But now with DEEPCRAFT, you have the AI angle that allows you to complement your solution, that allows you to come up with models, but at the same time already having all the drivers and all the different needs that you may have from the hardware perspective as well.

SWF: So the [Modus] Toolbox, and then DEEPCRAFT, are they separate tool chains today? Are you planning to combine them? Or how do they work together?

OC: They are separate tool chains, or they are separate tools. Let’s put it like that. So DEEPCRAFT tools, it has a different set of categories that I’m going to go into just in the next couple of minutes. But from the DEEPCRAFT tools, whether we’re talking about DEEPCRAFT Studio, whether we’re talking about the DEEPCRAFT model converter, whether we’re talking about the [OSS] system and the audio enhancement solutions, these tools are designed to generate content that can be used into [Modus] Toolbox. So when you do the programming into your device, you are going to rely on [Modus] Toolbox, but there is a deep interconnection between DEEPCRAFT and [Modus] Toolbox. And we have taken it further, right? When you download the newest versions of [Modus] Toolbox, you kind of have the ability to also download DEEPCRAFT on the go, and they are going to work really, really well together. Even though they’re different tools, there is a nice synergy between DEEPCRAFT and [Modus] Toolbox.

SWF: So if I am already working edge AI somewhere, I’ve got my own model, let’s say, I can bring that into the tool chain, right? Or what if I want to buy or license a model from a third party? It’s going to be easy for me to bring that along with me, right?

OC: It’s going to be easier for you. We also, as part of what we have launched recently, is the DEEPCRAFT model converter. So what we have here is a software application for converting, optimizing, and validating AI models to run on PSOC Edge. So let’s assume that you have a PyTorch model. You can use DEEPCRAFT Model Converter, choose even between an [institute’s] graphic user interface or a command line interface, and adapt your existing model to PSOC Edge, optimize it, and have it ready to go, thanks to the tool, software tool of DEEPCRAFT Model Converter.

SWF: If on the other hand I don’t have much experience with Edge AI, and I am trying to do an NLP application or something, is DEEPCRAFT still going to offer me the right place to start?

OC: Yes. DEEPCRAFT offers different layers of options for you, right? If you are starting from scratch, if you are starting from just, I want to start collecting data and I need help with that pre-processing, with that deployment of the model, that training of the model and the deployment of the model into the microcontroller, we have DEEPCRAFT Studio. So DEEPCRAFT Studio is a nice user interface that will allow you to have that data collection. But we have taken things even further, where we have also come up with solutions, and particular solutions like voice assistant, particular solutions like audio enhancement, where you can rely on what we have done already, and these are tools that will allow you to customize your design. So you can come up with your own wait word detection, where you can come up with your own keyword spotting type of words, put them together, put them in the tool, and our tool will allow you to just take that, come up with the model, right model, and just deploy into the microcontroller, in this case PSOC Edge. So easy to use, low complexity, and now you’re going to be able to have a real voice type of device into your, in your hands, I would say.

SWF: So the NLP capabilities I think are particularly intriguing. Could you overall give us an example, putting it all together then, what kind of NLP application can I achieve on a PSOC Edge microcontroller? Remember, this is a microcontroller. But how complex of an NLP application can I build with PSOC Edge?

OC: It can get pretty complex for a microcontroller. And we’re talking about even having 25 millions of parameters being deployed in the microcontroller. And this is something that we are optimizing. This is something that we have come up with, some benchmarking data. And even though it’s a small language model, what we have defined as edge language model, it’s quite powerful when we actually compare it to some of the large language models that are out there. So we are pretty proud of what we have come up with in this particular case. If you start thinking about the target applications, and this is something that it’s quite, for me quite interesting to just see the developers. We have smart watch developers. They can use a virtual assistant, where you don’t actually require a phone or network connectivity. You can have an assistant that you can rely on that edge language model, with more than 25 million of parameters, and it’s actually going to work really, really well. On the same approach or the same type of things, you can have a voice assisted oven and fridge. Of course you can already define the type of commands that you are going to have in those type of scenarios, but you can make it in a natural language processing kind of way that will allow you to have a pretty nice experience with your machine that you have in your home. Same type of thing but taking it into the factory automation. So think about the factory floor assistant with a voice enabled industrial scenario. That’s something that you can also bring with this set of small language models. I have developers working with smart healthcare devices. The patient may be at home, and perhaps they don’t need network connectivity, and they just would like to know, you know, what’s their cardiac frequency at that point. And this is something, sort of the use cases that I’m seeing, quite excited, and now bringing natural language processing capabilities in a device such as PSOC Edge and a microcontroller at the end of the day. It’s quite, quite interesting.

SWF: So if someone’s listening to this and they want to test out PSOC Edge for themselves, what’s the first thing they should do, or where should they go?

OC: I would recommend them to go to infineon.com/psocedge, and then from there you are going to see so many different assets. You are going to have the ability to buy either of two boards. We have an evolution kit where we have put all the different interfaces. We have put everything together to just leverage the full capabilities of PSOC Edge. We have also come up with a low cost kit that we call the PSOC Edge E84 AI Kit, that also is available now. That has a lot of sensors. That has radar. That has microphones. That has even the ability to connect to the display, and will also allow you to take things into the next level. That with the help of DEEPCRAFT, with the help of [Modus] Toolbox. We are in a really good place. If you need more help, we have trainings. We have demos that you can just deploy into the microcontroller, deploy into your board itself. So a lot of collateral has been created. That’s the reason why I’m so excited to be here, and be in this position with Infineon PSOC Edge.

SWF: I think that’s the perfect place to finish. Thank you so much, Omar, for joining us today.

OC: Yeah. Thanks, Sally.

SWF: Thank you very much, Omar, for the interesting conversation about NLP at the edge. This episode of AI with Sally was brought to you by our sponsor, Infineon Technologies. Thank you very much, Infineon, for sponsoring us. If you want to learn more about Infineon’s PSOC products, DEEPCRAFT, or Infineon’s other embedded AI offerings, please check the links on the podcast page at eetimes.com.

That brings us to the end of the episode. Please tune in again next time for more conversations around AI silicon and systems. If you are listening to this on the podcast page at eetimes.com, links to articles on topics we’ve discussed are shown on your left. AI with Sally is brought to you by AspenCore Media. The host is Sally Ward-Foxton, and the producer is James Ede. Thank you for listening.