With edge AI, the inference has moved from the cloud to the embedded device. Cameras, wearables, industrial sensors, robots, and vehicles can run trained models, making predictions or decisions on new and uncertain data. Because these devices are often powered by small batteries, an energy issue arises. Engineers developing embedded systems at the edge face three main challenges: Models are getting more capable and compute-hungry, customers and regulators ask for attention to sustainability aspects, and the device enclosure size isn't increasing. Why more TOPS is not the answer Traditionally, tera operations per second (TOPS) has been the most commonly used metric to evaluate the performance of edge AI systems. While it is indeed an easy spec to compare, it does not fully adapt to battery-powered devices operating within strict thermal design power constraints. For these, inference per joule (or its inverse, joules per inference) is more appropriate. Unlike peak throughput, inference per joule measures how many AI predictions or tasks a system can complete for each unit of electrical energy consumed. Higher values of inference per joule mean better efficiency , which in turn lowers operating costs and cooling requirements in edge devices. Two devices with the same peak TOPS can differ by an order of magnitude in energy draw, depending on the memory architecture, how well the compiler maps a model onto the hardware (i.e., using 8-bit integer versus 32-bit floating-point variables), and how much the device absorbs when there's nothing to do. For instance, TOPS and its floating-point counterpart (FLOPS) do not consider how memory affects energy computation. Moving data from memory to processing core often requires more energy than the computation itself. The 'memory wall' In the semiconductor industry, the processor computing power is growing much faster than the rate at which memory can provide it with data. This speed disparity creates a performance bottleneck known as a " memory wall ." Some memory technologies can be adopted to solve this bottleneck. The first is compute-in-memory (CIM). Traditionally, data is continually exchanged between memory, where the AI model's weights are stored, and the processor, where computation is performed. CIM flips this on its head, performing the math calculations inside the memory cell rather than shuttling data between processor and memory. CIM not only breaks the memory wall; it is also more energy-efficient and accelerates the execution of algorithms used in AI. CIM can be realized in both analog and digital forms. Analog CIM computes according to physical laws, such as Ohm's law and Kirchhoff's current law, by directly applying the voltage and current in memory cells. Digital CIM integrates basic digital logic gates, such as AND or NAND ports, inside or adjacent to the memory rows/columns, providing higher-precision results. Other memory technologies, as alternatives to CIM, include latest-generation nonvolatile memories, such as resistive RAM and magnetoresistive RAM, and hybrid architectures , such as Winbond's Customized Ultra-Bandwidth Elements (CUBE) memory architecture (Figure 1). Winbond's technology stacks memory vertically on top of the processor using microscopic through-silicon-via connections, which shorten the physical distance traveled by electrons. Suitable for larger edge AI systems, such as system-on-modules, this solution increases bandwidth and decreases power consumption. Figure 1: Winbond's CUBE 3D memory architecture claims to achieve high power efficiency, consuming less than one picojoule of energy for each transferred bit. (Source: Winbond) Algorithm optimization Inefficient models can significantly contribute to energy waste in edge AI systems. The conventional procedure of training large models in the cloud and then deploying them to the edge is no longer sustainable. The current trend is model optimization for specific hardware. Embedl, a spin-out from Chalmer