Bringing AI directly into embedded systems is much more than just running a model on the device. When a decision must come within milliseconds, every element of the chain becomes decisive. The sensors, correctly sized memory, optimized models, processor, AI accelerators, operating system, and software must be designed together to achieve adequate scheduling and predictable response times as well as high performance and low latency. Latency becomes a design requirement In edge AI applications, processing speed must be carefully evaluated by considering other factors, including the time it takes to run a neural network. True latency is the latency between the capture of the event and the availability of the consequent action. In an industrial control system, for example, the path includes reading the sensor, transferring data, pre-processing, inference, interpreting the result, and, finally, triggering an output. If a camera detects a moving object, it is not enough for the classifier to produce the result quickly; frame capture, memory access, and communication with the actuator must also be considered. This also changes the way performance is defined. A system that completes an inference on average in 5 ms is not necessarily suitable for a real-time application if it occasionally takes 30 or 40 ms. In fact, in reactive systems, the variability of response time also counts. The design of these systems must therefore consider average latency and worst-case behavior. Edge AI has become a technology of great interest precisely because it eliminates the transition to the cloud and allows decisions to be made directly on the device. Embedded processors, microcontrollers (MCUs), and dedicated accelerators can now run models locally with low power consumption and significantly reduced response times. The advantage, however, becomes tangible only when the entire architecture is built around the time requirement. From sensor to decision One of the most common causes of latency is not found in the AI model but in the operations that precede it. For this reason, design must start from the complete data flow, establishing the sampling rate, buffer size, information format, and maximum time available for each step. The connection between peripherals and memory is a very important point. The use of direct memory access (DMA), for example, allows samples or frames to be transferred without continuously committing the CPU. While DMA fills a buffer, the processor can process the previous buffer. With a double-buffer structure, a pipeline can be created in which acquisition and computation proceed in parallel. Parallel pipelines allow data to be acquired, pre-processed, and processed simultaneously, increasing throughput without increasing processor frequency. The same principle can be applied to video, audio, and data from industrial sensors. Pre-processing must also be considered part of latency. Image scaling, filtering, normalization, frequency-domain transforms, and feature computation can consume a significant amount of time. In some cases, pre-processing takes longer than inference itself. The digital-signal-processing phase can also affect the overall time of a classification performed on an MCU. The choice of data format is equally important. Processing 8-bit integers (INT8) instead of floating-point values can reduce memory and computational load, especially when the model and hardware are optimized to leverage this data format. Quantization, however, must be verified along with model accuracy because reducing the number of bits does not automatically mean a better system. Design responsiveness as a feature of the entire system Low latency in edge AI is not a property that is achieved simply by choosing a faster processor or a smaller model. It is the broader result of a series of architectural decisions involving sensors, memory, CPU, accelerators, software, and task management. The fundamental principle is to consider response time a
Designing Edge AI for Real-Time Systems
Bringing AI directly into embedded systems is much more than just running a model on the device. When a decision must come within milliseconds, every element of the chain becomes decisive. The sensors, correctly sized memory, optimized models, processor, AI accelerators, operating system, and software must be designed together to achieve adequate scheduling and predictable [...] The post Designing Edge AI for Real-Time Systems appeared first on Embedded .
本站仅提供摘要,完整内容请阅读原文(版权归原媒体所有)
https://www.embedded.com/designing-edge-ai-for-real-time-systems/
相关阅读
10-18 13:56
10-18 12:57
10-18 07:38
10-17 14:24