Sparse by Command: Task-Conditional Compute Skipping for Multi-Task Inference Accelerators
Merged summary
TL;DR - This HW/SW co-design uses task commands to skip irrelevant accelerator compute during multi-task inference. On an FPGA driving prototype, it cut FLOPs by 66–76%, latency by 51–59%, and energy per inference by roughly half without degrading driving quality.
- A jointly trained gating network generates task-conditioned binary masks for hardware-aligned output-channel tiles.
- ISA-level bitmasks let the accelerator skip masked tiles without software intervention or changes to the inference pipeline.
- The INT8 tiled accelerator achieved 2.1–2.4× speedups, reducing latency from 9.12 ms to 3.74–4.44 ms.
- Energy per inference fell from 263 mJ to 108–128 mJ in CARLA closed-loop visuomotor driving tests.
Sources (1)
Sparse by Command: Task-Conditional Compute Skipping for Multi-Task Inference Accelerators
TL;DR - This HW/SW co-design uses task commands to skip irrelevant accelerator compute during multi-task inference. On an FPGA driving prototype, it cut FLOPs by 66–76%, latency by 51–59%, and energy per inference by roughly half without degrading driving quality.
- A jointly trained gating network generates task-conditioned binary masks for hardware-aligned output-channel tiles.
- ISA-level bitmasks let the accelerator skip masked tiles without software intervention or changes to the inference pipeline.
- The INT8 tiled accelerator achieved 2.1–2.4× speedups, reducing latency from 9.12 ms to 3.74–4.44 ms.
- Energy per inference fell from 263 mJ to 108–128 mJ in CARLA closed-loop visuomotor driving tests.