Deci claims breakthrough inference performance on Intel Sapphire Rapids

Israeli startup Deci claims to have achieved breakthrough performance in running computer vision and natural language processing (NLP) models on Intel’s latest Xeon processor.

Delivering graphics processing unit (GPU)-like artificial intelligence (AI) inference performance, the breakthrough was accomplished with the company’s proprietary Automated Neural Architecture Construction (AutoNAC) technology, which generates custom hardware-aware AI model architectures.

For computer vision, Deci claimed to have delivered more than a three-fold increase in throughput, as well as a 1% boost in accuracy, when compared with an INT8 version of a ResNet50 convolutional neural network running on Intel’s fourth-generation Xeon processor, codenamed Sapphire Rapids.

For NLP, Deci also delivered a more than three-fold increase in acceleration compared with the INT8 version of the Bert language model on Intel Sapphire Rapids with improved accuracy. The models were compiled and quantised to INT8 with Intel’s Advanced Matrix Extensions (AMX) and the Intel extension for PyTorch.

Since 2019, Deci and Intel have been working together under the latter’s Ignite programme for early-stage startups to optimise deep learning inference on Intel chips, with inference results improving over generations of Xeon processors.

Deci, which raised $25m in Series B funding in July 2022, is also a member of the Intel Disruptor programme aimed at fostering innovation in AI and data-centric use cases and has collaborated with Intel on multiple MLPerf submissions.

Yonatan Geifman, Deci’s CEO and co-founder, said the latest performance breakthrough marks “another chapter in the Deci-Intel partnership which empowers AI developers to achieve unparalleled accuracy and inference performance with hardware-aware model architectures”.

On the implications of Deci’s breakthrough on hardware decisions for organisations running AI models, Geifman noted that inference hardware selection is an intricate process with many factors to consider, such as where the workload is deployed (cloud or edge), cost, power consumption and performance requirements.

“The combination of Deci and Intel Sapphire Rapids opens up new possibilities for computationally demanding deep learning applications to be deployed on CPUs [central processing units] that otherwise wouldn’t be considered, such as deploying large language models like Bert, and delivers a compelling solution from both cost-efficiency and performance perspectives,” he said.

Deci has also used AutoNAC to optimise the performance of AI models running on Nvidia GPUs. In August 2022, Deci joined Nvidia’s Metropolis, an application framework and set of developer tools to help organisations build AI applications.