aitk

DeepSeek-R1-Distill-Qwen-7B Model Optimization

This repository demonstrates the optimization of the DeepSeek-R1-Distill-Qwen-7B model using post-training quantization (PTQ) techniques. The optimization process is divided into these workflows:

• OpenVINO for Intel® GPU/NPU
  • This process uses OpenVINO specific passes like OpenVINOOptimumConversion, OpenVINOIoUpdate and OpenVINOEncapsulation
• NVModelOptQuantization for NVIDIA TRT for RTX GPU

Intel® Workflows

These workflows performs quantization with Optimum Intel®. It performs the optimization pipeline:

• HuggingFace Model -> Quantized OpenVINO model -> Quantized encapsulated ONNX OpenVINO IR model

NVModelOptQuantization for NVIDIA TRT for RTX GPU

To run this workflow, you need to install CUDA as required in Doc.