Skip to content

Fix mixed precision calibration encodings#199

Open
ptoupas wants to merge 3 commits intomainfrom
fix/rvc4-int8-int16-calibration-encodings
Open

Fix mixed precision calibration encodings#199
ptoupas wants to merge 3 commits intomainfrom
fix/rvc4-int8-int16-calibration-encodings

Conversation

@ptoupas
Copy link
Contributor

@ptoupas ptoupas commented Feb 6, 2026

Purpose

Fix an issue with the custom encodings generation for mixed-precision (W8A16) quantization on RVC4 and introduce a new quantization mode, INT8_INT16_MIXED_ACCURACY_FOCUSED, which combines mixed-precision quantization with the enhanced calibration method to prioritize accuracy.

Specification

  • Added a new function to retrieve tensors preceding the model outputs up to a specified depth and include them in the custom encodings generated for mixed-precision (W8A16) quantization on RVC4. This serves as a temporary workaround to maintain compatibility with DAI, which does not support INT16 inputs and outputs.
  • Introduced a new quantization mode, INT8_INT16_MIXED_ACCURACY_FOCUSED, which combines mixed-precision quantization with enhanced calibration to improve accuracy.

Dependencies & Potential Impact

None / not applicable

Deployment Plan

None / not applicable

Testing & Validation

Tested on a pool of different models, like yolov8n-seg, yolov10n, yolov11n, and tasks like classification, instance-segmentation, and detection.

…ed calibration method on INT8/INT16 quantization.
…m encodings at a certain depth (backwards) from the outputs of the model. Used for the INT8/INT16 quantization cases only.
@ptoupas ptoupas requested a review from a team as a code owner February 6, 2026 13:57
@ptoupas ptoupas requested review from conorsim, klemen1999, kozlov721 and tersekmatija and removed request for a team February 6, 2026 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant