Skip to content

chore: add support for quantized versions of CV models CLIP, Style Transfer, EfficientNetV2, SSDLite #940

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
barhanc wants to merge 11 commits into
base: main
Choose a base branch
from
+348 −102
Open

chore: add support for quantized versions of CV models CLIP, Style Transfer, EfficientNetV2, SSDLite #940

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
cc7f167
feat: add quantized models for style transfer
barhanc Mar 3, 2026
0968a67
feat: update EfficientNet V2 S model paths and add quantized model su…
barhanc Mar 6, 2026
05e85c1
refactor: rename quantized model constants for style transfer
barhanc Mar 6, 2026
e14a481
feat: add quantized model support for CLIP ViT Base Patch32 and updat…
barhanc Mar 6, 2026
3b1936d
docs: update manual docs
barhanc Mar 6, 2026
506f640
docs: add auto-generated docs
barhanc Mar 6, 2026
97e32d6
feat: update models in example apps to use quantized versions by default
barhanc Mar 9, 2026
720b4a5
feat: add modelName property for changed models
barhanc Mar 9, 2026
554208c
docs: update auto-generated docs
barhanc Mar 9, 2026
618a125
fix: update EFFICIENTNET_V2_S_QUANTIZED_MODEL to use CoreML for iOS
barhanc Mar 9, 2026
7f903fd
docs: update inference time and model size
barhanc Mar 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions apps/computer-vision/app/classification/index.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
import Spinner from '../../components/Spinner';
import { getImage } from '../../utils';
import { useClassification, EFFICIENTNET_V2_S } from 'react-native-executorch';
import {
useClassification,
EFFICIENTNET_V2_S_QUANTIZED,
} from 'react-native-executorch';
import { View, StyleSheet, Image, Text, ScrollView } from 'react-native';
import { BottomBar } from '../../components/BottomBar';
import React, { useContext, useEffect, useState } from 'react';
Expand All @@ -13,7 +16,7 @@ export default function ClassificationScreen() {
);
const [imageUri, setImageUri] = useState('');

const model = useClassification({ model: EFFICIENTNET_V2_S });
const model = useClassification({ model: EFFICIENTNET_V2_S_QUANTIZED });
const { setGlobalGenerating } = useContext(GeneratingContext);
useEffect(() => {
setGlobalGenerating(model.isGenerating);
Expand Down
4 changes: 2 additions & 2 deletions apps/computer-vision/app/semantic_segmentation/index.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import Spinner from '../../components/Spinner';
import { BottomBar } from '../../components/BottomBar';
import { getImage } from '../../utils';
import {
DEEPLAB_V3_RESNET50,
DEEPLAB_V3_MOBILENET_V3_LARGE_QUANTIZED,
useSemanticSegmentation,
} from 'react-native-executorch';
import {
Expand Down Expand Up @@ -46,7 +46,7 @@ export default function SemanticSegmentationScreen() {
const { setGlobalGenerating } = useContext(GeneratingContext);
const { isReady, isGenerating, downloadProgress, forward } =
useSemanticSegmentation({
model: DEEPLAB_V3_RESNET50,
model: DEEPLAB_V3_MOBILENET_V3_LARGE_QUANTIZED,
});
const [imageUri, setImageUri] = useState('');
const [imageSize, setImageSize] = useState({ width: 0, height: 0 });
Expand Down
4 changes: 2 additions & 2 deletions apps/computer-vision/app/style_transfer/index.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,15 @@ import { BottomBar } from '../../components/BottomBar';
import { getImage } from '../../utils';
import {
useStyleTransfer,
STYLE_TRANSFER_CANDY,
STYLE_TRANSFER_CANDY_QUANTIZED,
} from 'react-native-executorch';
import { View, StyleSheet, Image } from 'react-native';
import React, { useContext, useEffect, useState } from 'react';
import { GeneratingContext } from '../../context';
import ScreenWrapper from '../../ScreenWrapper';

export default function StyleTransferScreen() {
const model = useStyleTransfer({ model: STYLE_TRANSFER_CANDY });
const model = useStyleTransfer({ model: STYLE_TRANSFER_CANDY_QUANTIZED });
const { setGlobalGenerating } = useContext(GeneratingContext);
useEffect(() => {
setGlobalGenerating(model.isGenerating);
Expand Down
6 changes: 4 additions & 2 deletions apps/text-embeddings/app/clip-embeddings/index.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import {
useTextEmbeddings,
useImageEmbeddings,
CLIP_VIT_BASE_PATCH32_TEXT,
CLIP_VIT_BASE_PATCH32_IMAGE,
CLIP_VIT_BASE_PATCH32_IMAGE_QUANTIZED,
} from 'react-native-executorch';
import { launchImageLibrary } from 'react-native-image-picker';
import { useIsFocused } from '@react-navigation/native';
Expand All @@ -29,7 +29,9 @@ export default function ClipEmbeddingsScreenWrapper() {

function ClipEmbeddingsScreen() {
const textModel = useTextEmbeddings({ model: CLIP_VIT_BASE_PATCH32_TEXT });
const imageModel = useImageEmbeddings({ model: CLIP_VIT_BASE_PATCH32_IMAGE });
const imageModel = useImageEmbeddings({
model: CLIP_VIT_BASE_PATCH32_IMAGE_QUANTIZED,
});

const [inputSentence, setInputSentence] = useState('');
const [sentencesWithEmbeddings, setSentencesWithEmbeddings] = useState<
Expand Down
75 changes: 54 additions & 21 deletions docs/docs/02-benchmarks/inference-time.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,24 +8,41 @@ Times presented in the tables are measured as consecutive runs of the model. Ini

## Classification

| Model | iPhone 17 Pro (Core ML) [ms] | iPhone 16 Pro (Core ML) [ms] | iPhone SE 3 (Core ML) [ms] | Samsung Galaxy S24 (XNNPACK) [ms] | OnePlus 12 (XNNPACK) [ms] |
| ----------------- | :--------------------------: | :--------------------------: | :------------------------: | :-------------------------------: | :-----------------------: |
| EFFICIENTNET_V2_S | 64 | 68 | 217 | 205 | 198 |
:::info
Times presented below are _model inference times only_ and do not include time taken for pre-processing (e.g. image resizing, normalization) or post-processing (e.g. image resizing) which are dependent on input size.
:::

| Model | iPhone 17 Pro (Core ML) [ms] | Google Pixel 10 (XNNPACK) [ms] |
| --------------------------- | :--------------------------: | :----------------------------: |
| EFFICIENTNET_V2_S | 12 | 100 |
| EFFICIENTNET_V2_S_QUANTIZED | 5 | 38 |

## Object Detection

| Model | iPhone 17 Pro (XNNPACK) [ms] | iPhone 16 Pro (XNNPACK) [ms] | iPhone SE 3 (XNNPACK) [ms] | Samsung Galaxy S24 (XNNPACK) [ms] | OnePlus 12 (XNNPACK) [ms] |
| ------------------------------ | :--------------------------: | :--------------------------: | :------------------------: | :-------------------------------: | :-----------------------: |
| SSDLITE_320_MOBILENET_V3_LARGE | 71 | 74 | 257 | 115 | 109 |
:::info
Times presented below are _model inference times only_ and do not include time taken for pre-processing (e.g. image resizing, normalization) or post-processing (e.g. image resizing) which are dependent on input size.
:::

| Model | iPhone 17 Pro (Core ML) [ms] | Google Pixel 10 (XNNPACK) [ms] |
| ------------------------------ | :--------------------------: | :----------------------------: |
| SSDLITE_320_MOBILENET_V3_LARGE | 8 | 18 |

## Style Transfer

| Model | iPhone 17 Pro (Core ML) [ms] | iPhone 16 Pro (Core ML) [ms] | iPhone SE 3 (Core ML) [ms] | Samsung Galaxy S24 (XNNPACK) [ms] | OnePlus 12 (XNNPACK) [ms] |
| ---------------------------- | :--------------------------: | :--------------------------: | :------------------------: | :-------------------------------: | :-----------------------: |
| STYLE_TRANSFER_CANDY | 1400 | 1485 | 4255 | 2510 | 2355 |
| STYLE_TRANSFER_MOSAIC | 1400 | 1485 | 4255 | 2510 | 2355 |
| STYLE_TRANSFER_UDNIE | 1400 | 1485 | 4255 | 2510 | 2355 |
| STYLE_TRANSFER_RAIN_PRINCESS | 1400 | 1485 | 4255 | 2510 | 2355 |
:::info
Times presented below are _model inference times only_ and do not include time taken for pre-processing (e.g. image resizing, normalization) or post-processing (e.g. image resizing) which are dependent on input size.
:::

| Model | iPhone 17 Pro (Core ML) [ms] | Google Pixel 10 (XNNPACK) [ms] |
| -------------------------------------- | :--------------------------: | :----------------------------: |
| STYLE_TRANSFER_CANDY | 100 | 1025 |
| STYLE_TRANSFER_MOSAIC | 100 | 1025 |
| STYLE_TRANSFER_UDNIE | 100 | 1025 |
| STYLE_TRANSFER_RAIN_PRINCESS | 100 | 1025 |
| STYLE_TRANSFER_CANDY_QUANTIZED | 150 | 430 |
| STYLE_TRANSFER_MOSAIC_QUANTIZED | 150 | 430 |
| STYLE_TRANSFER_UDNIE_QUANTIZED | 150 | 430 |
| STYLE_TRANSFER_RAIN_PRINCESS_QUANTIZED | 150 | 430 |

## OCR

Expand Down Expand Up @@ -109,23 +126,39 @@ Benchmark times for text embeddings are highly dependent on the sentence length.

## Image Embeddings

| Model | iPhone 17 Pro (XNNPACK) [ms] | OnePlus 12 (XNNPACK) [ms] |
| --------------------------- | :--------------------------: | :-----------------------: |
| CLIP_VIT_BASE_PATCH32_IMAGE | 18 | 55 |
:::info
Times presented below are _model inference times only_ and do not include time taken for pre-processing (e.g. image resizing, normalization) or post-processing (e.g. image resizing) which are dependent on input size.
:::

:::info
Image embedding benchmark times are measured using 224×224 pixel images, as required by the model. All input images, whether larger or smaller, are resized to 224×224 before processing. Resizing is typically fast for small images but may be noticeably slower for very large images, which can increase total inference time.
Image embedding benchmark times are measured using 224×224 pixel images, as required by the model. All input images, whether larger or smaller, are resized to 224×224 before processing. Resizing is typically fast for small images but may be noticeably slower for very large images, which can increase total time.
:::

| Model | iPhone 17 Pro (XNNPACK) [ms] | Google Pixel 10 (XNNPACK) [ms] |
| ------------------------------------- | :--------------------------: | :----------------------------: |
| CLIP_VIT_BASE_PATCH32_IMAGE | 14 | 68 |
| CLIP_VIT_BASE_PATCH32_IMAGE_QUANTIZED | 11 | 31 |

## Semantic Segmentation

:::warning
Times presented in the tables are measured as consecutive runs of the model. Initial run times may be up to 2x longer due to model loading and initialization.
:::info
Times presented below are _model inference times only_ and do not include time taken for pre-processing (e.g. image resizing, normalization) or post-processing (e.g. image resizing) which are dependent on input size.
:::

| Model | iPhone 16 Pro (Core ML) [ms] | iPhone 14 Pro Max (Core ML) [ms] | Samsung Galaxy S24 (XNNPACK) [ms] |
| ----------------- | ---------------------------- | -------------------------------- | --------------------------------- |
| DEELABV3_RESNET50 | 1000 | 670 | 700 |
| Model | iPhone 17 Pro (XNNPACK) [ms] | Google Pixel 10 (XNNPACK) [ms] |
| --------------------------------------- | :--------------------------: | :----------------------------: |
| DEEPLAB_V3_RESNET50 | 2000 | 2200 |
| DEEPLAB_V3_RESNET50_QUANTIZED | 118 | 380 |
| DEEPLAB_V3_RESNET101 | 2900 | 3300 |
| DEEPLAB_V3_RESNET101_QUANTIZED | 174 | 660 |
| DEEPLAB_V3_MOBILENET_V3_LARGE | 131 | 153 |
| DEEPLAB_V3_MOBILENET_V3_LARGE_QUANTIZED | 17 | 40 |
| LRASPP_MOBILENET_V3_LARGE | 13 | 36 |
| LRASPP_MOBILENET_V3_LARGE_QUANTIZED | 12 | 20 |
| FCN_RESNET50 | 1800 | 2160 |
| FCN_RESNET50_QUANTIZED | 100 | 320 |
| FCN_RESNET101 | 2600 | 3160 |
| FCN_RESNET101_QUANTIZED | 160 | 620 |

## Text to image

Expand Down
41 changes: 23 additions & 18 deletions docs/docs/02-benchmarks/model-size.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,24 +4,24 @@ title: Model Size

## Classification

| Model | XNNPACK [MB] | Core ML [MB] |
| ----------------- | :----------: | :----------: |
| EFFICIENTNET_V2_S | 85.6 | 43.9 |
| Model | XNNPACK FP32 [MB] | XNNPACK INT8 [MB] | Core ML FP32 [MB] | Core ML FP16 [MB] |
| :---------------- | :---------------: | :---------------: | :---------------: | :---------------: |
| EFFICIENTNET_V2_S | 85.7 | 22.9 | 86.5 | 43.9 |

## Object Detection

| Model | XNNPACK [MB] |
| ------------------------------ | :----------: |
| SSDLITE_320_MOBILENET_V3_LARGE | 13.9 |
| Model | XNNPACK FP32 [MB] | Core ML FP32 [MB] | Core ML FP16 [MB] |
| ------------------------------ | :---------------: | :---------------: | :---------------: |
| SSDLITE_320_MOBILENET_V3_LARGE | 13.9 | 15.6 | 8.46 |

## Style Transfer

| Model | XNNPACK [MB] | Core ML [MB] |
| ---------------------------- | :----------: | :----------: |
| STYLE_TRANSFER_CANDY | 6.78 | 5.22 |
| STYLE_TRANSFER_MOSAIC | 6.78 | 5.22 |
| STYLE_TRANSFER_UDNIE | 6.78 | 5.22 |
| STYLE_TRANSFER_RAIN_PRINCESS | 6.78 | 5.22 |
| Model | XNNPACK FP32 [MB] | XNNPACK INT8 [MB] | Core ML FP32 [MB] | Core ML FP16 [MB] |
| ---------------------------- | :---------------: | :---------------: | :---------------: | :---------------: |
| STYLE_TRANSFER_CANDY | 6.82 | 1.84 | 7.12 | 3.79 |
| STYLE_TRANSFER_MOSAIC | 6.82 | 1.84 | 7.12 | 3.79 |
| STYLE_TRANSFER_UDNIE | 6.82 | 1.84 | 7.12 | 3.79 |
| STYLE_TRANSFER_RAIN_PRINCESS | 6.82 | 1.84 | 7.12 | 3.79 |

## OCR

Expand Down Expand Up @@ -82,15 +82,20 @@ title: Model Size

## Image Embeddings

| Model | XNNPACK [MB] |
| --------------------------- | :----------: |
| CLIP_VIT_BASE_PATCH32_IMAGE | 352 |
| Model | XNNPACK FP32 [MB] | XNNPACK INT8 [MB] |
| --------------------------- | :---------------: | :---------------: |
| CLIP_VIT_BASE_PATCH32_IMAGE | 352 | 96.4 |

## Semantic Segmentation

| Model | XNNPACK [MB] |
| ----------------- | ------------ |
| DEELABV3_RESNET50 | 168 |
| Model | XNNPACK FP32 [MB] | XNNPACK INT8 [MB] |
| ----------------------------- | :---------------: | :---------------: |
| DEEPLAB_V3_RESNET50 | 168 | 42.4 |
| DEEPLAB_V3_RESNET101 | 244 | 61.7 |
| DEEPLAB_V3_MOBILENET_V3_LARGE | 44.1 | 11.4 |
| LRASPP_MOBILENET_V3_LARGE | 12.9 | 3.53 |
| FCN_RESNET50 | 141 | 35.7 |
| FCN_RESNET101 | 217 | 55 |

## Text to image

Expand Down
6 changes: 3 additions & 3 deletions docs/docs/03-hooks/02-computer-vision/useClassification.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,6 @@ function App() {

## Supported models

| Model | Number of classes | Class list |
| ------------------------------------------------------------------------------------------------------ | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [efficientnet_v2_s](https://huggingface.co/software-mansion/react-native-executorch-efficientnet-v2-s) | 1000 | [ImageNet1k_v1](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/models/classification/Constants.h) |
| Model | Number of classes | Class list | Quantized |
| ------------------------------------------------------------------------------------------------------ | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------: |
| [efficientnet_v2_s](https://huggingface.co/software-mansion/react-native-executorch-efficientnet-v2-s) | 1000 | [ImageNet1k_v1](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/models/classification/Constants.h) | Yes |
6 changes: 6 additions & 0 deletions docs/docs/06-api-reference/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,12 @@
## Models - Classification

- [EFFICIENTNET_V2_S](variables/EFFICIENTNET_V2_S.md)
- [EFFICIENTNET_V2_S_QUANTIZED](variables/EFFICIENTNET_V2_S_QUANTIZED.md)

## Models - Image Embeddings

- [CLIP_VIT_BASE_PATCH32_IMAGE](variables/CLIP_VIT_BASE_PATCH32_IMAGE.md)
- [CLIP_VIT_BASE_PATCH32_IMAGE_QUANTIZED](variables/CLIP_VIT_BASE_PATCH32_IMAGE_QUANTIZED.md)

## Models - Image Generation

Expand Down Expand Up @@ -106,9 +108,13 @@
## Models - Style Transfer

- [STYLE_TRANSFER_CANDY](variables/STYLE_TRANSFER_CANDY.md)
- [STYLE_TRANSFER_CANDY_QUANTIZED](variables/STYLE_TRANSFER_CANDY_QUANTIZED.md)
- [STYLE_TRANSFER_MOSAIC](variables/STYLE_TRANSFER_MOSAIC.md)
- [STYLE_TRANSFER_MOSAIC_QUANTIZED](variables/STYLE_TRANSFER_MOSAIC_QUANTIZED.md)
- [STYLE_TRANSFER_RAIN_PRINCESS](variables/STYLE_TRANSFER_RAIN_PRINCESS.md)
- [STYLE_TRANSFER_RAIN_PRINCESS_QUANTIZED](variables/STYLE_TRANSFER_RAIN_PRINCESS_QUANTIZED.md)
- [STYLE_TRANSFER_UDNIE](variables/STYLE_TRANSFER_UDNIE.md)
- [STYLE_TRANSFER_UDNIE_QUANTIZED](variables/STYLE_TRANSFER_UDNIE_QUANTIZED.md)

## Models - Text Embeddings

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/06-api-reference/typedoc-sidebar.cjs
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/docs/06-api-reference/variables/ALL_MINILM_L6_V2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

> `const` **ALL_MINILM_L6_V2**: `object`
Defined in: [constants/modelUrls.ts:695](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/constants/modelUrls.ts#L695)
Defined in: [constants/modelUrls.ts:773](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/constants/modelUrls.ts#L773)

## Type Declaration

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/06-api-reference/variables/ALL_MPNET_BASE_V2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

> `const` **ALL_MPNET_BASE_V2**: `object`
Defined in: [constants/modelUrls.ts:703](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/constants/modelUrls.ts#L703)
Defined in: [constants/modelUrls.ts:781](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/constants/modelUrls.ts#L781)

## Type Declaration

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/06-api-reference/variables/BK_SDM_TINY_VPRED_256.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

> `const` **BK_SDM_TINY_VPRED_256**: `object`
Defined in: [constants/modelUrls.ts:748](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/constants/modelUrls.ts#L748)
Defined in: [constants/modelUrls.ts:827](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/constants/modelUrls.ts#L827)

## Type Declaration

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/06-api-reference/variables/BK_SDM_TINY_VPRED_512.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

> `const` **BK_SDM_TINY_VPRED_512**: `object`
Defined in: [constants/modelUrls.ts:737](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/constants/modelUrls.ts#L737)
Defined in: [constants/modelUrls.ts:816](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/constants/modelUrls.ts#L816)

## Type Declaration

Expand Down
Loading
Loading