[Experimental][TorchFX] quantize_pt2e + X86Quantizer introduction #3121

daniil-lyakhov · 2024-11-28T16:40:09Z

Changes

Introduction of quantize_pt2e method

Reason for changes

Related tickets

#2766

Tests

graph tests: tests/torch/fx/test_quantizer.py

nncf/experimental/common/quantization/algorithms/post_training/algorithm.py

nncf/quantization/algorithms/min_max/algorithm.py

nncf/experimental/torch/fx/quantization/quantize_pt2e.py

nncf/experimental/common/quantization/algorithms/quantizer/base_quantizer.py

nncf/experimental/common/quantization/algorithms/quantizer/fx_quantizer.py

nncf/experimental/torch/fx/quantization/quantize_pt2e.py

nncf/experimental/common/quantization/algorithms/post_training/algorithm.py

nncf/experimental/common/quantization/algorithms/range_estimator/range_estimator.py

nncf/quantization/algorithms/min_max/algorithm.py

nncf/experimental/torch/fx/quantization/quantize_pt2e.py

alexsu52

@daniil-lyakhov, I would suggest to create an example how to use nncf.torch.quantize_pt2e for torch.ao.Quantizer. Please open a ticket for this task if it was not open yet.

alexsu52 · 2025-01-08T06:08:32Z

nncf/experimental/quantization/algorithms/quantizer/torch_ao_adapter.py

+        self._quantizer = quantizer
+
+    def get_quantization_setup(self, model: torch.fx.GraphModule, nncf_graph: NNCFGraph) -> SingleConfigQuantizerSetup:
+        anotated_model = deepcopy(model)


Could you briefly explain why you are doing deep copying of the model here?

That's because the .anotate and .validate methods alter the model meta. I believe get_quantization_setup metho d should not change the input model in any way

alexsu52 · 2025-01-08T06:23:25Z

nncf/experimental/quantization/algorithms/quantizer/__init__.py

@@ -0,0 +1,10 @@
+# Copyright (c) 2024 Intel Corporation


I would sugget to move nncf/experimental/quantization/algorithms/quantizer -> nncf/experimental/quantization/quantizer

Done. Additionally, I renamed base_quantizer.py -> quantizer.py

alexsu52 · 2025-01-08T06:38:11Z

nncf/experimental/quantization/algorithms/quantizer/torch_ao_adapter.py

+                    per_channel = False
+                else:
+                    raise nncf.InternalError(f"Unknown qscheme: {qspec.qscheme}")
+                signed = qspec.dtype is torch.uint8


Please, double check this condition. I believe signed=True if torch.int8.

This line is incorrect, you are right! But this parameter does not affect the quantization parameters as signdness_to_force is ignored in PTQ min max here https://github.com/openvinotoolkit/nncf/blob/develop/nncf/quantization/algorithms/min_max/torch_fx_backend.py#L230

alexsu52 · 2025-01-08T06:47:01Z

nncf/experimental/quantization/algorithms/quantizer/torch_ao_adapter.py

+                qconfig = QuantizerConfig(mode=mode, signedness_to_force=signed, per_channel=per_channel)
+                qps = []
+                # If input node is a constant and placed not at activations port (0)
+                if from_n.op == "get_attr" and to_n.args.index(from_n) != 0:


As far as I understand, WeightQuantizationInsertionPoint differs from ActivationQuantizationInsertionPoint in the method of collecting statistics, am I right? Then why did you add the activation port checking?

I have refactored it, now it should works in common way. The problem which is fixed there is that for some models (swin_v2_s for example) some activations could be a constants as well

alexsu52 · 2025-01-08T06:48:58Z

nncf/experimental/quantization/algorithms/quantizer/torch_ao_adapter.py

+                    q_setup.add_independent_quantization_point(qp)
+
+            elif isinstance(qspec, SharedQuantizationSpec):
+                pass


Will the support of the SharedQuantizationSpec be added in the follow-up PR? If so, please add a todo here.

SharedQuantizationSpec is not produced by the X86InductorQuantizer, but could be produced by a different one. Warning and a todo string was added

alexsu52 · 2025-01-08T06:50:46Z

nncf/experimental/quantization/algorithms/quantizer/base_quantizer.py

+TModel = TypeVar("TModel")
+
+
+class Quantizer:


Please, add a doctring for the class.

alexsu52 · 2025-01-08T06:51:03Z

nncf/experimental/quantization/algorithms/quantizer/torch_ao_adapter.py

+EdgeOrNode = Union[Tuple[torch.fx.Node, torch.fx.Node]]
+
+
+class TorchAOQuantizerAdapter(NNCFQuantizer):


Please, add a docsting for the class

alexsu52 · 2025-01-08T06:52:04Z

nncf/experimental/quantization/algorithms/range_estimator/algorithm.py

+        weights_range_estimator_params: Optional[RangeEstimatorParameters] = None,
+    ):
+        """
+        :param subset_size: Size of a subset to calculate activations statistics used


The docstring for the quantizer parameter is missed.

The gap is filled, thanks

daniil-lyakhov · 2025-01-09T14:30:45Z

@daniil-lyakhov, I would suggest to create an example how to use nncf.torch.quantize_pt2e for torch.ao.Quantizer. Please open a ticket for this task if it was not open yet.

ticket: #3185

github-actions bot added NNCF PT Pull requests that updates NNCF PyTorch experimental NNCF PTQ Pull requests that updates NNCF PTQ labels Nov 28, 2024

daniil-lyakhov changed the title ~~Dl/fx/experimental quantization~~ [Experimental][TorchFX] quantize_pt2e + X86Quantizer introduction Nov 28, 2024

daniil-lyakhov force-pushed the dl/fx/experimental_quantization branch 2 times, most recently from efd3367 to d1941f3 Compare November 28, 2024 17:32

github-actions bot added the NNCF Common Pull request that updates NNCF Common label Dec 2, 2024

daniil-lyakhov force-pushed the dl/fx/experimental_quantization branch 2 times, most recently from aea0bdf to 52e80c8 Compare December 4, 2024 09:59

daniil-lyakhov requested review from alexsu52 and anzr299 December 4, 2024 12:16

daniil-lyakhov marked this pull request as ready for review December 4, 2024 12:17

daniil-lyakhov requested a review from a team as a code owner December 4, 2024 12:17

anzr299 reviewed Dec 4, 2024

View reviewed changes

nncf/experimental/common/quantization/algorithms/post_training/algorithm.py Outdated Show resolved Hide resolved

daniil-lyakhov requested a review from anzr299 December 5, 2024 10:45

daniil-lyakhov force-pushed the dl/fx/experimental_quantization branch 2 times, most recently from 9178921 to 43bc251 Compare December 5, 2024 12:32

alexsu52 requested changes Dec 23, 2024

View reviewed changes

daniil-lyakhov force-pushed the dl/fx/experimental_quantization branch 3 times, most recently from 7ede33d to 20147ab Compare December 23, 2024 22:15

alexsu52 reviewed Dec 24, 2024

View reviewed changes

nncf/quantization/algorithms/min_max/algorithm.py Outdated Show resolved Hide resolved

nncf/experimental/torch/fx/quantization/quantize_pt2e.py Outdated Show resolved Hide resolved

alexsu52 reviewed Dec 24, 2024

View reviewed changes

nncf/experimental/torch/fx/quantization/quantize_pt2e.py Outdated Show resolved Hide resolved

nncf/experimental/torch/fx/quantization/quantize_pt2e.py Outdated Show resolved Hide resolved

daniil-lyakhov force-pushed the dl/fx/experimental_quantization branch from f62e0b8 to 38c82a4 Compare December 29, 2024 18:27

daniil-lyakhov added 6 commits January 7, 2025 16:25

WIP experimental quantization

df4125b

Experimental quantization

bf6d2df

Reuse MinMax algo instead of copy-paste

be24462

Correct use of transform_for_annotation

af0c19e

Comments/fixes

160ddd4

batchwise_statistics

869bc8f

daniil-lyakhov added 2 commits January 7, 2025 16:25

Comments

0d1e14f

Comments

4891184

daniil-lyakhov force-pushed the dl/fx/experimental_quantization branch from 38c82a4 to 206f606 Compare January 7, 2025 16:53

daniil-lyakhov requested a review from alexsu52 January 7, 2025 17:01

alexsu52 reviewed Jan 8, 2025

View reviewed changes

Code migrated to adapters/ comments

b923176

daniil-lyakhov force-pushed the dl/fx/experimental_quantization branch from 206f606 to 983f5fa Compare January 8, 2025 14:19

daniil-lyakhov requested a review from alexsu52 January 8, 2025 14:20

daniil-lyakhov force-pushed the dl/fx/experimental_quantization branch from 983f5fa to e9bc7a8 Compare January 8, 2025 14:23

daniil-lyakhov added 2 commits January 8, 2025 15:28

Adapter refactoring/ comments

e9bc7a8

2024 -> 2025

93e808b

daniil-lyakhov mentioned this pull request Jan 9, 2025

[Example][TorchFX] Example on quantize_pt2e with the X86InductorQuantizer #3185

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Experimental][TorchFX] quantize_pt2e + X86Quantizer introduction #3121

[Experimental][TorchFX] quantize_pt2e + X86Quantizer introduction #3121

daniil-lyakhov commented Nov 28, 2024 •

edited

Loading

alexsu52 left a comment

alexsu52 Jan 8, 2025

daniil-lyakhov Jan 8, 2025 •

edited

Loading

alexsu52 Jan 8, 2025

daniil-lyakhov Jan 8, 2025

alexsu52 Jan 8, 2025

daniil-lyakhov Jan 8, 2025

daniil-lyakhov Jan 8, 2025

alexsu52 Jan 8, 2025

daniil-lyakhov Jan 8, 2025

alexsu52 Jan 8, 2025

daniil-lyakhov Jan 8, 2025 •

edited

Loading

alexsu52 Jan 8, 2025

daniil-lyakhov Jan 8, 2025

alexsu52 Jan 8, 2025

daniil-lyakhov Jan 8, 2025

alexsu52 Jan 8, 2025

daniil-lyakhov Jan 8, 2025

daniil-lyakhov commented Jan 9, 2025

		EdgeOrNode = Union[Tuple[torch.fx.Node, torch.fx.Node]]


		class TorchAOQuantizerAdapter(NNCFQuantizer):

[Experimental][TorchFX] quantize_pt2e + X86Quantizer introduction #3121

Are you sure you want to change the base?

[Experimental][TorchFX] quantize_pt2e + X86Quantizer introduction #3121

Conversation

daniil-lyakhov commented Nov 28, 2024 • edited Loading

Changes

Reason for changes

Related tickets

Tests

alexsu52 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daniil-lyakhov Jan 8, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daniil-lyakhov Jan 8, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daniil-lyakhov commented Jan 9, 2025

daniil-lyakhov commented Nov 28, 2024 •

edited

Loading

daniil-lyakhov Jan 8, 2025 •

edited

Loading

daniil-lyakhov Jan 8, 2025 •

edited

Loading