Changing the hashing methodology for cache folder creation of models. #481

quic-dhirajku · 2025-06-24T08:30:42Z

Detaching hash function for model cache path calculation. changes for QNN compilation not included yet.

Cache folder mechanism has been modified to have a parent directory for a model based on the architecture that we retrieve from the model config. The hash calculation for the ONNX export now incorporates all model kwargs as well as export kwargs and parameters. the parameters that were used to create the hash also gets dumped as a serialized JSON file in the ONNX folder, the same happens for the compile parameters inside the respective qpc folder.

ochougul

review WIP.

ochougul · 2025-06-24T10:23:04Z

QEfficient/base/modeling_qeff.py

@@ -5,7 +5,7 @@
 #
 # ----------------------------------------------------------------------------

-import hashlib
+# import hashlib


commented code.
Make sure that commented code is not there in ready to review PRs.

ochougul · 2025-06-24T14:08:58Z

QEfficient/base/modeling_qeff.py

+        self.model_params.update(kwargs)
+        self.model_params["config"] = self.model.config.to_diff_dict()
+        self.model_params["_transform_names"] = self._transform_names()
+        self.compile_params = {}


initialize this only when compile is called.
No point in creating this dictionary if user not calling compile.

ochougul · 2025-06-24T14:10:06Z

QEfficient/base/modeling_qeff.py

+        self.model_params = {}
+        self.model_params.update(kwargs)


Better to do self.model_params = copy.deepcopy(kwargs)
This lets other methods mutate kwargs.
Otherwise we would need to ensure that no other method mutates the kwargs.

ochougul · 2025-06-24T14:13:11Z

QEfficient/base/modeling_qeff.py

+        if export_kwargs is not None:
+            self.model_params.update(export_kwargs)
+        if onnx_transform_kwargs is not None:
+            self.model_params.update(onnx_transform_kwargs)


One liners are better

self.model_params.update(export_kwargs) if export_kwargs is not None else None self.model_params.update(onnx_transform_kwargs) if export_kwargs is not None else None

ochougul · 2025-06-24T14:14:42Z

QEfficient/base/modeling_qeff.py

+        self.model_params["output_names"] = output_names
+        self.model_params["dynamic_axes"] = dynamic_axes


Better to keep them in one more level as
self.model_params["export_params"] = export_params
And add all exports params in export_params which is another dict.

Makes the dumped JSON readable by user.

ochougul · 2025-06-24T14:16:12Z

QEfficient/base/modeling_qeff.py

+        model_params_json = export_dir / "model_params.json"
+        with open(model_params_json, "w") as fp:
+            json.dump(
+                {
+                    "model_params": [
+                        {k: make_serializable(self.model_params[k]) for k in sorted(self.model_params.keys())}
+                    ]
+                },
+                fp,
+                indent=4,
+            )


Dumping should happen after export.
If model errors out during export and we still dump the json, it does not make sense

ochougul · 2025-06-24T14:16:40Z

QEfficient/transformers/models/modeling_auto.py


-        self.pretrained_model_name_or_path = kwargs.get("pretrained_model_name_or_path", None)
+        # self.pretrained_model_name_or_path = kwargs.get("pretrained_model_name_or_path", None)


quic-amitraj

To control the user should not initialize the model directly using the init of modeling class. Please use Metaclass Control to control the flow and print the warning or error. For example-

class NoInitMeta(type):
    def __call__(cls, *args, **kwargs):
        raise RuntimeError("Use `from_pretrained` to create an instance.")

class MyModel(metaclass=NoInitMeta):
    def __init__(self, data):
        self.data = data

    @classmethod
    def from_pretrained(cls, path):
        instance = object.__new__(cls)
        instance.__init__(f"Loaded from {path}")
        return instance

More about this you can read from here- https://stackoverflow.com/questions/100003/what-are-metaclasses-in-python.
Put that meta class in the utils and use this for all the modelling class.

quic-amitraj · 2025-07-02T05:33:09Z

QEfficient/base/modeling_qeff.py

@@ -357,6 +388,19 @@ def _compile(
        logger.info(f"Running compiler: {' '.join(command)}")
        try:
            subprocess.run(command, capture_output=True, check=True)
+
+            # Dumping compile paramters in a JSON file after successful QPC compilation


Remove all the code related to compile_param_json from here including dumping and handle these inside the decorator dump_qconfig. Lets keep the base methods clean.

There is a typo in "paramters" line 391

Define a method for creating compile/export params_json in utils, use the same method for creating the export and compiler json and dump using dump_json method.

quic-amitraj · 2025-07-02T05:35:17Z

QEfficient/base/modeling_qeff.py

+            model_params_json = export_dir / "model_params.json"
+            with open(model_params_json, "w") as fp:
+                json.dump(
+                    {


Same like compile create a decorator and handle all these param updates and dumping inside that.

Also use the dump_json from the utils.

quic-amitraj · 2025-07-02T05:35:56Z

QEfficient/base/modeling_qeff.py

+        export_params["output_names"] = output_names
+        export_params["dynamic_axes"] = dynamic_axes
+
+        self.model_params["export_params"] = export_params


Handle in decorator. Lets keep our base methods clean.

Agree, lets write decorator implementation to handle this.

Do we need a decorator here? how we would be able to return the hash from the decorator and construct the export dir?

Wouldnt it be better if we can move the hash creation to a separate function and call it from here?

quic-rishinr · 2025-07-11T06:49:02Z

QEfficient/base/modeling_qeff.py

+
+        # Store Model parameters to Calculate Hash for caching
+        self.model_params = {}
+        self.model_params = copy.deepcopy(kwargs)


Do we need self.model_params = {}?

move the updating model_params to a method. it keeps the init clean

quic-rishinr · 2025-07-11T07:52:42Z

QEfficient/base/modeling_qeff.py

+        export_params["output_names"] = output_names
+        export_params["dynamic_axes"] = dynamic_axes
+
+        self.model_params["export_params"] = export_params


Do we need a decorator here? how we would be able to return the hash from the decorator and construct the export dir?

Wouldnt it be better if we can move the hash creation to a separate function and call it from here?

quic-rishinr · 2025-07-11T08:57:14Z

QEfficient/base/modeling_qeff.py

+            model_params_json = export_dir / "model_params.json"
+            with open(model_params_json, "w") as fp:
+                json.dump(
+                    {


Also use the dump_json from the utils.

quic-rishinr · 2025-07-11T09:02:19Z

QEfficient/base/modeling_qeff.py

@@ -357,6 +388,19 @@ def _compile(
        logger.info(f"Running compiler: {' '.join(command)}")
        try:
            subprocess.run(command, capture_output=True, check=True)
+
+            # Dumping compile paramters in a JSON file after successful QPC compilation


There is a typo in "paramters" line 391

Define a method for creating compile/export params_json in utils, use the same method for creating the export and compiler json and dump using dump_json method.

QEfficient/transformers/models/modeling_auto.py

quic-rishinr · 2025-07-31T04:59:54Z

QEfficient/base/modeling_qeff.py

+        model_params = copy.deepcopy(kwargs)
+
+        model_params["config"] = self.model.config.to_diff_dict()
+        model_params["_transform_names"] = self._transform_names()


you can make it as applied transforms instead of "_transform_names", Also move this method below init

quic-rishinr · 2025-07-31T05:04:59Z

QEfficient/base/modeling_qeff.py

+        if hasattr(self.model.config, "architectures"):
+            self.model_architecture = self.model.config.architectures[0]


This would raise an attribute error if config doesnt have architectures. instead you can use
self.model_architecture = getattr(self.model.config, "architectures", [None])[0]

quic-rishinr · 2025-07-31T05:09:09Z

QEfficient/base/modeling_qeff.py

+            dynamic_axes=dynamic_axes,
+            export_kwargs=export_kwargs,
+            onnx_transform_kwargs=onnx_transform_kwargs,
+            export_dir=export_dir,


Why do we need export dir in hash params?

quic-rishinr · 2025-07-31T05:14:24Z

QEfficient/base/modeling_qeff.py

@@ -210,6 +236,11 @@ def _export(
        finally:
            shutil.rmtree(tmp_onnx_dir, ignore_errors=True)

+        # Dump JSON file with hashed parameters
+        hashed_params_export_path = export_dir / "hashed_model_params.json"


(Suggestion) would it be better if we name it as export_model_params or something like this? Since we have compile_model_params as well

QEfficient/base/modeling_qeff.py

quic-rishinr · 2025-07-31T05:17:42Z

QEfficient/base/modeling_qeff.py

        # Check if already compiled
-        compile_hash = compile_hash.hexdigest()[:16]
+        compile_hash, hashed_params = hash_compile_params(


remove the above comment as its not needed here " # Check if already compiled"

quic-rishinr · 2025-07-31T05:31:20Z

QEfficient/transformers/models/modeling_auto.py


        # Make Embedding specific transforms like appending pooling
        if pooling:
            self.model, _ = PoolingTransform.apply(self.model, pooling)

        self.model.base_model.config.use_cache = True
-
-        self.pretrained_model_name_or_path = kwargs.get("pretrained_model_name_or_path", None)
+        self.hash_params["qeff_class"] = self.__class__.__name__


nit: which would be better to better readability "qeff_class" or "qeff_auto_class"?

quic-rishinr · 2025-07-31T05:35:59Z

QEfficient/utils/cache.py

@@ -39,3 +43,11 @@ def to_hashable(obj) -> bytes:
        default=json_serializable,
        sort_keys=True,
    ).encode()
+
+
+def hash_dict_params(dict_items: Dict, hash_string_size: int = HASH_HEXDIGEST_STR_LEN):


it would be better to create a new python file in utils and maintain all hash related modules there.

… QNN compilation not included yet. Cache folder mechanism has been modified to have a parent directory for a model based on the architecture that we retrieve from the model config. The hash calculation for the ONNX export now incorporates all model kwargs as well as export kwargs and parameters. the parameters that were used to create the hash also gets dumped as a serialized JSON file in the ONNX folder, the same happens for the compile parameters inside the respective qpc folder. Signed-off-by: Dhiraj Kumar Sah <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

This fixed the issue with higher BS compilation for SwiftKV models ``` Compiler command: ['/opt/qti-aic/exec/qaic-exec', '-aic-hw', '-aic-hw-version=2.0', '-m=/prj/qct/aisyssol_scratch/users/shagsood/quic_shagun/LlamaSwiftKVForCausalLM-a5879ebc0e59ab40/LlamaSwiftKVForCausal LM.onnx', '-compile-only', '-retained-state', '-convert-to-fp16', '-aic-num-cores=16', '-network-specialization-config=/prj/qct/aisyssol_scratch/users/shagsood/quic_shagun/LlamaSwiftKVForCausalLM-a5879eb c0e59ab40/qpc-60f86f912a187346/specializations.json', '-custom-IO-list-file=/prj/qct/aisyssol_scratch/users/shagsood/quic_shagun/LlamaSwiftKVForCausalLM-a5879ebc0e59ab40/qpc-60f86f912a187346/custom_io.ya ml', '-mdp-load-partition-config=/prj/qct/aisyssol_scratch/users/shagsood/quic_shagun/LlamaSwiftKVForCausalLM-a5879ebc0e59ab40/qpc-60f86f912a187346/mdp_ts_4.json', '-aic-binary-dir=/prj/qct/aisyssol_scra tch/users/shagsood/quic_shagun/LlamaSwiftKVForCausalLM-a5879ebc0e59ab40/qpc-60f86f912a187346/qpc'] Compiler exitcode: 1 Compiler stderr: QAIC_ERROR: Error message: [Operator-'/model/layers.16/self_attn/Reshape'] : Reshape: input shape (4, 4, 4096) and output shape (4, 1, 32, 128) have different number of elements (in 65536 vs. out 16384) Unable to AddNodesToGraphFromModel ``` Tested with BS4. Able to compile now Signed-off-by: quic-shagun <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

CI enablement and other minor fixes for Gemma3 --------- Signed-off-by: Ann Kuruvilla <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Added fix for spdtransform due to change in hash --------- Signed-off-by: Dipankar Sarkar <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

- Enabled CI tests for Finetuning. - Updated Jenkins file to install torch_qaic as it is required during FT tests. - Added finetune as a new pytest flag and updated other existing tests not to trigger for this flag. --------- Signed-off-by: meetkuma <[email protected]> Co-authored-by: Meet Patel <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

CI enablement and other minor fixes for Gemma3 --------- --------- Signed-off-by: Ann Kuruvilla <[email protected]> Signed-off-by: Dipankar Sarkar <[email protected]> Co-authored-by: Dipankar Sarkar <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Reverts quic#484 Signed-off-by: Dhiraj Kumar Sah <[email protected]>

…boarded features in docs (quic#423) This PR is created for updating the readme and docs for adding the latest features added in this release. --------- Signed-off-by: Abukhoyer Shaik <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

…next file. (quic#475) Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Signed-off-by: Rishin Raj <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Padding the dataset with dummy samples (they won't contribute in total_loss) to make the #samples a multiple of degree of ddp*batch_size) in case of 1) Fine tuning through DDP 2) train_batch_size > 1 or val_batch_size > 0 --------- Signed-off-by: Swati Allabadi <[email protected]> Co-authored-by: Swati Allabadi <[email protected]> Co-authored-by: Mamta Singh <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Generating data format config file fails for encoder onnx graph without past key or past value. Fixed a coding bug in the function. --------- Signed-off-by: Shubham Agrawal <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Changed Total (E2E) inference time from decode/sec to sec. Signed-off-by: Asmita Goswami <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

…ng optimizer step only. (quic#477) Disabling gradient is necessary when using gradient_accumulation_step > 1 with ddp enabled. Currently, we are syncing gradient at every loss.backward() call, which is called at all steps. When using gradient accumulation, the weight update during opt.step() step. Only during that step, the gradients across each devices should sync with each other. with model.no_sync() --> context manager solves this issue. Here, we are not using it but instead setting ddp_model.require_backward_grad_sync to True or False depending on which step we are. --------- Signed-off-by: Meet Patel <[email protected]> Signed-off-by: meetkuma <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

…uic#371) 1. Implement logger for finetuning 2. enable dumping logs by given flag --------- Signed-off-by: Mamta Singh <[email protected]> Co-authored-by: Mamta Singh <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Falcon Modeling fix to accommodate multiple config. This is a fix for falcon 40b Signed-off-by: Dipankar Sarkar <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

…ic#482) - Removed all the references of samsum dataset from finetuning code. - Samsum dataset can be used via custom dataset path. --------- Signed-off-by: Meet Patel <[email protected]> Signed-off-by: meetkuma <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Signed-off-by: Rishin <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Upgrading onnx , onnxruntime ,onnscript and protobuff. Also Updating transformer to 4.52.3 1.onnx==1.18.0 2.onnxruntime==1.22 3.onnxscript==0.2.5 4. protobuff ==6.31.0 --------- Signed-off-by: Dipankar Sarkar <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

1. fix task_type variable in configs 2. enabled passing peft_config yaml/json file from command line. 3. updated run_ft_model.py --------- Signed-off-by: Mamta Singh <[email protected]> Co-authored-by: Mamta Singh <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Signed-off-by: Dhiraj Kumar Sah <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Signed-off-by: Dhiraj Kumar Sah <[email protected]>

…or export Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Functions made to filter and modify hashes. Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Signed-off-by: Dhiraj Kumar Sah <[email protected]>

…m_pretrained based model creation. Will add that functionality separately after the hashing methodlogy is finalized. Signed-off-by: Dhiraj Kumar Sah <[email protected]>

…ents on naming and ordering as well. Signed-off-by: Dhiraj Kumar Sah <[email protected]>

…file to contain all hashing related methods and tools. Minor code clean ups. Signed-off-by: Dhiraj Kumar Sah <[email protected]>

…' instead of 'model_params'. Signed-off-by: Dhiraj Kumar Sah <[email protected]>

…into an error if that parameter doesn't exist in the config. Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Signed-off-by: Dhiraj Kumar Sah <[email protected]>

quic-dhirajku requested review from quic-rishinr, ochougul, quic-hemagnih and quic-amitraj as code owners June 24, 2025 08:30

quic-rishinr added the 1.21.0 label Jun 24, 2025

ochougul requested changes Jun 24, 2025

View reviewed changes

ochougul mentioned this pull request Jun 24, 2025

Bug fix for spdTransform #467

Merged

quic-rishinr marked this pull request as draft June 27, 2025 12:24

quic-amitraj requested changes Jul 2, 2025

View reviewed changes

quic-rishinr requested changes Jul 11, 2025

View reviewed changes

quic-dhirajku force-pushed the hash_utility branch from 99ff668 to 7b9cfba Compare July 16, 2025 08:41

quic-dhirajku marked this pull request as ready for review July 17, 2025 05:52

quic-rishinr requested changes Jul 31, 2025

View reviewed changes

quic-dhirajku force-pushed the hash_utility branch from b28d33e to a07406b Compare August 4, 2025 09:55

quic-dhirajku and others added 15 commits August 4, 2025 09:56

Gemma 3 minor fixes (quic#476)

efa32b8

CI enablement and other minor fixes for Gemma3 --------- Signed-off-by: Ann Kuruvilla <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Bug fix for spdTransform (quic#467)

e925939

Added fix for spdtransform due to change in hash --------- Signed-off-by: Dipankar Sarkar <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Revert "Gemma 3 minor fixes (quic#476) - CPR" (quic#485)

66e3859

Reverts quic#484 Signed-off-by: Dhiraj Kumar Sah <[email protected]>

QUICKFIX: Removed the redundant breakpoint comment in modeling_llava_…

5dd6147

…next file. (quic#475) Signed-off-by: Dhiraj Kumar Sah <[email protected]>

MDP hash support (quic#479)

2a4f02c

Signed-off-by: Rishin Raj <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Corrected Total Inference Time unit (quic#505)

52dc6f3

Changed Total (E2E) inference time from decode/sec to sec. Signed-off-by: Asmita Goswami <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

qcdipankar and others added 14 commits August 4, 2025 09:56

Adding Fix for Falcon model (quic#508)

e61ca38

Falcon Modeling fix to accommodate multiple config. This is a fix for falcon 40b Signed-off-by: Dipankar Sarkar <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Dynamic cache support on llama4 (quic#494)

e3f5ab4

Signed-off-by: Rishin <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Incorporated changes suggested in comments

908ab65

Signed-off-by: Dhiraj Kumar Sah <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Edited a comment on compile params dump

6f99b2c

Signed-off-by: Dhiraj Kumar Sah <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Modifications made based on Rishin's suggestion. WIP

bd419b3

Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Modifications to the flow of hash creation and filtration of params f…

a2606f1

…or export Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Clean-up post rebase was done.

78b7950

Functions made to filter and modify hashes. Signed-off-by: Dhiraj Kumar Sah <[email protected]>

commit for Linter issues

f401f0a

Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Removed partial changes done for Metaclass utilization to enforce fro…

f5e8f8c

…m_pretrained based model creation. Will add that functionality separately after the hashing methodlogy is finalized. Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Made changes to incorporate PEFT model configs and addressed the comm…

7e1df0a

…ents on naming and ordering as well. Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Updated path to import 'to_hashable' method, as we have 'hash_utils' …

c5bed92

…file to contain all hashing related methods and tools. Minor code clean ups. Signed-off-by: Dhiraj Kumar Sah <[email protected]>

quic-dhirajku force-pushed the hash_utility branch from a07406b to c5bed92 Compare August 4, 2025 09:57

quic-dhirajku added 3 commits August 5, 2025 05:53

edited 'QEffAutoModelForCausalLM' to store class name in 'hash_params…

6eabbeb

…' instead of 'model_params'. Signed-off-by: Dhiraj Kumar Sah <[email protected]>

modified the way 'model_architecture' is stored so that we don't run …

46f5ffd

…into an error if that parameter doesn't exist in the config. Signed-off-by: Dhiraj Kumar Sah <[email protected]>

Updated the test scripts with changes required for appropriate testing

133c076

Signed-off-by: Dhiraj Kumar Sah <[email protected]>

		self.model_params["output_names"] = output_names
		self.model_params["dynamic_axes"] = dynamic_axes


		self.pretrained_model_name_or_path = kwargs.get("pretrained_model_name_or_path", None)
		# self.pretrained_model_name_or_path = kwargs.get("pretrained_model_name_or_path", None)

		if hasattr(self.model.config, "architectures"):
		self.model_architecture = self.model.config.architectures[0]

Changing the hashing methodology for cache folder creation of models. #481

Are you sure you want to change the base?

Changing the hashing methodology for cache folder creation of models. #481

Conversation

quic-dhirajku commented Jun 24, 2025

Uh oh!

ochougul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

quic-amitraj left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

quic-amitraj left a comment •

edited

Loading