How to export entire model?

[YuanzeSun]

Hi, the sliced model are exported to .pt file (only parameters, without structure). However, I want to deploy the model otherwhere, so I have to have the model structure and then can load the pretrained parameters.
So how can I do this? How can I export the entire model?
thanks!

[joyce0105-ops]

I have the same question

[nailimixaM]

Are you able to import slicegpt when deploying? We don't have an explicit model structure definition, we load checkpoints using hf_utils.load_sliced_model

[YuanzeSun]

Are you able to import slicegpt when deploying? We don't have an explicit model structure definition, we load checkpoints using hf_utils.load_sliced_model

Thank you for replying and I‘m afraid not. If want to deploy the model with llama.cpp, not PyTorch. If I use PyTorch, I can certainly import slicegpt when deploying the model. However, if llama.cpp, it has to have the entire structure if I want convert the model to .gguf.

[tu2022]

I have the same question, too. I wonder how to convert the sliced model to huggingface format (.bin) again, for inference.

[YuanzeSun]

I have the same question, too. I wonder how to convert the sliced model to huggingface format (.bin) again, for inference.

yes，if I use 'save_pretrained', the code is just the original one, no matching the modified model.

[nailimixaM]

Thanks for the clarifications - we don't have an explicit model class definition yet but would love to add this to the repo, e.g. in a modeling_sliced_llama2.py file which would also enable uploading models to HuggingFace. If either of you or anyone else from the opensource community would like to contribute this to the repo that would be amazing! Specifically the tasks:

• Export the model to gguf format
• Export the model to HF format

[SijunWang]

can anyone realize the task for exporting the model to HF format?

[nailimixaM]

this is currently being worked on in #139

[sriyachakravarthy]

what if we can't use slicegpt module to load? i am using configuration file and weights that were saved after pruning(using slice gpt) using AutoModelforCausalLlm and other Auto methods. I am observing mismatch in the dimensions between configuration file and state dictionary (converted .pt to .bin format) as shown below:

How to proceed with loading models using transformers package?

[NamburiSrinath]

Hi @nailimixaM,

I am wondering if someone can successfully load the sliced model to normally used Huggingface formats.

Raised an issue with a different message - #188

Thanks a ton