The Easiest and Cheapest Way to Deploy Finetuned Mistral 7B Instruct Model (or Any Model) | by Qendel AI - Freedium

Copy
# Mount your drive 
drive.mount('/content/drive')

# Define the data type 
dtype = torch.bfloat16 if torch.cuda.get_device_capability()[0] == 8 else torch.float16

# Your finetuned model's path (either in google drive or local)
model_path = "/path/to/your/finetuned/model/"

# Load the finetuned model and tokenizer
finetuned_model = AutoPeftModelForCausalLM.from_pretrained(
    model_path,
    low_cpu_mem_usage=True,
    torch_dtype=dtype,
    device_map='auto'
)
tokenizer = AutoTokenizer.from_pretrained(model_path)