合并lora权重到基础模型权重

合并lora权重到基础模型权重无论是用NEMO_ENABLE_USER_MODULES1 CUDA_VISIBLE_DEVICES0,1 automodel --nproc-per-node2 train-Qwen-Qwen-2.5-0.5B-Instruct.yaml执行继续预训练还是SFT微调都支持lora微调方法使用lora微调后会得到adapter权重文件合并lora权重到基础模型权重可以使用AutoModel提供的merge_lora.py脚本指令如下所示python /opt/Automodel/tools/merge_lora.py\--base-model /hmlp/data/storage/Qwen2.5-0.5B-Instruct/V1/Qwen2.5-0.5B-Instruct\--adapter-path /hmlp/data/finetune/8e418d7f-b009-4f4f-a1a3-a19fc27b7f0c/f1a28b50-b8e5-4318-8545-5a8b57b67be8/output/epoch_0_step_229/model\--output-dir /hmlp/data/finetune/8e418d7f-b009-4f4f-a1a3-a19fc27b7f0c/f1a28b50-b8e5-4318-8545-5a8b57b67be8/merge/Qwen2.5-0.5B-Instruct-v4base-model指定基础模型的 HuggingFace 模型名或本地路径adapter-path指定 PEFT adapter 目录路径目录下需要包含 adapter_config.json 和 adapter 权重文件output-dir指定合并后模型的输出目录合并后的模型权重和配置会保存到这里默认也会把 tokenizer 一起保存qlora告诉脚本这是 QLoRA 场景类型: 布尔开关 默认值: False。根据训练时是否包含如下配置判断是否是QLoRA 场景quantization:load_in_4bit:Trueload_in_8bit:Falsebnb_4bit_compute_dtype:bfloat16bnb_4bit_use_double_quant:Truebnb_4bit_quant_type:nf4开启后基础模型会以 4-bit 方式加载然后先反量化再做 merge用于避免直接对量化权重做 naive merge 带来的效果下降dtype指定合并后模型权重的数据类型默认为 float16。可选值如下float16bfloat16float32device 指定模型加载时使用的设备映射默认为 “auto”表示自动检测和使用可用的 GPU 设备。也可以指定为 “cpu” 或 “cuda”。--device cpu--device auto在 --qlora 模式下这个参数实际上没有生效因为代码里固定用了 device_map{“”: 0}见 tools/merge_lora.py:233-234no-save-tokenizer不保存 tokenizer类型: 布尔开关 默认值: Falsetrust-remote-code加载模型时允许执行 HuggingFace 仓库中的自定义代码。 必填: 否 类型: 布尔开关 默认值: False。对一些自定义模型结构是必须的model-class显式指定使用哪个 transformers 的 AutoModel* 类来加载基础模型必填: 否 默认值: None--model-class AutoModel--model-class AutoModelForCausalLM如果不传脚本会尝试从 adapter_config.json 的 task_type 自动推断映射关系定义在 TASK_TYPE_TO_AUTO_CLASS见 tools/merge_lora.py:76如果既没有显式传入也无法识别 task_type则回退到 AutoModelForCausalLMmodel-class自动推断模型类规则–model-class 不传时脚本按下面顺序决定加载类见 tools/merge_lora.py:86-121优先使用你显式传入的 --model-class否则读取 adapter_path/adapter_config.json 里的 task_type再映射到对应的 transformers Auto 类如果还是不行默认使用 AutoModelForCausalLM当前支持的 task_type 映射为CAUSAL_LM - AutoModelForCausalLMSEQ_CLS - AutoModelForSequenceClassificationSEQ_2_SEQ_LM - AutoModelForSeq2SeqLMTOKEN_CLS - AutoModelForTokenClassificationQUESTION_ANS - AutoModelForQuestionAnsweringFEATURE_EXTRACTION - AutoModelcheckpoint.save_consolidatedcheckpoint:save_consolidated:true当为 true 时将一个与 HuggingFace 兼容的单个检查点写入model/consolidated/可直接由 Transformers、vLLM 等加载。需要 safetensors 格式。此参数并不会自动合并lora权重生成完整模型save_consolidated: true 的本质 是“额外保存一份可直接 from_pretrained 加载的完整 HF 格式模型权重”不会在 LoRA/PEFT 微调时自动合并 base model adapter 生成最终完整模型。 原因是 PEFT 场景下 is_peftTrue而 consolidated 导出逻辑明确排除了 PEFT。训练保存的仍然主要是 adapter 权重合并后模型部署合并后模型部署会报如下错误:INFO 06-01 20:26:39 [__init__.py:239] Automatically detected platform cuda. INFO 06-01 20:26:44 [api_server.py:1043] vLLM API server version 0.8.5 INFO 06-01 20:26:44 [api_server.py:1044] args: Namespace(subparserserve, model_tag/hmlp/data/finetune/ead13c8f-2a8f-4952-b0ca-ce6bc70b5e1d/f1a28b50-b8e5-4318-8545-5a8b57b67be8/merge/Qwen-2.5-0.5B-Instruct-0.3-V6, config, host0.0.0.0, port38109, uvicorn_log_levelinfo, disable_uvicorn_access_logFalse, allow_credentialsFalse, allowed_origins[*], allowed_methods[*], allowed_headers[*], api_keyNone, lora_modulesNone, prompt_adaptersNone, chat_templateNone, chat_template_content_formatauto, response_roleassistant, ssl_keyfileNone, ssl_certfileNone, ssl_ca_certsNone, enable_ssl_refreshFalse, ssl_cert_reqs0, root_pathNone, middleware[], return_tokens_as_token_idsFalse, disable_frontend_multiprocessingFalse, enable_request_id_headersFalse, enable_auto_tool_choiceFalse, tool_call_parserNone, tool_parser_plugin, model/hmlp/data/finetune/ead13c8f-2a8f-4952-b0ca-ce6bc70b5e1d/f1a28b50-b8e5-4318-8545-5a8b57b67be8/merge/Qwen-2.5-0.5B-Instruct-0.3-V6, taskauto, tokenizerNone, hf_config_pathNone, skip_tokenizer_initFalse, revisionNone, code_revisionNone, tokenizer_revisionNone, tokenizer_modeauto, trust_remote_codeFalse, allowed_local_media_pathNone, load_formatauto, download_dirNone, model_loader_extra_config{}, use_tqdm_on_loadTrue, config_formatConfigFormat.AUTO: auto, dtypeauto, max_model_lenNone, guided_decoding_backendauto, reasoning_parserNone, logits_processor_patternNone, model_implauto, distributed_executor_backendNone, pipeline_parallel_size1, tensor_parallel_size2, data_parallel_size1, enable_expert_parallelFalse, max_parallel_loading_workersNone, ray_workers_use_nsightFalse, disable_custom_all_reduceFalse, block_sizeNone, gpu_memory_utilization0.25, swap_space4, kv_cache_dtypeauto, num_gpu_blocks_overrideNone, enable_prefix_cachingNone, prefix_caching_hash_algobuiltin, cpu_offload_gb0, calculate_kv_scalesFalse, disable_sliding_windowFalse, use_v2_block_managerTrue, seedNone, max_logprobs20, disable_log_statsFalse, quantizationNone, rope_scalingNone, rope_thetaNone, hf_tokenNone, hf_overridesNone, enforce_eagerFalse, max_seq_len_to_capture8192, tokenizer_pool_size0, tokenizer_pool_typeray, tokenizer_pool_extra_config{}, limit_mm_per_prompt{}, mm_processor_kwargsNone, disable_mm_preprocessor_cacheFalse, enable_loraNone, enable_lora_biasFalse, max_loras1, max_lora_rank16, lora_extra_vocab_size256, lora_dtypeauto, long_lora_scaling_factorsNone, max_cpu_lorasNone, fully_sharded_lorasFalse, enable_prompt_adapterNone, max_prompt_adapters1, max_prompt_adapter_token0, deviceauto, speculative_configNone, ignore_patterns[], served_model_name[Qwen-2.5-0.5B-Instruct-0.3 2070948db0fc473488dcc8c0f9250b8e], qlora_adapter_name_or_pathNone, show_hidden_metrics_for_versionNone, otlp_traces_endpointNone, collect_detailed_tracesNone, disable_async_output_procFalse, max_num_batched_tokensNone, max_num_seqsNone, max_num_partial_prefills1, max_long_partial_prefills1, long_prefill_token_threshold0, num_lookahead_slots0, scheduler_delay_factor0.0, preemption_modeNone, num_scheduler_steps1, multi_step_stream_outputsTrue, scheduling_policyfcfs, enable_chunked_prefillNone, disable_chunked_mm_inputFalse, scheduler_clsvllm.core.scheduler.Scheduler, override_neuron_configNone, override_pooler_configNone, compilation_configNone, kv_transfer_configNone, worker_clsauto, worker_extension_cls, generation_configauto, override_generation_configNone, enable_sleep_modeFalse, additional_configNone, enable_reasoningFalse, disable_cascade_attnFalse, disable_log_requestsFalse, max_log_lenNone, disable_fastapi_docsFalse, enable_prompt_tokens_detailsFalse, enable_server_load_trackingFalse, dispatch_functionfunction ServeSubcommand.cmd at 0x7faf290a0540) INFO 06-01 20:26:44 [config.py:2968] Downcasting torch.float32 to torch.float16. INFO 06-01 20:26:49 [config.py:717] This model supports multiple tasks: {classify, generate, score, embed, reward}. Defaulting to generate. INFO 06-01 20:26:49 [config.py:1770] Defaulting to use mp for distributed inference INFO 06-01 20:26:49 [config.py:2003] Chunked prefill is enabled with max_num_batched_tokens8192. Traceback (most recent call last): File /usr/local/bin/vllm, line 10, in module sys.exit(main()) ^^^^^^ File /usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py, line 53, in main args.dispatch_function(args) File /usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py, line 27, in cmd uvloop.run(run_server(args)) File /usr/local/lib/python3.12/dist-packages/uvloop/__init__.py, line 109, in run return __asyncio.run( ^^^^^^^^^^^^^^ File /usr/lib/python3.12/asyncio/runners.py, line 195, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File /usr/lib/python3.12/asyncio/runners.py, line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File uvloop/loop.pyx, line 1518, in uvloop.loop.Loop.run_until_complete File /usr/local/lib/python3.12/dist-packages/uvloop/__init__.py, line 61, in wrapper return await main ^^^^^^^^^^ File /usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py, line 1078, in run_server async with build_async_engine_client(args) as engine_client: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /usr/lib/python3.12/contextlib.py, line 210, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File /usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py, line 146, in build_async_engine_client async with build_async_engine_client_from_engine_args( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /usr/lib/python3.12/contextlib.py, line 210, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File /usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py, line 178, in build_async_engine_client_from_engine_args async_llm AsyncLLM.from_vllm_config( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File /usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py, line 150, in from_vllm_config return cls( ^^^^ File /usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py, line 97, in __init__ self.tokenizer init_tokenizer_from_configs( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/tokenizer_group.py, line 101, in init_tokenizer_from_configs return TokenizerGroup( ^^^^^^^^^^^^^^^ File /usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/tokenizer_group.py, line 23, in __init__ self.tokenizer get_tokenizer(self.tokenizer_id, **tokenizer_config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/tokenizer.py, line 221, in get_tokenizer tokenizer AutoTokenizer.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /usr/local/lib/python3.12/dist-packages/transformers/models/auto/tokenization_auto.py, line 1009, in from_pretrained return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py, line 2062, in from_pretrained return cls._from_pretrained( ^^^^^^^^^^^^^^^^^^^^^ File /usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py, line 2302, in _from_pretrained tokenizer cls(*init_inputs, **init_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /usr/local/lib/python3.12/dist-packages/transformers/models/qwen2/tokenization_qwen2_fast.py, line 120, in __init__ super().__init__( File /usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_fast.py, line 178, in __init__ super().__init__(**kwargs) File /usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py, line 1452, in __init__ self._set_model_specific_special_tokens(special_tokensself.extra_special_tokens) File /usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py, line 1190, in _set_model_specific_special_tokens self.SPECIAL_TOKENS_ATTRIBUTES self.SPECIAL_TOKENS_ATTRIBUTES list(special_tokens.keys()) ^^^^^^^^^^^^^^^^^^^ AttributeError: list object has no attribute keys上述报错原因是tokenizer_config.json文件中extra_special_tokens字段类型不被VLLM识别失败tokenizer_config.json文件在做微调时输出的Lora文件中就已经出问题了没有参考基础模型的tokenizer_config.json。但llama factory那边的checkpoint里面基本上与基础模型保持一致的automodel微调后产生的权重文件哪些没有问题但tokenizer文件是有问题的没有参考基础模型里面的tokenizer。问题其实不是合并脚本的问题是微调后tokenizer就出问题英伟达工程师给的临时解决方案使用如下新的merge_lora.py脚本# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the License); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an AS IS BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. Merge a LoRA/QLoRA adapter into a base HuggingFace model and save the result. Supports both dense and Mixture-of-Experts (MoE) models. For QLoRA adapters the base model is loaded in 4-bit, dequantized, and only then merged so that the adapter delta is applied to the correct weight representation (avoids the naive merge quality degradation described in https://kaitchup.substack.com/p/lora-adapters-when-a-naive-merge). Usage examples -------------- Standard LoRA merge (dense):: python tools/merge_lora.py \ --base-model meta-llama/Llama-3.2-1B \ --adapter-path checkpoints/adapter/ \ --output-dir merged_model/ QLoRA merge (dequantize then merge):: python tools/merge_lora.py \ --base-model meta-llama/Llama-3.2-1B \ --adapter-path checkpoints/adapter/ \ --output-dir merged_model/ \ --qlora MoE model merge:: python tools/merge_lora.py \ --base-model deepseek-ai/DeepSeek-V2-Lite \ --adapter-path checkpoints/adapter/ \ --output-dir merged_model/ Embedding / non-CausalLM model merge (auto-detected from adapter task_type):: python tools/merge_lora.py \ --base-model BAAI/bge-base-en-v1.5 \ --adapter-path checkpoints/adapter/ \ --output-dir merged_model/ Explicit model class override:: python tools/merge_lora.py \ --base-model BAAI/bge-base-en-v1.5 \ --adapter-path checkpoints/adapter/ \ --output-dir merged_model/ \ --model-class AutoModel import argparse import copy import gc import json import logging import os import torch logging.basicConfig(levellogging.INFO, format%(asctime)s %(levelname)s %(message)s) logger logging.getLogger(__name__) TASK_TYPE_TO_AUTO_CLASS { CAUSAL_LM: AutoModelForCausalLM, SEQ_CLS: AutoModelForSequenceClassification, SEQ_2_SEQ_LM: AutoModelForSeq2SeqLM, TOKEN_CLS: AutoModelForTokenClassification, QUESTION_ANS: AutoModelForQuestionAnswering, FEATURE_EXTRACTION: AutoModel, } def _resolve_auto_cls(adapter_path: str, model_class: str | None None): Return the transformers.AutoModel* class to use for loading. Resolution order: 1. Explicit *model_class* string (e.g. AutoModel). 2. task_type field in the adapters adapter_config.json. 3. Fall back to AutoModelForCausalLM. import transformers if model_class is not None: cls getattr(transformers, model_class, None) if cls is None: raise ValueError( fUnknown model class {model_class}. fMust be an attribute of the transformers package f(e.g. AutoModelForCausalLM, AutoModel). ) return cls config_path os.path.join(adapter_path, adapter_config.json) if os.path.isfile(config_path): with open(config_path, r) as f: adapter_cfg json.load(f) task_type adapter_cfg.get(task_type) if task_type and task_type in TASK_TYPE_TO_AUTO_CLASS: cls_name TASK_TYPE_TO_AUTO_CLASS[task_type] logger.info(Detected task_type%s → using %s, task_type, cls_name) return getattr(transformers, cls_name) if task_type: logger.warning( Unrecognised task_type %s in adapter config; falling back to AutoModelForCausalLM., task_type, ) return transformers.AutoModelForCausalLM def dequantize_model(model, dtypetorch.float16, devicecpu): Replace every bitsandbytes.nn.Linear4bit with a plain nn.Linear. The 4-bit weights are dequantized using the stored quant_state so that the full-precision delta from the LoRA adapter is applied correctly. Args: model: A HuggingFace model loaded with 4-bit quantization. dtype: Target dtype for the dequantized weights. device: Device to place the new linear layers on. Returns: The same model object, modified in-place. import bitsandbytes as bnb from bitsandbytes.functional import dequantize_4bit with torch.no_grad(): for name, module in model.named_modules(): if isinstance(module, bnb.nn.Linear4bit): logger.info(Dequantizing %s, name) quant_state copy.deepcopy(module.weight.quant_state) quant_state[2] dtype weights dequantize_4bit(module.weight.data, quant_statequant_state, quant_typenf4).to(dtype) new_module torch.nn.Linear( module.in_features, module.out_features, biasmodule.bias is not None, dtypedtype ) new_module.weight torch.nn.Parameter(weights) if module.bias is not None: new_module.bias torch.nn.Parameter(module.bias.data.to(dtype)) new_module.to(devicedevice, dtypedtype) # Walk to the parent and swap the child module. parts name.rsplit(., 1) if len(parts) 2: parent model.get_submodule(parts[0]) setattr(parent, parts[1], new_module) else: setattr(model, name, new_module) model.is_loaded_in_4bit False return model _V5_RUNTIME_TOKENIZER_KEYS frozenset({backend, is_local}) _SPECIAL_TOKENS_LIST_KEYS (additional_special_tokens, extra_special_tokens) def _load_json_file(path: str) - dict | None: Load a JSON file, returning None when missing or unreadable. if not os.path.isfile(path): return None try: with open(path, r) as f: return json.load(f) except (json.JSONDecodeError, OSError) as e: logger.warning(Could not read %s: %s, path, e) return None def _merge_special_token_lists(base_config: dict, adapter_config: dict) - list | None: Merge special-token lists from both configs, preserving order. merged: list [] seen: set set() for key in _SPECIAL_TOKENS_LIST_KEYS: for cfg in (base_config, adapter_config): tokens cfg.get(key) if not tokens: continue for token in tokens: if token not in seen: seen.add(token) merged.append(token) return merged or None def _merge_tokenizer_configs(base_config: dict | None, adapter_config: dict | None) - dict | None: Merge base-model and adapter tokenizer_config.json contents. The base config is used as the foundation so fields such as added_tokens_decoder and chat_template are preserved. Adapter values override the base for keys explicitly present in the adapter config. Transformers v5 runtime-only keys (backend, is_local) are dropped. if base_config is None and adapter_config is None: return None if base_config is None: merged dict(adapter_config) elif adapter_config is None: merged dict(base_config) else: merged dict(base_config) for key, value in adapter_config.items(): if key in _V5_RUNTIME_TOKENIZER_KEYS: continue if key added_tokens_decoder: decoder dict(base_config.get(added_tokens_decoder) or {}) decoder.update(value or {}) merged[added_tokens_decoder] decoder elif key in _SPECIAL_TOKENS_LIST_KEYS: continue else: merged[key] value special_tokens _merge_special_token_lists(base_config, adapter_config) if special_tokens is not None: merged[additional_special_tokens] special_tokens for key in _V5_RUNTIME_TOKENIZER_KEYS: merged.pop(key, None) extra_tokens merged.pop(extra_special_tokens, None) if extra_tokens and additional_special_tokens not in merged: merged[additional_special_tokens] extra_tokens return merged def _apply_merged_tokenizer_config(base_model: str, adapter_path: str, output_dir: str) - None: Write a merged tokenizer_config.json into *output_dir*. base_config _load_json_file(os.path.join(base_model, tokenizer_config.json)) adapter_config _load_json_file(os.path.join(adapter_path, tokenizer_config.json)) saved_config _load_json_file(os.path.join(output_dir, tokenizer_config.json)) merged _merge_tokenizer_configs(base_config, adapter_config) if merged is None: if saved_config is None: logger.warning(No tokenizer_config.json found in base model, adapter, or output directory) return if saved_config and tokenizer_class not in merged and tokenizer_class in saved_config: merged[tokenizer_class] saved_config[tokenizer_class] config_path os.path.join(output_dir, tokenizer_config.json) with open(config_path, w) as f: json.dump(merged, f, indent2, ensure_asciiFalse) f.write(\n) logger.info(Merged tokenizer_config.json saved to %s, config_path) def _clean_quantization_config(output_dir): Remove quantization_config from the saved config.json. config_path os.path.join(output_dir, config.json) if not os.path.exists(config_path): return with open(config_path, r) as f: config_data json.load(f) changed False for key in (quantization_config, pretraining_tp): if key in config_data: config_data.pop(key) changed True if changed: with open(config_path, w) as f: json.dump(config_data, f, indent2) def merge_lora( base_model: str, adapter_path: str, output_dir: str, *, qlora: bool False, dtype: str float16, device: str auto, save_tokenizer: bool True, trust_remote_code: bool False, model_class: str | None None, ): Load a base model, apply a LoRA adapter, merge, and save. Args: base_model: HuggingFace model name or path. adapter_path: Path to the PEFT adapter directory. output_dir: Where to write the merged model. qlora: If True, load the base model in 4-bit and dequantize before merging. dtype: Weight dtype for the merged model (float16, bfloat16, float32). device: auto, cpu, or cuda:N. save_tokenizer: Whether to save the tokenizer alongside the model. trust_remote_code: Passed through to from_pretrained. model_class: Explicit transformers Auto class name (e.g. AutoModel, AutoModelForCausalLM). When None the class is inferred from the adapters task_type. from peft import PeftModel from transformers import AutoTokenizer auto_cls _resolve_auto_cls(adapter_path, model_class) torch_dtype getattr(torch, dtype) # --- Load base model --- load_kwargs dict( trust_remote_codetrust_remote_code, ) if qlora: from transformers import BitsAndBytesConfig bnb_config BitsAndBytesConfig( load_in_4bitTrue, bnb_4bit_quant_typenf4, bnb_4bit_compute_dtypetorch_dtype, bnb_4bit_use_double_quantTrue, ) load_kwargs[quantization_config] bnb_config load_kwargs[device_map] {: 0} logger.info(Loading base model in 4-bit for QLoRA merge (%s): %s, auto_cls.__name__, base_model) else: load_kwargs[torch_dtype] torch_dtype load_kwargs[device_map] device logger.info(Loading base model (%s): %s, auto_cls.__name__, base_model) model auto_cls.from_pretrained(base_model, **load_kwargs) # --- Dequantize if QLoRA --- if qlora: logger.info(Dequantizing model weights from 4-bit to %s, dtype) model dequantize_model(model, dtypetorch_dtype, devicecuda) # --- Apply adapter and merge --- logger.info(Loading adapter from %s, adapter_path) model PeftModel.from_pretrained(model, adapter_path) logger.info(Merging adapter into base model) model model.merge_and_unload() # --- Save --- os.makedirs(output_dir, exist_okTrue) logger.info(Saving merged model to %s, output_dir) model.save_pretrained(output_dir, safe_serializationTrue) if qlora: _clean_quantization_config(output_dir) if save_tokenizer: try: tokenizer_source base_model if os.path.isfile(os.path.join(adapter_path, tokenizer_config.json)): tokenizer_source adapter_path tokenizer AutoTokenizer.from_pretrained(tokenizer_source, trust_remote_codetrust_remote_code) tokenizer.save_pretrained(output_dir) _apply_merged_tokenizer_config(base_model, adapter_path, output_dir) logger.info(Tokenizer saved to %s, output_dir) except Exception as e: logger.warning(Could not save tokenizer: %s, e) logger.info(Merge complete.) # Free memory del model torch.cuda.empty_cache() gc.collect() def parse_args() - argparse.Namespace: parser argparse.ArgumentParser(descriptionMerge a LoRA / QLoRA adapter into a HuggingFace base model.) parser.add_argument( --base-model, -m, requiredTrue, helpHuggingFace model name/path (e.g. meta-llama/Llama-3.2-1B or /path/to/model)., ) parser.add_argument( --adapter-path, -a, requiredTrue, helpPath to the PEFT adapter directory (must contain adapter_config.json)., ) parser.add_argument( --output-dir, -o, requiredTrue, helpDirectory to save the merged model., ) parser.add_argument( --qlora, actionstore_true, defaultFalse, helpLoad base model in 4-bit and dequantize before merging (for QLoRA adapters)., ) parser.add_argument( --dtype, choices[float16, bfloat16, float32], defaultfloat16, helpWeight dtype for the merged model (default: float16)., ) parser.add_argument( --device, defaultauto, helpDevice map for model loading (default: auto). Use cpu for CPU-only., ) parser.add_argument( --no-save-tokenizer, actionstore_true, defaultFalse, helpSkip saving the tokenizer., ) parser.add_argument( --trust-remote-code, actionstore_true, defaultFalse, helpTrust remote code when loading the model., ) parser.add_argument( --model-class, defaultNone, help( Explicit transformers Auto class name (e.g. AutoModel, AutoModelForCausalLM). When omitted, inferred from the adapters task_type in adapter_config.json. ), ) return parser.parse_args() def main(): args parse_args() merge_lora( base_modelargs.base_model, adapter_pathargs.adapter_path, output_dirargs.output_dir, qloraargs.qlora, dtypeargs.dtype, deviceargs.device, save_tokenizernot args.no_save_tokenizer, trust_remote_codeargs.trust_remote_code, model_classargs.model_class, ) if __name__ __main__: main()