文本分类示例step1 导入相关包fromtransformersimportAutoTokenizer,AutoModelForSequenceClassification,Trainer,TrainingArgumentsfromdatasetsimportload_data_setstep2 加载数据集datasetload_dataset(csv,data_files./ChnSentiCorp.csv,splittrain)datasetdataset.filter(lambdax:x[review]isnotNone)step3 划分数据集datasetsdataset.train_test_split(test_size0.1)step4 数据集预处理importtorch# 加载 rbt3 对应的分词器tokenizerAutoTokenizer.from_pretrained(hfl/rbt3)# 定义预处理函数batchedTrue时examples是一批样本defprocess_function(examples):# 对review字段评论文本分词编码tokenized_examplestokenizer(examples[review],max_length128,truncationTrue)# 把原始标签复制到labels字段Trainer训练要求固定名字labelstokenized_examples[labels]examples[label]returntokenized_examples# 批量处理整个数据集tokenized_datasetsdatasets.map(process_function,batchedTrue,remove_columnsdatasets[train].column_names)step5 创建模型modelAutoModelForSequenceClassification.from_pretrained(hfl/chinese-macbert-large)step6 创建评估函数importevaluate# 如果网络不太好也可以使用本地加载的方式acc_metricevaluate.load(./metric_accuracy.py)f1_metircevaluate.load(./metric_f1.py)defeval_metric(eval_predict):predictions,labelseval_predict predictionspredictions.argmax(axis-1)accacc_metric.compute(predictionspredictions,referenceslabels)f1f1_metirc.compute(predictionspredictions,referenceslabels)acc.update(f1)returnaccStep7 创建TrainingArgumentstrain_argsTrainingArguments(output_dir./checkpoints,# 输出文件夹per_device_train_batch_size2,# 训练时的batch_sizegradient_accumulation_steps32,# *** 梯度累加 ***gradient_checkpointingTrue,# *** 梯度检查点 ***optimadafactor,# *** adafactor优化器 ***per_device_eval_batch_size4,# 验证时的batch_sizenum_train_epochs1,# 训练轮数logging_steps10,# log 打印的频率eval_strategyepoch,# 评估策略save_strategyepoch,# 保存策略save_total_limit3,# 最大保存数learning_rate2e-5,# 学习率weight_decay0.001,# weight_decaymetric_for_best_modelf1,# 设定评估指标load_best_model_at_endTrue)# 训练完成后加载最优模型train_argsStep8 创建TrainerfromtransformersimportDataCollatorWithPadding# *** 参数冻结 ***forname,paraminmodel.bert.named_parameters():param.requires_gradFalsetrainerTrainer(modelmodel,argstrain_args,tokenizertokenizer,train_datasettokenized_datasets[train],eval_datasettokenized_datasets[test],data_collatorDataCollatorWithPadding(tokenizertokenizer),compute_metricseval_metric)Step9 模型训练trainer.train()trainer.evaluate(tokenized_datasets[test])trainer.predict(tokenized_datasets[test])Step10 模型预测sen我觉得这家酒店不错饭很好吃id2_label{0:差评,1:好评}model.eval()withtorch.inference_mode():inputstokenizer(sen,return_tensorspt)inputs{k:v.cuda()fork,vininputs.items()}logitsmodel(**inputs).logits predtorch.argmax(logits,dim-1)print(f输入{sen}\n模型预测结果:{id2_label.get(pred.item())})fromtransformersimportpipeline model.config.id2labelid2_label pipepipeline(text-classification,modelmodel,tokenizertokenizer,device0)显存优化策略模型权重 4B模型参数量优化器状态 8B模型参数量对于常用的AdamW优化器而言梯度 4B*模型参数量前向激活值 取决于序列长度、隐层维度、batch大小等多个因素
文本分类示例step1 导入相关包fromtransformersimportAutoTokenizer,AutoModelForSequenceClassification,Trainer,TrainingArgumentsfromdatasetsimportload_data_setstep2 加载数据集datasetload_dataset(csv,data_files./ChnSentiCorp.csv,splittrain)datasetdataset.filter(lambdax:x[review]isnotNone)step3 划分数据集datasetsdataset.train_test_split(test_size0.1)step4 数据集预处理importtorch# 加载 rbt3 对应的分词器tokenizerAutoTokenizer.from_pretrained(hfl/rbt3)# 定义预处理函数batchedTrue时examples是一批样本defprocess_function(examples):# 对review字段评论文本分词编码tokenized_examplestokenizer(examples[review],max_length128,truncationTrue)# 把原始标签复制到labels字段Trainer训练要求固定名字labelstokenized_examples[labels]examples[label]returntokenized_examples# 批量处理整个数据集tokenized_datasetsdatasets.map(process_function,batchedTrue,remove_columnsdatasets[train].column_names)step5 创建模型modelAutoModelForSequenceClassification.from_pretrained(hfl/chinese-macbert-large)step6 创建评估函数importevaluate# 如果网络不太好也可以使用本地加载的方式acc_metricevaluate.load(./metric_accuracy.py)f1_metircevaluate.load(./metric_f1.py)defeval_metric(eval_predict):predictions,labelseval_predict predictionspredictions.argmax(axis-1)accacc_metric.compute(predictionspredictions,referenceslabels)f1f1_metirc.compute(predictionspredictions,referenceslabels)acc.update(f1)returnaccStep7 创建TrainingArgumentstrain_argsTrainingArguments(output_dir./checkpoints,# 输出文件夹per_device_train_batch_size2,# 训练时的batch_sizegradient_accumulation_steps32,# *** 梯度累加 ***gradient_checkpointingTrue,# *** 梯度检查点 ***optimadafactor,# *** adafactor优化器 ***per_device_eval_batch_size4,# 验证时的batch_sizenum_train_epochs1,# 训练轮数logging_steps10,# log 打印的频率eval_strategyepoch,# 评估策略save_strategyepoch,# 保存策略save_total_limit3,# 最大保存数learning_rate2e-5,# 学习率weight_decay0.001,# weight_decaymetric_for_best_modelf1,# 设定评估指标load_best_model_at_endTrue)# 训练完成后加载最优模型train_argsStep8 创建TrainerfromtransformersimportDataCollatorWithPadding# *** 参数冻结 ***forname,paraminmodel.bert.named_parameters():param.requires_gradFalsetrainerTrainer(modelmodel,argstrain_args,tokenizertokenizer,train_datasettokenized_datasets[train],eval_datasettokenized_datasets[test],data_collatorDataCollatorWithPadding(tokenizertokenizer),compute_metricseval_metric)Step9 模型训练trainer.train()trainer.evaluate(tokenized_datasets[test])trainer.predict(tokenized_datasets[test])Step10 模型预测sen我觉得这家酒店不错饭很好吃id2_label{0:差评,1:好评}model.eval()withtorch.inference_mode():inputstokenizer(sen,return_tensorspt)inputs{k:v.cuda()fork,vininputs.items()}logitsmodel(**inputs).logits predtorch.argmax(logits,dim-1)print(f输入{sen}\n模型预测结果:{id2_label.get(pred.item())})fromtransformersimportpipeline model.config.id2labelid2_label pipepipeline(text-classification,modelmodel,tokenizertokenizer,device0)显存优化策略模型权重 4B模型参数量优化器状态 8B模型参数量对于常用的AdamW优化器而言梯度 4B*模型参数量前向激活值 取决于序列长度、隐层维度、batch大小等多个因素
平方复杂度无法被常规优化消除｜TaoToken 统一 Key 通道实测)
