如何使用 Cloud Run 作业微调 LLM

1. 简介

概览

在此 Codelab 中，您将使用 Cloud Run 作业微调 Gemma 模型，然后使用 vLLM 在 Cloud Run 上提供结果。

在本 Codelab 中，您将使用文本到 SQL 数据集，以便 LLM 在收到自然语言问题时以 SQL 查询的形式进行回答。

学习内容

• 如何使用 Cloud Run 作业 GPU 进行微调
• 如何使用 Cloud Run 搭配 vLLM 提供模型
• 如何为 GPU 作业使用直接 VPC 配置，以更快地上传和分发模型

2. 准备工作

启用 API

在开始使用此 Codelab 之前，请通过运行以下命令启用以下 API：

gcloud services enable run.googleapis.com \
    compute.googleapis.com \
    run.googleapis.com \
    cloudbuild.googleapis.com \
    secretmanager.googleapis.com \
    artifactregistry.googleapis.com

GPU 配额

为受支持的区域申请增加配额。配额为 nvidia_l4_gpu_allocation_no_zonal_redundancy，位于 Cloud Run Admin API 下。

注意：如果您使用的是新项目，则可能需要几分钟时间，API 配额才会显示在此页面中。

Hugging Face

此 Codelab 使用托管在 Hugging Face 上的模型。如需获取此模型，请请求具有“读取”权限的 Hugging Face 用户访问令牌。在下文中，我们会用 YOUR_HF_TOKEN 来引用它。

您还需要同意使用条款才能使用该模型：https://huggingface.co/google/gemma-2b

3. 设置和要求

设置以下资源：

• IAM 服务账号和关联的 IAM 权限，
• 用于存储 Hugging Face 令牌的 Secret Manager Secret，
• Cloud Storage 存储分区（用于存储经过微调的模型），以及
• Artifact Registry 代码库，用于存储您将构建的用于微调模型的映像。

1. 为此 Codelab 设置环境变量。我们已为您预先填充了一些变量。指定您的项目 ID、区域和 Hugging Face 令牌。

   export PROJECT_ID=<YOUR_PROJECT_ID>
   export REGION=<YOUR_REGION>
   export HF_TOKEN=<YOUR_HF_TOKEN>
   
   export AR_REPO=codelab-finetuning-jobs
   export IMAGE_NAME=finetune-to-gcs
   export JOB_NAME=finetuning-to-gcs-job
   export BUCKET_NAME=$PROJECT_ID-codelab-finetuning-jobs
   export SECRET_ID=HF_TOKEN
   export SERVICE_ACCOUNT="finetune-job-sa"
   export SERVICE_ACCOUNT_ADDRESS=$SERVICE_ACCOUNT@$PROJECT_ID.iam.gserviceaccount.com
   

2. 运行以下命令来创建服务账号：

   gcloud iam service-accounts create $SERVICE_ACCOUNT \
     --display-name="Service account for fine-tuning codelab"
   

3. 使用 Secret Manager 存储 Hugging Face 访问令牌：

   gcloud secrets create $SECRET_ID \
         --replication-policy="automatic"
   
   printf $HF_TOKEN | gcloud secrets versions add $SECRET_ID --data-file=-
   

4. 向您的服务账号授予 Secret Manager Secret Accessor 角色：

   gcloud secrets add-iam-policy-binding $SECRET_ID \
     --member serviceAccount:$SERVICE_ACCOUNT_ADDRESS \
     --role='roles/secretmanager.secretAccessor'
   

5. 创建一个用于托管经过微调的模型的存储分区：

   gcloud storage buckets create -l $REGION gs://$BUCKET_NAME
   

6. 向您的服务账号授予对存储分区的访问权限：

   gcloud storage buckets add-iam-policy-binding gs://$BUCKET_NAME \
     --member=serviceAccount:$SERVICE_ACCOUNT_ADDRESS \
     --role=roles/storage.objectAdmin
   

7. 创建 Artifact Registry 制品库以存储容器映像：

   gcloud artifacts repositories create $AR_REPO \
       --repository-format=docker \
       --location=$REGION \
       --description="codelab for finetuning using CR jobs" \
       --project=$PROJECT_ID
   

4. 创建 Cloud Run 作业映像

在下一步中，您将创建以下代码：

• 从 Hugging Face 导入 Gemma 模型
• 使用 Hugging Face 中的数据集对模型进行微调。该作业使用单个 L4 GPU 进行微调。
• 将名为 new_model 的精调模型上传到您的 Cloud Storage 存储分区

1. 为微调作业代码创建一个目录。

   mkdir codelab-finetuning-job
   cd codelab-finetuning-job
   

2. 创建一个名为 finetune.py 的文件

   # Copyright 2024 Google LLC
   #
   # Licensed under the Apache License, Version 2.0 (the "License");
   # you may not use this file except in compliance with the License.
   # You may obtain a copy of the License at
   #
   #      http://www.apache.org/licenses/LICENSE-2.0
   #
   # Unless required by applicable law or agreed to in writing, software
   # distributed under the License is distributed on an "AS IS" BASIS,
   # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   # See the License for the specific language governing permissions and
   # limitations under the License.
   
   import os
   import torch
   from datasets import load_dataset
   from transformers import (
       AutoModelForCausalLM,
       AutoTokenizer,
       BitsAndBytesConfig,
       TrainingArguments,
   
   )
   from peft import LoraConfig, PeftModel
   
   from trl import SFTTrainer
   
   # Cloud Storage bucket to upload the model
   bucket_name = os.getenv("BUCKET_NAME", "YOUR_BUCKET_NAME")
   
   # The model that you want to train from the Hugging Face hub
   model_name = os.getenv("MODEL_NAME", "google/gemma-2b")
   
   # The instruction dataset to use
   dataset_name = "b-mc2/sql-create-context"
   
   # Fine-tuned model name
   new_model = os.getenv("NEW_MODEL", "gemma-2b-sql")
   
   ################################################################################
   # QLoRA parameters
   ################################################################################
   
   # LoRA attention dimension
   lora_r = int(os.getenv("LORA_R", "4"))
   
   # Alpha parameter for LoRA scaling
   lora_alpha = int(os.getenv("LORA_ALPHA", "8"))
   
   # Dropout probability for LoRA layers
   lora_dropout = 0.1
   
   ################################################################################
   # bitsandbytes parameters
   ################################################################################
   
   # Activate 4-bit precision base model loading
   use_4bit = True
   
   # Compute dtype for 4-bit base models
   bnb_4bit_compute_dtype = "float16"
   
   # Quantization type (fp4 or nf4)
   bnb_4bit_quant_type = "nf4"
   
   # Activate nested quantization for 4-bit base models (double quantization)
   use_nested_quant = False
   
   ################################################################################
   # TrainingArguments parameters
   ################################################################################
   
   # Output directory where the model predictions and checkpoints will be stored
   output_dir = "./results"
   
   # Number of training epochs
   num_train_epochs = 1
   
   # Enable fp16/bf16 training (set bf16 to True with an A100)
   fp16 = True
   bf16 = False
   
   # Batch size per GPU for training
   per_device_train_batch_size = int(os.getenv("TRAIN_BATCH_SIZE", "1"))
   
   # Batch size per GPU for evaluation
   per_device_eval_batch_size = int(os.getenv("EVAL_BATCH_SIZE", "2"))
   
   # Number of update steps to accumulate the gradients for
   gradient_accumulation_steps = int(os.getenv("GRADIENT_ACCUMULATION_STEPS", "1"))
   
   # Enable gradient checkpointing
   gradient_checkpointing = True
   
   # Maximum gradient normal (gradient clipping)
   max_grad_norm = 0.3
   
   # Initial learning rate (AdamW optimizer)
   learning_rate = 2e-4
   
   # Weight decay to apply to all layers except bias/LayerNorm weights
   weight_decay = 0.001
   
   # Optimizer to use
   optim = "paged_adamw_32bit"
   
   # Learning rate schedule
   lr_scheduler_type = "cosine"
   
   # Number of training steps (overrides num_train_epochs)
   max_steps = -1
   
   # Ratio of steps for a linear warmup (from 0 to learning rate)
   warmup_ratio = 0.03
   
   # Group sequences into batches with same length
   # Saves memory and speeds up training considerably
   group_by_length = True
   
   # Save checkpoint every X updates steps
   save_steps = 0
   
   # Log every X updates steps
   logging_steps = int(os.getenv("LOGGING_STEPS", "50"))
   
   ################################################################################
   # SFT parameters
   ################################################################################
   
   # Maximum sequence length to use
   max_seq_length = int(os.getenv("MAX_SEQ_LENGTH", "512"))
   
   # Pack multiple short examples in the same input sequence to increase efficiency
   packing = False
   
   # Load the entire model on the GPU 0
   device_map = {'':torch.cuda.current_device()}
   
   # Set limit to a positive number
   limit = int(os.getenv("DATASET_LIMIT", "5000"))
   
   dataset = load_dataset(dataset_name, split="train")
   if limit != -1:
       dataset = dataset.shuffle(seed=42).select(range(limit))
   
   
   def transform(data):
       question = data['question']
       context = data['context']
       answer = data['answer']
       template = "Question: {question}\nContext: {context}\nAnswer: {answer}"
       return {'text': template.format(question=question, context=context, answer=answer)}
   
   
   transformed = dataset.map(transform)
   
   # Load tokenizer and model with QLoRA configuration
   compute_dtype = getattr(torch, bnb_4bit_compute_dtype)
   
   bnb_config = BitsAndBytesConfig(
       load_in_4bit=use_4bit,
       bnb_4bit_quant_type=bnb_4bit_quant_type,
       bnb_4bit_compute_dtype=compute_dtype,
       bnb_4bit_use_double_quant=use_nested_quant,
   )
   
   # Check GPU compatibility with bfloat16
   if compute_dtype == torch.float16 and use_4bit:
       major, _ = torch.cuda.get_device_capability()
       if major >= 8:
           print("=" * 80)
           print("Your GPU supports bfloat16")
           print("=" * 80)
   
   # Load base model
   model = AutoModelForCausalLM.from_pretrained(
       model_name,
       quantization_config=bnb_config,
       device_map=device_map,
       torch_dtype=torch.float16,
   )
   model.config.use_cache = False
   model.config.pretraining_tp = 1
   
   # Load LLaMA tokenizer
   tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
   tokenizer.pad_token = tokenizer.eos_token
   tokenizer.padding_side = "right"
   
   # Load LoRA configuration
   peft_config = LoraConfig(
       lora_alpha=lora_alpha,
       lora_dropout=lora_dropout,
       r=lora_r,
       bias="none",
       task_type="CAUSAL_LM",
       target_modules=["q_proj", "v_proj"]
   )
   
   # Set training parameters
   training_arguments = TrainingArguments(
       output_dir=output_dir,
       num_train_epochs=num_train_epochs,
       per_device_train_batch_size=per_device_train_batch_size,
       gradient_accumulation_steps=gradient_accumulation_steps,
       optim=optim,
       save_steps=save_steps,
       logging_steps=logging_steps,
       learning_rate=learning_rate,
       weight_decay=weight_decay,
       fp16=fp16,
       bf16=bf16,
       max_grad_norm=max_grad_norm,
       max_steps=max_steps,
       warmup_ratio=warmup_ratio,
       group_by_length=group_by_length,
       lr_scheduler_type=lr_scheduler_type,
   )
   
   trainer = SFTTrainer(
       model=model,
       train_dataset=transformed,
       peft_config=peft_config,
       dataset_text_field="text",
       max_seq_length=max_seq_length,
       tokenizer=tokenizer,
       args=training_arguments,
       packing=packing,
   )
   
   trainer.train()
   
   trainer.model.save_pretrained(new_model)
   
   # Reload model in FP16 and merge it with LoRA weights
   base_model = AutoModelForCausalLM.from_pretrained(
       model_name,
       low_cpu_mem_usage=True,
       return_dict=True,
       torch_dtype=torch.float16,
       device_map=device_map,
   )
   model = PeftModel.from_pretrained(base_model, new_model)
   model = model.merge_and_unload()
   
   # push to Cloud Storage
   
   file_path_to_save_the_model = '/finetune/new_model'
   model.save_pretrained(file_path_to_save_the_model)
   tokenizer.save_pretrained(file_path_to_save_the_model)
   

3. 创建 requirements.txt 文件：

   accelerate==0.34.2
   bitsandbytes==0.45.5
   datasets==2.19.1
   transformers==4.51.3
   peft==0.11.1
   trl==0.8.6
   torch==2.3.0
   

4. 创建 Dockerfile：

   FROM nvidia/cuda:12.6.2-runtime-ubuntu22.04
   
   RUN apt-get update && \
       apt-get -y --no-install-recommends install python3-dev gcc python3-pip git && \
       rm -rf /var/lib/apt/lists/*
   
   COPY requirements.txt /requirements.txt
   
   RUN pip3 install -r requirements.txt --no-cache-dir
   
   COPY finetune.py /finetune.py
   
   ENV PYTHONUNBUFFERED 1
   
   CMD python3 /finetune.py --device cuda
   

5. 在 Artifact Registry 代码库中构建容器：

   gcloud builds submit \
     --tag $REGION-docker.pkg.dev/$PROJECT_ID/$AR_REPO/$IMAGE_NAME \
     --region $REGION
   

5. 部署和执行作业

在此步骤中，您将为作业创建 YAML 配置，并使用直接 VPC 出站流量，以便更快地将数据上传到 Google Cloud Storage。

请注意，此文件包含您将在后续步骤中更新的变量。

1. 创建名为 finetune-job.yaml.tmpl 的文件：

   apiVersion: run.googleapis.com/v1
   kind: Job
   metadata:
     name: $JOB_NAME
     labels:
       cloud.googleapis.com/location: $REGION
     annotations:
       run.googleapis.com/launch-stage: ALPHA
   spec:
     template:
       metadata:
         annotations:
           run.googleapis.com/execution-environment: gen2
           run.googleapis.com/network-interfaces: '[{"network":"default","subnetwork":"default"}]'
       spec:
         parallelism: 1
         taskCount: 1
         template:
           spec:
             serviceAccountName: $SERVICE_ACCOUNT_ADDRESS
             containers:
             - name: $IMAGE_NAME
               image: $REGION-docker.pkg.dev/$PROJECT_ID/$AR_REPO/$IMAGE_NAME
               env:
               - name: MODEL_NAME
                 value: "google/gemma-2b"
               - name: NEW_MODEL
                 value: "gemma-2b-sql-finetuned"
               - name: BUCKET_NAME
                 value: "$BUCKET_NAME"
               - name: LORA_R
                 value: "8"
               - name: LORA_ALPHA
                 value: "16"
               - name: GRADIENT_ACCUMULATION_STEPS
                 value: "2"
               - name: DATASET_LIMIT
                 value: "1000"
               - name: LOGGING_STEPS
                 value: "5"
               - name: HF_TOKEN
                 valueFrom:
                   secretKeyRef:
                     key: 'latest'
                     name: HF_TOKEN
               resources:
                 limits:
                   cpu: 8000m
                   nvidia.com/gpu: '1'
                   memory: 32Gi
               volumeMounts:
               - mountPath: /finetune/new_model
                 name: finetuned_model
             volumes:
             - name: finetuned_model
               csi:
                 driver: gcsfuse.run.googleapis.com
                 readOnly: false
                 volumeAttributes:
                   bucketName: $BUCKET_NAME
             maxRetries: 3
             timeoutSeconds: '3600'
             nodeSelector:
               run.googleapis.com/accelerator: nvidia-l4
   

2. 运行以下命令，将 YAML 中的变量替换为您的环境变量：

   envsubst < finetune-job.yaml.tmpl > finetune-job.yaml
   

3. 创建 Cloud Run 作业：

   gcloud alpha run jobs replace finetune-job.yaml
   

4. 执行作业：

   gcloud alpha run jobs execute $JOB_NAME --region $REGION --async
   

该作业大约需要 10 分钟才能完成。您可以使用上一个命令的输出中提供的链接来查看状态。

6. 使用 Cloud Run 服务通过 vLLM 提供经过微调的模型

在此步骤中，您将部署一个 Cloud Run 服务。此配置使用直接 VPC 通过专用网络访问 Cloud Storage 存储分区，以加快下载速度。

请注意，此文件包含您将在后续步骤中更新的变量。

1. 创建 service.yaml.tmpl 文件：

   apiVersion: serving.knative.dev/v1
   kind: Service
   metadata:
     name: serve-gemma-sql
     labels:
       cloud.googleapis.com/location: $REGION
     annotations:
       run.googleapis.com/launch-stage: BETA
       run.googleapis.com/ingress: all
       run.googleapis.com/ingress-status: all
   spec:
     template:
       metadata:
         labels:
         annotations:
           autoscaling.knative.dev/maxScale: '1'
           run.googleapis.com/cpu-throttling: 'false'
           run.googleapis.com/gpu-zonal-redundancy-disabled: 'true'
           run.googleapis.com/network-interfaces: '[{"network":"default","subnetwork":"default"}]'
       spec:
         containers:
         - name: serve-finetuned
           image: us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20250505_0916_RC00
           ports:
           - name: http1
             containerPort: 8000
           resources:
             limits:
               cpu: 8000m
               nvidia.com/gpu: '1'
               memory: 32Gi
           volumeMounts:
           - name: fuse
             mountPath: /finetune/new_model
           command: ["python3", "-m", "vllm.entrypoints.api_server"]
           args:
           - --model=/finetune/new_model
           - --tensor-parallel-size=1
           env:
           - name: MODEL_ID
             value: 'new_model'
           - name: HF_HUB_OFFLINE
             value: '1'
         volumes:
         - name: fuse
           csi:
             driver: gcsfuse.run.googleapis.com
             volumeAttributes:
               bucketName: $BUCKET_NAME
         nodeSelector:
           run.googleapis.com/accelerator: nvidia-l4
   

2. 使用您的存储分区名称更新 service.yaml 文件。

   envsubst < service.yaml.tmpl > service.yaml
   

3. 部署 Cloud Run 服务：

   gcloud alpha run services replace service.yaml
   

7. 测试微调后的模型

在此步骤中，您将提示模型测试微调。

1. 获取 Cloud Run 服务的服务网址：

   SERVICE_URL=$(gcloud run services describe serve-gemma-sql --platform managed --region $REGION --format 'value(status.url)')
   

2. 为模型创建提示。

   USER_PROMPT="Question: What are the first name and last name of all candidates? Context: CREATE TABLE candidates (candidate_id VARCHAR); CREATE TABLE people (first_name VARCHAR, last_name VARCHAR, person_id VARCHAR)"
   

3. 使用 C网址 调用您的服务以提示模型：

   curl -X POST $SERVICE_URL/generate \
     -H "Content-Type: application/json" \
     -H "Authorization: bearer $(gcloud auth print-identity-token)" \
     -d @- <<EOF
   {
       "prompt": "${USER_PROMPT}"
   }
   EOF
   

您应该会看到如下所示的响应：

{"predictions":["Prompt:\nQuestion: What are the first name and last name of all candidates? Context: CREATE TABLE candidates (candidate_id VARCHAR); CREATE TABLE people (first_name VARCHAR, last_name VARCHAR, person_id VARCHAR)\nOutput:\n CREATE TABLE people_to_candidates (candidate_id VARCHAR, person_id VARCHAR) CREATE TABLE people_to_people (person_id VARCHAR, person_id VARCHAR) CREATE TABLE people_to_people_to_candidates (person_id VARCHAR, candidate_id"]}

8. 恭喜！

恭喜您完成此 Codelab！

建议您参阅 Cloud Run 文档。

所学内容

• 如何使用 Cloud Run 作业 GPU 进行微调
• 如何使用 Cloud Run 搭配 vLLM 提供模型
• 如何为 GPU 作业使用直接 VPC 配置，以更快地上传和分发模型

9. 清理

为避免意外产生费用，例如，如果 Cloud Run 服务的意外调用次数超出了免费层级的 Cloud Run 调用次数配额，您可以删除在第 6 步中创建的 Cloud Run 服务。

如需删除 Cloud Run 服务，请前往 Cloud Run Cloud 控制台 (https://console.cloud.google.com/run)，然后删除 serve-gemma-sql 服务。

如需删除整个项目，请前往管理资源，选择您在第 2 步中创建的项目，然后选择“删除”。如果您删除该项目，则需要在 Cloud SDK 中更改项目。您可以通过运行 gcloud projects list 来查看所有可用项目的列表。