Stable Diffusion 原理与实战从DDPM到ControlNet可控生成一、引言2022年 Stable Diffusion 的发布引爆了 AI 图像生成革命。从 DDPM 扩散模型到 Stable Diffusion 的潜空间设计再到 ControlNet/IP-Adapter 的可控生成技术演进之快令人目不暇接。本文将从头推导扩散模型数学原理实现完整的 SD 推理管道并深入 LoRA/DreamBooth/ControlNet 三大可控技术。二、扩散模型数学原理2.1 前向扩散过程importtorchimportnumpyasnpimportmatplotlib.pyplotaspltdefforward_diffusion(x_0,beta,t): 前向扩散q(x_t | x_0) N(x_t; sqrt(ᾱ_t)·x_0, (1-ᾱ_t)·I) 从原始图像 x_0 逐步添加噪声得到 x_t # β 是每个时间步的噪声方差alphas1-beta alpha_barstorch.cumprod(alphas,dim0)# ᾱ_t ∏ α_s# 第t步的累积因子sqrt_alpha_bar_ttorch.sqrt(alpha_bars[t])sqrt_one_minus_alpha_bar_ttorch.sqrt(1-alpha_bars[t])# 采样噪声noisetorch.randn_like(x_0)# q(x_t | x_0)x_tsqrt_alpha_bar_t*x_0sqrt_one_minus_alpha_bar_t*noisereturnx_t,noise2.2 反向去噪过程defreverse_diffusion(model,x_t,t,beta,alpha_bars): 反向去噪p_θ(x_{t-1} | x_t) N(x_{t-1}; μ_θ(x_t,t), σ_t²·I) 模型预测噪声 ε_θ(x_t, t)然后通过以下公式恢复 x_{t-1} x_{t-1} 1/√α_t · (x_t - β_t/√(1-ᾱ_t) · ε_θ(x_t,t)) σ_t·z alpha_t1-beta[t]# 模型预测噪声predicted_noisemodel(x_t,t)# 计算均值coeff11.0/torch.sqrt(alpha_t)coeff2beta[t]/torch.sqrt(1-alpha_bars[t])meancoeff1*(x_t-coeff2*predicted_noise)# 添加随机噪声t1时ift1:noisetorch.randn_like(x_t)sigma_ttorch.sqrt(beta[t])else:noise0sigma_t0x_prevmeansigma_t*noisereturnx_prevdefddpm_sampling(model,img_size,timesteps1000):完整 DDPM 采样从纯噪声生成图像model.eval()# 从 N(0, I) 开始x_ttorch.randn(1,3,img_size,img_size)# 预计算 beta 调度betalinear_beta_schedule(timesteps)alpha1-beta alpha_barstorch.cumprod(alpha,dim0)# 从 T 到 1 逐步去噪fortinreversed(range(1,timesteps)):withtorch.no_grad():x_treverse_diffusion(model,x_t,t,beta,alpha_bars)ift%1000:print(fSampling step{timesteps-t}/{timesteps})# 最终图像t0returnx_t2.3 DDPM vs DDIMDDIM 以更少步数实现更快采样defddim_sampling(model,x_t,timesteps50,eta0.0):DDIM 快速采样只需要 50 步fortinreversed(range(1,timesteps1)):t_tensortorch.tensor([t]).long()# 预测噪声epsmodel(x_t,t_tensor)# DDIM 更新公式alpha_bar_talpha_bars[t]alpha_bar_prevalpha_bars[t-1]ift1else1.0# 预测原始图像 x_0pred_x0(x_t-torch.sqrt(1-alpha_bar_t)*eps)/torch.sqrt(alpha_bar_t)# DDIM 方向directiontorch.sqrt(1-alpha_bar_prev-sigma_t**2)*eps# 更新x_ttorch.sqrt(alpha_bar_prev)*pred_x0directionsigma_t*torch.randn_like(x_t)returnx_t三、Stable Diffusion 架构3.1 潜空间扩散Latent DiffusionSD 的核心创新在压缩的潜空间而非像素空间做扩散。fromdiffusersimportStableDiffusionPipelineimporttorch# 加载 SD 1.5pipeStableDiffusionPipeline.from_pretrained(runwayml/stable-diffusion-v1-5,torch_dtypetorch.float16,safety_checkerNone,)pipepipe.to(cuda)# 生成图像prompta serene lake at sunset, mountains in background, photorealisticnegative_promptblurry, low quality, distortedimagepipe(promptprompt,negative_promptnegative_prompt,num_inference_steps30,# DDIM步数guidance_scale7.5,# CFG引导强度width512,height512,generatortorch.Generator(cuda).manual_seed(42),).images[0]image.save(generated.png)3.2 SDXL 高清生成fromdiffusersimportStableDiffusionXLPipeline pipeStableDiffusionXLPipeline.from_pretrained(stabilityai/stable-diffusion-xl-base-1.0,torch_dtypetorch.float16,variantfp16,use_safetensorsTrue,)pipepipe.to(cuda)imagepipe(promptprofessional product photography of a futuristic smartwatch, studio lighting, 8k,negative_prompttext, watermark, low quality,num_inference_steps40,guidance_scale7.5,height1024,width1024,).images[0]四、LoRA 微调风格/角色定制fromdiffusersimportStableDiffusionPipelineimporttorch# 加载基础模型 LoRA权重pipeStableDiffusionPipeline.from_pretrained(runwayml/stable-diffusion-v1-5,torch_dtypetorch.float16).to(cuda)# 加载角色 LoRApipe.load_lora_weights(path/to/character_lora.safetensors)pipe.fuse_lora(lora_scale0.8)# 融合LoRAscale控制强度# 生成角色图像imagepipe(promptcharacter_name sitting in a coffee shop, detailed illustration,negative_promptbad anatomy, extra limbs,num_inference_steps30,guidance_scale7.0,).images[0]# 切换 LoRApipe.unfuse_lora()pipe.load_lora_weights(path/to/style_lora.safetensors)LoRA 训练脚本fromdiffusersimportStableDiffusionPipelinefrompeftimportLoraConfig,get_peft_modelimporttorch# 1. 准备训练数据10-20张图片 描述文本training_data[{image:person1.jpg,caption:a photo of sks person, portrait},{image:person2.jpg,caption:a photo of sks person, standing},# ...]# 2. LoRA 配置lora_configLoraConfig(r8,# LoRA秩lora_alpha16,target_modules[to_q,to_k,to_v,to_out.0],# 注意力层lora_dropout0.1,)# 3. 训练fromdiffusersimportDPMSolverMultistepScheduler pipe.schedulerDPMSolverMultistepScheduler.from_config(pipe.scheduler.config)forepochinrange(num_epochs):forbatchindataloader:# 编码图像到潜空间latentspipe.vae.encode(batch[image]).latent_dist.sample()latentslatents*pipe.vae.config.scaling_factor# 添加噪声noisetorch.randn_like(latents)timestepstorch.randint(0,pipe.scheduler.num_train_timesteps,(latents.shape[0],))noisy_latentspipe.scheduler.add_noise(latents,noise,timesteps)# 编码文本text_embedspipe.text_encoder(batch[caption])[0]# 预测噪声noise_predpipe.unet(noisy_latents,timesteps,text_embeds).sample# MSE 损失losstorch.nn.functional.mse_loss(noise_pred,noise)optimizer.zero_grad()loss.backward()optimizer.step()五、DreamBooth主体定制fromdiffusersimportStableDiffusionPipelinefromdiffusersimportDreamBoothTrainer,DreamBoothConfigclassDreamBoothTrainer:DreamBooth用少量图片学习新概念def__init__(self,instance_images_dir,class_images_dirNone):self.instance_dirinstance_images_dir# 3-5张目标图片self.class_dirclass_images_dir# 先验保留图片deftrain(self,instance_prompta photo of sks dog,class_prompta photo of a dog):训练步骤# 1. 先验保留损失Preservation Loss# L E[||ε - ε_θ(x_t, c_instance)||²]# λ · E[||ε - ε_θ(x_t, c_class)||²]# 2. 文本编码器微调# 为 sks 学习新的 token embedding# 3. UNet 微调# 同时更新 UNet 权重或仅更新注意力层passdefgenerate(self,pipe,prompt):使用训练好的 DreamBooth 生成pipe.load_dreambooth_lora(trained_model)returnpipe(prompt).images[0]# 使用 HuggingFace diffusers 的 DreamBoothfromdiffusersimportDiffusionPipeline pipeDiffusionPipeline.from_pretrained(sd-dreambooth-library/dog-sks)imagepipe(a photo of sks dog in a bucket).images[0]六、ControlNet可控生成6.1 各种控制条件fromdiffusersimportStableDiffusionControlNetPipeline,ControlNetModelfromdiffusers.utilsimportload_imagefromPILimportImageimportcv2importnumpyasnp# 加载 ControlNet 模型controlnets{canny:ControlNetModel.from_pretrained(lllyasviel/sd-controlnet-canny),depth:ControlNetModel.from_pretrained(lllyasviel/sd-controlnet-depth),pose:ControlNetModel.from_pretrained(lllyasviel/sd-controlnet-openpose),scribble:ControlNetModel.from_pretrained(lllyasviel/sd-controlnet-scribble),seg:ControlNetModel.from_pretrained(lllyasviel/sd-controlnet-seg),ip2p:ControlNetModel.from_pretrained(lllyasviel/sd-controlnet-ip2p),}# Canny 边缘控制pipeStableDiffusionControlNetPipeline.from_pretrained(runwayml/stable-diffusion-v1-5,controlnetcontrolnets[canny],torch_dtypetorch.float16).to(cuda)# 准备控制图像input_imageload_image(sketch.png).convert(RGB)canny_imagenp.array(input_image)canny_imagecv2.Canny(canny_image,100,200)canny_imagecanny_image[:,:,None]canny_imagenp.concatenate([canny_image,canny_image,canny_image],axis2)canny_imageImage.fromarray(canny_image)# 生成imagepipe(prompta beautiful house on a hill, detailed, 8k,imagecanny_image,num_inference_steps20,controlnet_conditioning_scale0.8,# 控制强度).images[0]6.2 多重 ControlNetfromdiffusersimportStableDiffusionControlNetPipeline# 同时使用 Canny Depthcontrolnet_cannyControlNetModel.from_pretrained(lllyasviel/sd-controlnet-canny)controlnet_depthControlNetModel.from_pretrained(lllyasviel/sd-controlnet-depth)pipeStableDiffusionControlNetPipeline.from_pretrained(runwayml/stable-diffusion-v1-5,controlnet[controlnet_canny,controlnet_depth],# 多个ControlNet!torch_dtypetorch.float16).to(cuda)imagepipe(prompta modern office interior,image[canny_image,depth_image],controlnet_conditioning_scale[0.7,0.5],).images[0]6.3 IP-Adapter参考图风格迁移fromdiffusersimportStableDiffusionPipelinefromdiffusersimportIPAdapter pipeStableDiffusionPipeline.from_pretrained(runwayml/stable-diffusion-v1-5,torch_dtypetorch.float16).to(cuda)# 加载 IP-Adapterpipe.load_ip_adapter(h94/IP-Adapter,subfoldermodels,weight_nameip-adapter-plus_sd15.safetensors)# 参考图风格来源style_imageload_image(van_gogh_style.jpg)imagepipe(prompta city street at night,ip_adapter_imagestyle_image,ip_adapter_scale0.6,# 风格强度num_inference_steps30,).images[0]七、推理优化importtorch# 1. xFormers 加速2-3xpipe.enable_xformers_memory_efficient_attention()# 2. FP16 推理pipepipe.to(dtypetorch.float16)# 3. CPU Offload降低显存pipe.enable_model_cpu_offload()# 4. VAE 切片pipe.enable_vae_slicing()# 5. 注意力切片pipe.enable_attention_slicing()# 6. Token Merging (ToMe) 加速importtomesd tomesd.apply_patch(pipe,ratio0.5)# 减少50% token# 性能数据 (512×512, RTX 3090):# 默认: ~4s, 8.5GB VRAM# xFormers: ~1.5s, 6.2GB VRAM# ToMe 0.5: ~1.0s, 5.5GB VRAM八、总结Stable Diffusion 技术栈全景DDPM→DDIM从 1000 步到 50 步采样Latent Diffusion在压缩潜空间工作效率革命LoRA10 张图即可定制风格/角色DreamBooth学习全新概念3-5张图ControlNet边缘/深度/姿态… 精确控制IP-Adapter参考图风格迁移xFormers/ToMe推理加速 2-3x
Stable Diffusion 原理与实战从DDPM到ControlNet可控生成一、引言2022年 Stable Diffusion 的发布引爆了 AI 图像生成革命。从 DDPM 扩散模型到 Stable Diffusion 的潜空间设计再到 ControlNet/IP-Adapter 的可控生成技术演进之快令人目不暇接。本文将从头推导扩散模型数学原理实现完整的 SD 推理管道并深入 LoRA/DreamBooth/ControlNet 三大可控技术。二、扩散模型数学原理2.1 前向扩散过程importtorchimportnumpyasnpimportmatplotlib.pyplotaspltdefforward_diffusion(x_0,beta,t): 前向扩散q(x_t | x_0) N(x_t; sqrt(ᾱ_t)·x_0, (1-ᾱ_t)·I) 从原始图像 x_0 逐步添加噪声得到 x_t # β 是每个时间步的噪声方差alphas1-beta alpha_barstorch.cumprod(alphas,dim0)# ᾱ_t ∏ α_s# 第t步的累积因子sqrt_alpha_bar_ttorch.sqrt(alpha_bars[t])sqrt_one_minus_alpha_bar_ttorch.sqrt(1-alpha_bars[t])# 采样噪声noisetorch.randn_like(x_0)# q(x_t | x_0)x_tsqrt_alpha_bar_t*x_0sqrt_one_minus_alpha_bar_t*noisereturnx_t,noise2.2 反向去噪过程defreverse_diffusion(model,x_t,t,beta,alpha_bars): 反向去噪p_θ(x_{t-1} | x_t) N(x_{t-1}; μ_θ(x_t,t), σ_t²·I) 模型预测噪声 ε_θ(x_t, t)然后通过以下公式恢复 x_{t-1} x_{t-1} 1/√α_t · (x_t - β_t/√(1-ᾱ_t) · ε_θ(x_t,t)) σ_t·z alpha_t1-beta[t]# 模型预测噪声predicted_noisemodel(x_t,t)# 计算均值coeff11.0/torch.sqrt(alpha_t)coeff2beta[t]/torch.sqrt(1-alpha_bars[t])meancoeff1*(x_t-coeff2*predicted_noise)# 添加随机噪声t1时ift1:noisetorch.randn_like(x_t)sigma_ttorch.sqrt(beta[t])else:noise0sigma_t0x_prevmeansigma_t*noisereturnx_prevdefddpm_sampling(model,img_size,timesteps1000):完整 DDPM 采样从纯噪声生成图像model.eval()# 从 N(0, I) 开始x_ttorch.randn(1,3,img_size,img_size)# 预计算 beta 调度betalinear_beta_schedule(timesteps)alpha1-beta alpha_barstorch.cumprod(alpha,dim0)# 从 T 到 1 逐步去噪fortinreversed(range(1,timesteps)):withtorch.no_grad():x_treverse_diffusion(model,x_t,t,beta,alpha_bars)ift%1000:print(fSampling step{timesteps-t}/{timesteps})# 最终图像t0returnx_t2.3 DDPM vs DDIMDDIM 以更少步数实现更快采样defddim_sampling(model,x_t,timesteps50,eta0.0):DDIM 快速采样只需要 50 步fortinreversed(range(1,timesteps1)):t_tensortorch.tensor([t]).long()# 预测噪声epsmodel(x_t,t_tensor)# DDIM 更新公式alpha_bar_talpha_bars[t]alpha_bar_prevalpha_bars[t-1]ift1else1.0# 预测原始图像 x_0pred_x0(x_t-torch.sqrt(1-alpha_bar_t)*eps)/torch.sqrt(alpha_bar_t)# DDIM 方向directiontorch.sqrt(1-alpha_bar_prev-sigma_t**2)*eps# 更新x_ttorch.sqrt(alpha_bar_prev)*pred_x0directionsigma_t*torch.randn_like(x_t)returnx_t三、Stable Diffusion 架构3.1 潜空间扩散Latent DiffusionSD 的核心创新在压缩的潜空间而非像素空间做扩散。fromdiffusersimportStableDiffusionPipelineimporttorch# 加载 SD 1.5pipeStableDiffusionPipeline.from_pretrained(runwayml/stable-diffusion-v1-5,torch_dtypetorch.float16,safety_checkerNone,)pipepipe.to(cuda)# 生成图像prompta serene lake at sunset, mountains in background, photorealisticnegative_promptblurry, low quality, distortedimagepipe(promptprompt,negative_promptnegative_prompt,num_inference_steps30,# DDIM步数guidance_scale7.5,# CFG引导强度width512,height512,generatortorch.Generator(cuda).manual_seed(42),).images[0]image.save(generated.png)3.2 SDXL 高清生成fromdiffusersimportStableDiffusionXLPipeline pipeStableDiffusionXLPipeline.from_pretrained(stabilityai/stable-diffusion-xl-base-1.0,torch_dtypetorch.float16,variantfp16,use_safetensorsTrue,)pipepipe.to(cuda)imagepipe(promptprofessional product photography of a futuristic smartwatch, studio lighting, 8k,negative_prompttext, watermark, low quality,num_inference_steps40,guidance_scale7.5,height1024,width1024,).images[0]四、LoRA 微调风格/角色定制fromdiffusersimportStableDiffusionPipelineimporttorch# 加载基础模型 LoRA权重pipeStableDiffusionPipeline.from_pretrained(runwayml/stable-diffusion-v1-5,torch_dtypetorch.float16).to(cuda)# 加载角色 LoRApipe.load_lora_weights(path/to/character_lora.safetensors)pipe.fuse_lora(lora_scale0.8)# 融合LoRAscale控制强度# 生成角色图像imagepipe(promptcharacter_name sitting in a coffee shop, detailed illustration,negative_promptbad anatomy, extra limbs,num_inference_steps30,guidance_scale7.0,).images[0]# 切换 LoRApipe.unfuse_lora()pipe.load_lora_weights(path/to/style_lora.safetensors)LoRA 训练脚本fromdiffusersimportStableDiffusionPipelinefrompeftimportLoraConfig,get_peft_modelimporttorch# 1. 准备训练数据10-20张图片 描述文本training_data[{image:person1.jpg,caption:a photo of sks person, portrait},{image:person2.jpg,caption:a photo of sks person, standing},# ...]# 2. LoRA 配置lora_configLoraConfig(r8,# LoRA秩lora_alpha16,target_modules[to_q,to_k,to_v,to_out.0],# 注意力层lora_dropout0.1,)# 3. 训练fromdiffusersimportDPMSolverMultistepScheduler pipe.schedulerDPMSolverMultistepScheduler.from_config(pipe.scheduler.config)forepochinrange(num_epochs):forbatchindataloader:# 编码图像到潜空间latentspipe.vae.encode(batch[image]).latent_dist.sample()latentslatents*pipe.vae.config.scaling_factor# 添加噪声noisetorch.randn_like(latents)timestepstorch.randint(0,pipe.scheduler.num_train_timesteps,(latents.shape[0],))noisy_latentspipe.scheduler.add_noise(latents,noise,timesteps)# 编码文本text_embedspipe.text_encoder(batch[caption])[0]# 预测噪声noise_predpipe.unet(noisy_latents,timesteps,text_embeds).sample# MSE 损失losstorch.nn.functional.mse_loss(noise_pred,noise)optimizer.zero_grad()loss.backward()optimizer.step()五、DreamBooth主体定制fromdiffusersimportStableDiffusionPipelinefromdiffusersimportDreamBoothTrainer,DreamBoothConfigclassDreamBoothTrainer:DreamBooth用少量图片学习新概念def__init__(self,instance_images_dir,class_images_dirNone):self.instance_dirinstance_images_dir# 3-5张目标图片self.class_dirclass_images_dir# 先验保留图片deftrain(self,instance_prompta photo of sks dog,class_prompta photo of a dog):训练步骤# 1. 先验保留损失Preservation Loss# L E[||ε - ε_θ(x_t, c_instance)||²]# λ · E[||ε - ε_θ(x_t, c_class)||²]# 2. 文本编码器微调# 为 sks 学习新的 token embedding# 3. UNet 微调# 同时更新 UNet 权重或仅更新注意力层passdefgenerate(self,pipe,prompt):使用训练好的 DreamBooth 生成pipe.load_dreambooth_lora(trained_model)returnpipe(prompt).images[0]# 使用 HuggingFace diffusers 的 DreamBoothfromdiffusersimportDiffusionPipeline pipeDiffusionPipeline.from_pretrained(sd-dreambooth-library/dog-sks)imagepipe(a photo of sks dog in a bucket).images[0]六、ControlNet可控生成6.1 各种控制条件fromdiffusersimportStableDiffusionControlNetPipeline,ControlNetModelfromdiffusers.utilsimportload_imagefromPILimportImageimportcv2importnumpyasnp# 加载 ControlNet 模型controlnets{canny:ControlNetModel.from_pretrained(lllyasviel/sd-controlnet-canny),depth:ControlNetModel.from_pretrained(lllyasviel/sd-controlnet-depth),pose:ControlNetModel.from_pretrained(lllyasviel/sd-controlnet-openpose),scribble:ControlNetModel.from_pretrained(lllyasviel/sd-controlnet-scribble),seg:ControlNetModel.from_pretrained(lllyasviel/sd-controlnet-seg),ip2p:ControlNetModel.from_pretrained(lllyasviel/sd-controlnet-ip2p),}# Canny 边缘控制pipeStableDiffusionControlNetPipeline.from_pretrained(runwayml/stable-diffusion-v1-5,controlnetcontrolnets[canny],torch_dtypetorch.float16).to(cuda)# 准备控制图像input_imageload_image(sketch.png).convert(RGB)canny_imagenp.array(input_image)canny_imagecv2.Canny(canny_image,100,200)canny_imagecanny_image[:,:,None]canny_imagenp.concatenate([canny_image,canny_image,canny_image],axis2)canny_imageImage.fromarray(canny_image)# 生成imagepipe(prompta beautiful house on a hill, detailed, 8k,imagecanny_image,num_inference_steps20,controlnet_conditioning_scale0.8,# 控制强度).images[0]6.2 多重 ControlNetfromdiffusersimportStableDiffusionControlNetPipeline# 同时使用 Canny Depthcontrolnet_cannyControlNetModel.from_pretrained(lllyasviel/sd-controlnet-canny)controlnet_depthControlNetModel.from_pretrained(lllyasviel/sd-controlnet-depth)pipeStableDiffusionControlNetPipeline.from_pretrained(runwayml/stable-diffusion-v1-5,controlnet[controlnet_canny,controlnet_depth],# 多个ControlNet!torch_dtypetorch.float16).to(cuda)imagepipe(prompta modern office interior,image[canny_image,depth_image],controlnet_conditioning_scale[0.7,0.5],).images[0]6.3 IP-Adapter参考图风格迁移fromdiffusersimportStableDiffusionPipelinefromdiffusersimportIPAdapter pipeStableDiffusionPipeline.from_pretrained(runwayml/stable-diffusion-v1-5,torch_dtypetorch.float16).to(cuda)# 加载 IP-Adapterpipe.load_ip_adapter(h94/IP-Adapter,subfoldermodels,weight_nameip-adapter-plus_sd15.safetensors)# 参考图风格来源style_imageload_image(van_gogh_style.jpg)imagepipe(prompta city street at night,ip_adapter_imagestyle_image,ip_adapter_scale0.6,# 风格强度num_inference_steps30,).images[0]七、推理优化importtorch# 1. xFormers 加速2-3xpipe.enable_xformers_memory_efficient_attention()# 2. FP16 推理pipepipe.to(dtypetorch.float16)# 3. CPU Offload降低显存pipe.enable_model_cpu_offload()# 4. VAE 切片pipe.enable_vae_slicing()# 5. 注意力切片pipe.enable_attention_slicing()# 6. Token Merging (ToMe) 加速importtomesd tomesd.apply_patch(pipe,ratio0.5)# 减少50% token# 性能数据 (512×512, RTX 3090):# 默认: ~4s, 8.5GB VRAM# xFormers: ~1.5s, 6.2GB VRAM# ToMe 0.5: ~1.0s, 5.5GB VRAM八、总结Stable Diffusion 技术栈全景DDPM→DDIM从 1000 步到 50 步采样Latent Diffusion在压缩潜空间工作效率革命LoRA10 张图即可定制风格/角色DreamBooth学习全新概念3-5张图ControlNet边缘/深度/姿态… 精确控制IP-Adapter参考图风格迁移xFormers/ToMe推理加速 2-3x
