【202504】Wan-01-数据篇：数十亿视频和图像组成的数据集

wan: Open and Advanced Large-Scale Video Generative ModelsWan：开放且先进的大规模视频生成模型Wan Team, Alibaba Group Wan 团队，Alibaba Group3 Data Processing Pipeline3 数据处理流水线High-quality data is essential for training large generative models, and an automated data construction pipeline significantly enhances the efficiency of the training process. In developing our dataset, we prioritized three core principles: high quality, high diversity, and substantial scale. Following these principles, we curated a dataset comprisingbillions of videos and images. This section offers a detailed introduction to the data construction pipeline employed for Wan.高质量数据对于训练大型生成模型至关重要，而自动化的数据构建流水线显著提升了训练过程的效率。在开发我们的数据集时，我们优先遵循三个核心原则：高质量、高多样性大规模。基于这些原则，我们构建了一个由数十亿视频和图像组成的数据集。本节将详细介绍 Wan 所采用的数据构建流水线。We curated and deduplicated a candidate dataset sourced from both internal copyrighted sources and publicly accessible data. In the pre-training stage, our goal is to select high-quality and diverse data from this expansive yet noisy dataset to facilitate effective training. Throughout the data mining process, we designed a four-step data cleaning process, focusing on fundamental dimensions, visual quality, and motion quality. Subsequently, we will also highlight our data processing workflow for constructing visual text data.我们从内部版权来源和公开可访问的数据中整理并去重了一个候选数据集。在预训练阶段，我们的目标是从这个规模庞大但噪声较多的数据集中筛选出高质量且多样化的数据，以促进有效训练。在整个数据挖掘过程中，我们设计了一个四步数据清洗流程，重点关注基础维度、视觉质量运动质量。随后，我们还将重点介绍我们用于构建视觉文本数据的数据处理工作流。Fundamental dimensions. The fundamental dimensions of our data filtering framework focus on the intrinsic attributes of the source video and image data, enabling efficient preliminary filtering out of all the unsuitable data. Specifically, our multidimensional filtering approach encompasses the following critical aspects: Text detection. A lightweight OCR detector is implemented to quantify text coverage ratios, effectively excluding videos and images with excessive textual elements to maintain visual clarity. Aesthetic evaluation: We use the widely adopted LAION-⋅5B^ { \cdot 5 \mathrm { B } }⋅5B(Schuhmann et al., 2021) aesthetic classifier to perform an initial quality assessment of our images, quickly filtering out lowquality data. NSFW Score. Through our internal safety assessment model, we systematically evaluate and filter inappropriate content based on computed NSFW scores in all training data. Watermark and logo detection. We detect whether the video or images contain watermarks and logos, and crop these elements during training. Black border detection. Utilizing heuristic-based detection methods, we automatically crop extraneous black borders to maintain focus on content-rich regions. Overexposure detection. Our trained expert classifier evaluates and filters out data with abnormal tonal distributions, ensuring optimal visual quality in the training dataset. Synthetic image detection. Empirical evidence indicates that even minimal contaminationKaTeX parse error: Undefined control sequence: \textless at position 3: ( \̲t̲e̲x̲t̲l̲e̲s̲s̲ ̲1 0 \%) by generated images can significantly degrade the performance of the model. Therefore, we train an expert classifier to filter out these “contaminating” images. Blur detection. An internally developed model assigns quantitative blur scores to training materials, enabling the systematic removal of visually indistinct content. Duration and resolution. We also enforce constraints where video duration must exceed 4 seconds, and resolution thresholds are applied at different training stages to filter out low-quality data. Through the implementation of these efficient preprocessing strategies, we successfully eliminated approximately50%5 0 \%50%of the initial dataset. The retained high-quality data subsequently progresses to a more superior semantic-driven selection stage for further refinement.基础维度。我们数据过滤框架的基础维度聚焦于源视频和图像数据的内在属性，从而能够高效地初步过滤掉所有不合适的数据。具体而言，我们的多维过滤方法涵盖以下关键方面：文本检测。我们实现了一个轻量级 OCR 检测器来量化文本覆盖率，有效排除包含过多文本元素的视频和图像，以保持视觉清晰度。美学评估：我们使用被广泛采用的 LAION-5B(Schuhmann et al., 2021) 美学分类器，对图像进行初步质量评估，快速过滤低质量数据。NSFW 分数。通过我们的内部安全评估模型，我们基于计算得到的 NSFW 分数，对所有训练数据中的不当内容进行系统性评估和过滤。水印和 logo 检测。我们检测视频或图像中是否包含水印和 logo，并在训练过程中裁剪这些元素。黑边检测。利用基于启发式的检测方法，我们自动裁剪多余的黑边，以保持对内容丰富区域的关注。过曝检测。我们训练的专家分类器会评估并过滤掉色调分布异常的数据，确保训练数据集具有最佳视觉质量。合成图像检测。实证表明，即使是极少量（