1. 项目背景与核心价值得物作为国内领先的潮流电商平台其商品数据对于市场分析、价格监控、选品决策具有重要价值。通过Python构建商品详情采集接口能够实现实时获取商品基础信息标题、价格、销量动态监测库存状态与促销活动结构化存储SKU规格参数为竞品分析提供数据支撑这个方案特别适合电商从业者进行市场调研开发者构建比价工具数据分析师采集样本数据注意所有数据采集行为需遵守平台robots.txt协议禁止高频请求影响服务器正常运行2. 技术方案设计2.1 接口逆向分析通过浏览器开发者工具分析得物商品页网络请求可以发现核心数据接口特征接口地址https://www.dewu.com/api/v1/goods/detail请求方式GET/POST关键参数goodsId: 商品唯一标识timestamp: 请求时间戳sign: 动态签名参数2.2 签名算法破解经过多次请求对比测试发现sign参数生成规律import hashlib import time def generate_sign(goods_id): timestamp str(int(time.time() * 1000)) raw_str fgoodsId{goods_id}timestamp{timestamp}key得物固定密钥 return hashlib.md5(raw_str.encode()).hexdigest()实操技巧使用Fiddler抓包工具可以自动重放请求测试签名有效性2.3 反爬应对策略得物采用的多层防御机制包括请求头校验User-Agent/CookieIP访问频率限制行为验证码触发解决方案headers { User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36, Referer: https://www.dewu.com/, Cookie: 从浏览器复制有效cookie }3. 完整代码实现3.1 基础请求模块import requests import json from urllib.parse import quote class DewuAPI: def __init__(self): self.session requests.Session() self.base_url https://www.dewu.com/api/v1 self.headers { User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64), X-Requested-With: XMLHttpRequest } def get_goods_detail(self, goods_id): api_path /goods/detail timestamp str(int(time.time() * 1000)) sign self._generate_sign(goods_id, timestamp) params { goodsId: goods_id, timestamp: timestamp, sign: sign } response self.session.get( self.base_url api_path, paramsparams, headersself.headers ) if response.status_code 200: return response.json() else: raise Exception(f请求失败: {response.status_code})3.2 数据解析模块典型返回数据结构示例{ code: 0, data: { goodsInfo: { goodsId: 123456, title: Nike Air Force 1 纯白, price: 899, originalPrice: 999, stock: 42, sales: 1560, specs: [ {name: 尺码, values: [38, 39, 40]}, {name: 颜色, values: [纯白]} ] } } }关键字段提取方法def parse_goods_data(json_data): goods json_data[data][goodsInfo] return { 商品ID: goods[goodsId], 标题: goods[title].strip(), 现价: float(goods[price])/100, 原价: float(goods[originalPrice])/100, 销量: goods[sales], 库存: goods[stock], 规格: {spec[name]: spec[values] for spec in goods[specs]} }4. 高级功能扩展4.1 异步并发采集使用aiohttp提升采集效率import aiohttp import asyncio async def fetch_goods(session, goods_id): async with session.get(API_URL, paramsbuild_params(goods_id)) as resp: return await resp.json() async def batch_fetch(goods_ids): async with aiohttp.ClientSession(headersHEADERS) as session: tasks [fetch_goods(session, gid) for gid in goods_ids] return await asyncio.gather(*tasks)4.2 数据持久化方案MongoDB存储示例from pymongo import MongoClient class DataSaver: def __init__(self): self.client MongoClient(mongodb://localhost:27017/) self.db self.client[dewudb] def save_goods(self, goods_data): collection self.db[goods] collection.update_one( {goodsId: goods_data[商品ID]}, {$set: goods_data}, upsertTrue )5. 常见问题排查5.1 请求返回403错误可能原因及解决方案Cookie失效 - 重新登录获取新CookieIP被封禁 - 使用代理IP轮换请求头不完整 - 补全Referer等必要字段5.2 数据字段缺失处理方案def safe_get(data, keys, defaultNone): try: for key in keys.split(.): data data[key] return data except (KeyError, TypeError): return default # 使用示例 sales safe_get(response_json, data.goodsInfo.sales, 0)5.3 验证码触发应对策略降低请求频率建议间隔2秒以上模拟鼠标移动轨迹使用自动化工具处理验证码需评估法律风险6. 性能优化建议连接池配置adapter requests.adapters.HTTPAdapter( pool_connections10, pool_maxsize50, max_retries3 ) session.mount(https://, adapter)缓存机制实现from diskcache import Cache cache Cache(request_cache) cache.memoize(expire3600) def get_goods_cached(goods_id): return get_goods_detail(goods_id)异常重试逻辑from tenacity import retry, stop_after_attempt retry(stopstop_after_attempt(3)) def safe_request(url): response requests.get(url) response.raise_for_status() return response
1. 项目背景与核心价值得物作为国内领先的潮流电商平台其商品数据对于市场分析、价格监控、选品决策具有重要价值。通过Python构建商品详情采集接口能够实现实时获取商品基础信息标题、价格、销量动态监测库存状态与促销活动结构化存储SKU规格参数为竞品分析提供数据支撑这个方案特别适合电商从业者进行市场调研开发者构建比价工具数据分析师采集样本数据注意所有数据采集行为需遵守平台robots.txt协议禁止高频请求影响服务器正常运行2. 技术方案设计2.1 接口逆向分析通过浏览器开发者工具分析得物商品页网络请求可以发现核心数据接口特征接口地址https://www.dewu.com/api/v1/goods/detail请求方式GET/POST关键参数goodsId: 商品唯一标识timestamp: 请求时间戳sign: 动态签名参数2.2 签名算法破解经过多次请求对比测试发现sign参数生成规律import hashlib import time def generate_sign(goods_id): timestamp str(int(time.time() * 1000)) raw_str fgoodsId{goods_id}timestamp{timestamp}key得物固定密钥 return hashlib.md5(raw_str.encode()).hexdigest()实操技巧使用Fiddler抓包工具可以自动重放请求测试签名有效性2.3 反爬应对策略得物采用的多层防御机制包括请求头校验User-Agent/CookieIP访问频率限制行为验证码触发解决方案headers { User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36, Referer: https://www.dewu.com/, Cookie: 从浏览器复制有效cookie }3. 完整代码实现3.1 基础请求模块import requests import json from urllib.parse import quote class DewuAPI: def __init__(self): self.session requests.Session() self.base_url https://www.dewu.com/api/v1 self.headers { User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64), X-Requested-With: XMLHttpRequest } def get_goods_detail(self, goods_id): api_path /goods/detail timestamp str(int(time.time() * 1000)) sign self._generate_sign(goods_id, timestamp) params { goodsId: goods_id, timestamp: timestamp, sign: sign } response self.session.get( self.base_url api_path, paramsparams, headersself.headers ) if response.status_code 200: return response.json() else: raise Exception(f请求失败: {response.status_code})3.2 数据解析模块典型返回数据结构示例{ code: 0, data: { goodsInfo: { goodsId: 123456, title: Nike Air Force 1 纯白, price: 899, originalPrice: 999, stock: 42, sales: 1560, specs: [ {name: 尺码, values: [38, 39, 40]}, {name: 颜色, values: [纯白]} ] } } }关键字段提取方法def parse_goods_data(json_data): goods json_data[data][goodsInfo] return { 商品ID: goods[goodsId], 标题: goods[title].strip(), 现价: float(goods[price])/100, 原价: float(goods[originalPrice])/100, 销量: goods[sales], 库存: goods[stock], 规格: {spec[name]: spec[values] for spec in goods[specs]} }4. 高级功能扩展4.1 异步并发采集使用aiohttp提升采集效率import aiohttp import asyncio async def fetch_goods(session, goods_id): async with session.get(API_URL, paramsbuild_params(goods_id)) as resp: return await resp.json() async def batch_fetch(goods_ids): async with aiohttp.ClientSession(headersHEADERS) as session: tasks [fetch_goods(session, gid) for gid in goods_ids] return await asyncio.gather(*tasks)4.2 数据持久化方案MongoDB存储示例from pymongo import MongoClient class DataSaver: def __init__(self): self.client MongoClient(mongodb://localhost:27017/) self.db self.client[dewudb] def save_goods(self, goods_data): collection self.db[goods] collection.update_one( {goodsId: goods_data[商品ID]}, {$set: goods_data}, upsertTrue )5. 常见问题排查5.1 请求返回403错误可能原因及解决方案Cookie失效 - 重新登录获取新CookieIP被封禁 - 使用代理IP轮换请求头不完整 - 补全Referer等必要字段5.2 数据字段缺失处理方案def safe_get(data, keys, defaultNone): try: for key in keys.split(.): data data[key] return data except (KeyError, TypeError): return default # 使用示例 sales safe_get(response_json, data.goodsInfo.sales, 0)5.3 验证码触发应对策略降低请求频率建议间隔2秒以上模拟鼠标移动轨迹使用自动化工具处理验证码需评估法律风险6. 性能优化建议连接池配置adapter requests.adapters.HTTPAdapter( pool_connections10, pool_maxsize50, max_retries3 ) session.mount(https://, adapter)缓存机制实现from diskcache import Cache cache Cache(request_cache) cache.memoize(expire3600) def get_goods_cached(goods_id): return get_goods_detail(goods_id)异常重试逻辑from tenacity import retry, stop_after_attempt retry(stopstop_after_attempt(3)) def safe_request(url): response requests.get(url) response.raise_for_status() return response
