FBA出单资料、实体生成资料、全套资料实体抽取落库及应用
需求详细文档:https://alidocs.dingtalk.com/i/nodes/oP0MALyR8k7y4Br5hYYAg4AD83bzYmDO?utm_scene=person_space
一、需求背景
在新架构标题生成过程中，需支持根据不同的输入资料来源进行差异化处理，以满足多场景下的实体抽取与应用需求。
结论：FBA出单资料、实体生成、全套资料优于首套资料实体，数据证明：
足7天出单率方向上实体替换组高于未进行实体替换的基准组，提39.70% 绝对 +0.00532 个百分点，呈现正向趋势（足7天观察期）。
备注：基准组为算法版本的首套资料实体抽取
二、实体生成类型定义
全套资料
三、实体抽取类型定义
见：实体分类类型划分：https://alidocs.dingtalk.com/i/nodes/Y1OQX0akWm30rO4Atbqk6gDGJGlDd3mE?utm_scene=person_space
四、服务修改说明
需求优化：
基于 https://pm.zhcxkj.com/zentao/story-view-36114.html 的需求基础上进行修改。36114 需求的核心是：AI直发通实体抽取应用，将实体来源从单一首套扩展为首套、全套可选，并在刊登/刷新服务中新增实体类型字段。
1.本次需求进一步优化，在 “传全套” 场景下增加对描述资料的标准化清洗（参考 4.2 类型 1 步骤二）。
2.新增FBA出单资料、实体生成资料初始化及自动执行
3.新增业务指定任务表
4.1 新增指定任务表
新增 指定FBA链接进行实体抽取表，
支持业务针对特定的SiteCode + SKU组合，触发“FBA出单资料实体抽取”的任务
输入字段
● SiteCode
● SKU
● OrderSourceId
● OrderSourceSku
● 标题
● 五点
● 描述
4.2 自动服务执行类型
类型 1：FBA出单资料
● 步骤一：筛选并确定FBA链接，筛选满足以下条件的FBA链接：
➢ 链接销量大于0；
➢ 链接状态为在线；
➢ 所属店铺状态正常；
➢ 所属站点为英语站点（US、AU、CA、GB）；
➢ 所属部门为“智能研发”或“亚马逊平台运营中心”。
按 SiteCode + SKU + OrderSourceId + OrderSourceSku 聚合有效订单：
➢ 聚合后筛选销量大于0的链接；
➢ 每个 SiteCode + SKU 按销量降序，最多保留前5条链接。
● 步骤二：根据入选链接的 OrderSourceSku、OrderSourceId 调用接口http://amazonapi.zhcxkj.com/ListingInfoAttribute/GetMergeListingInfoAttributes 获取最新的：
➢ 标题；
➢ 五点描述；
➢ 产品描述。
产品描述按以下规则清洗：
1. 将转义的换行符还原为真实换行；
2. 解码HTML实体；
3. 移除HTML标签并保留段落换行；
4. 将不间断空格转换为普通空格；
5. 合并连续空白字符；
6. 删除空行及首尾空格；
7. 输出干净的纯文本描述。
def clean_product_desc(text):
if pd.isna(text):
return text
text = str(text).replace("n", "n").replace("r", "")
text = html.unescape(text)
soup = BeautifulSoup(text, "html.parser")
text = soup.get_text(separator="n")
text = text.replace("xa0", " ")
lines = [re.sub(r"s+", " ", line).strip() for line in text.splitlines()]
lines = [line for line in lines if line]
return "n".join(lines)
● 步骤三：以聚合后的 SiteCode + SKU 资料调用实体抽取批量接口 http://showdoc.zhcxkj.com/web//79/6726，接口返回实体。
● 步骤四：实体入库：
➢ 数据落到FBA出单资料实体库；
➢ 已存在的SiteCode + SKU 则覆盖；
➢ 新增实体的来源类型标记为“FBA出单资料”。
● 执行时机：
○ 首次全量执行：统计全部有效历史订单；预计处理约 33 万链接，23 万站点 SKU，英语站点（US、AU、CA、GB）
○ 增量执行：
➢ 每天执行一次；付款时间范围为昨天00:00至今天00:00，采用左闭右开区间；
➢ 仅处理该时间范围内有效订单的链接；
➢ 若该链接的站点SKU在系统中无历史实体抽取记录，则直接执行正常的资料抽取流程。若已有历史抽取记录，则检查历史已抽取的链接总数，若数量 ≥ 5，则本次不进行任何抽取操作，若数量 ＜ 5，则需触发补充抽取：将历史任务指定的链接（链接状态为在线，所属店铺状态正常，英语站点（US、AU、CA、GB））与昨日产生出单的链接（按销量降序取前5条，不足5条从历史任务表中补足5条），重新聚合资料并进行实体抽取，覆盖该站点SKU原有的实体数据。
类型 2：实体生成
● 步骤一：
1. 处理对象筛选筛选符合以下条件的SKU、客户ID为1、非组合SKU、非成人用品、非高危产品、非停产产品、非仿牌产品、产品终审通过、主类目复审通过。产品表仅用于筛选SKU及获取图片。标题、五点、描述、实体名称和产品中文名称统一通过多套资料接口获取。
2. 确认已有的实体信息
执行实体生成前，必须确认当前 SiteCode + SKU 对应的以下任务均已执行完成一次完整的流程：
➢ 英文全套资料实体抽取成功；
➢ FBA出单资料实体抽取成功。
排除“实体生成来源”，读取全套资料、FBA出单资料下四类实体
➢ Applicable Object
➢ Applicable Scenario
➢ Functional Features
➢ Target User Group
同一类型内按实体词去重。某个类型无数据时传空数组；四个类型均无数据时传空对象，并继续根据产品文本和图片生成。
confirmed_entities: {
Applicable Object: xxx,
Applicable Scenario: xxx,
Functional Features: xxx,
Target User Group: xxx
}
3. 获取产品图片：
➢ 从 rb_product.ImageUrl 获取产品首图；
➢ 从 rb_product_image 获取有效的非封面场景图，图片数量上限为 8 张（不含首图）。
● 处理规则：
➢ 首图固定排在第一位；
➢ 场景图排在首图之后；
➢ 图片URL为空或无效时丢弃；
➢ 首图与场景图按URL去重；
➢ 无可用图片时，使用文本资料继续生成。
WITH product_image AS (
SELECT
U.SKU
,GROUP_CONCAT(CONCAT(", U.ImageUrl, ") SEPARATOR ,) AS ImageUrl
FROM zhcxkj_dw_bi.rb_product_image U
WHERE U.CustomerID = 1 AND U.Status != 2 AND U.IsCover != 1 AND U.IsScene = 1 AND EXISTS (
SELECT 1 FROM zhcxkj_bi.rb_amazon_sku_entity_information e where U.sku = e.sku)
GROUP BY
U.SKU
)
SELECT p.SKU sku,
p.ProductName,
-- p.ProductDescription productDesc,
TRIM(
REGEXP_REPLACE(
REGEXP_REPLACE(
REPLACE(
REPLACE(ProductDescription, , ),
<br />, n
),
<[>]+>,
),
n[[:space:]]*n+,
n
)
) AS productDesc,
p.FeatureList featureList,
p.AccurateKeyword product_entity_name,
p.ProductNameCN product_chinese_title,
PARSE_JSON(
CONCAT(
[,
CASE
WHEN p.ImageUrl IS NOT NULL AND p.ImageUrl <> THEN CONCAT(", p.ImageUrl, ")
ELSE
END,
CASE
WHEN p.ImageUrl IS NOT NULL AND p.ImageUrl <>
AND e.ImageUrl IS NOT NULL AND e.ImageUrl <> THEN ,
ELSE
END,
CASE
WHEN e.ImageUrl IS NOT NULL AND e.ImageUrl <> THEN e.ImageUrl
ELSE
END,
]
)
) AS ImageUrl
FROM zhcxkj_dw_bi.rb_product AS p
LEFT JOIN zhcxkj_dw_bi.rb_product_admin pa ON p.sku = pa.sku
LEFT JOIN product_image e ON e.SKU = p.SKU
WHERE p.CustomerID = 1
AND p.IsGroup = 0 -- 是否为组合SKU产品(0:不是,1:是)
AND p.IsAdultsOnly != 1 -- 非成人用品
AND p.RiskLevel != 1 -- 非高危
AND p.ProductState != 2 -- 非停产(去掉停产)
AND p.ProductProperty != 2 -- 去掉仿牌 【产品属性】 产品属性 0 未选择 1 非仿牌 2 仿牌 3 灰色产品
AND pa.CheckStatus = 1 -- 终审通过 审核状态 0 未审核 1 已审核 2 不通过
AND pa.MainCategoryCheckStatus = 3 -- 复审通过 主类目审核状态，初审状态 0:待初审 1:初审通过 2:初审不通过 3：复审通过 4：复审不通过
AND EXISTS (
SELECT 1 FROM zhcxkj_bi.rb_amazon_sku_entity_information e where p.sku = e.sku and e.source = 全套资料
)
;
● 步骤二：调用批量实体生成批量接口 http://showdoc.zhcxkj.com/web//79/6997。
入参说明：model_name 固定为 gpt-5。
[{
task_id: 15434076,
sku: ,
site_code: US,
productName: 3Pcs Rugby Cake Molds Fondant Cooking Molds Creative Rugby Molds Transparent Rugby Molds,
productDesc: DescriptionnMade of high-grade material, this baking mold is very durable. Nonstick surface is designed for easy release and cleaning. Decorative rugby shaped design can make the baked food more creative and attractive. Perfect for small rice cakes, custard cakes, cookies, chocolate, puddings, jelly and so on.nFeaturesn-Color:Transparentn-Material:ABSn- Size: 15.00X9.20X7.20cm/5.89X3.62X2.83in n- Size: 11.00X7.00X5.00cm/4.32X2.75X1.97in n- Size: 9.00X6.00X3.70cm/3.54X2.36X1.45in n-It is made of premium and sturdy material, sturdy and practical for a long-lasting use.n-Excellent heat conductivity and uniform baking to prevent hot spots and burning.n-Compact size and light in weight design, and this food making mold is easy to store and carry.n-Non-stick, this mold is easy and quick to demould with keeping the baked food intact.n-Use it to bake some cake, jelly, chocolate and rice cake for your families and friends.,
featureList: It is made of premium and sturdy material, sturdy and practical for a long-lasting use.nExcellent heat conductivity and uniform baking to prevent hot spots and burning.nCompact size and light in weight design, and this food making mold is easy to store and carry.nNon-stick, this mold is easy and quick to demould with keeping the baked food intact.nUse it to bake some cake, jelly, chocolate and rice cake for your families and friends.,
product_entity_name: 橄榄球模具,
product_chinese_title: 3D立体橄榄球巧克模具美式足球3种不同大小蛋糕糖果烘焙装饰,
image_urls: [https://irobotbox-hd1.oss-cn-hangzhou.aliyuncs.com/1/ProductImages/15434076/2022/11/08/87095784d0b44a9caaeaaa6982ee0714/63cc86b9-e110-4cf0-83b1-eab9a4bed2b0.jpg, https://irobotbox-hd1.oss-cn-hangzhou.aliyuncs.com/1/ProductImages/15434076/2022/11/08/e6b7e322c8374255bb5ac3a405018254/1dd6e612-08f5-45e5-bfab-853cbfe98286.jpg, https://irobotbox-hd1.oss-cn-hangzhou.aliyuncs.com/1/ProductImages/15434076/2022/11/08/e2a3c86888794c36a8d17abab65fe0bd/25bb178c-1a30-434e-812f-49bab6d742d0.jpg, https://irobotbox-hd1.oss-cn-hangzhou.aliyuncs.com/1/ProductImages/15434076/2022/11/08/e00cda487b2d4793b9e02ee827f911de/89fc9fcb-fbfd-4bb3-aa6c-1a5271d83f46.jpg, https://irobotbox-hd1.oss-cn-hangzhou.aliyuncs.com/1/ProductImages/15434076/2022/11/08/660cc72de5514e55b0cd1fbcc7db5502/bf78dbd7-b067-4324-be10-4fc708e083c0.jpg],
confirmed_entities: {
Applicable Object: [biscuits, chocolate, cookies, custard cakes, jelly, pudding, rice cake],
Applicable Scenario: [DIY baking, bakery, cake shop, dessert shop, home kitchen, hotel, restaurant],
Functional Features: [compact size, easy demoulding, easy to clean, easy to demold, easy to store, easy to tidy, healthy and practical, heat-resistant, high-temperature resistant, lightweight, non-stick, safe for food, uniform baking]
},
model_name: gpt-5,
competitor_image_urls: [],
competitor_data:
}
]
● 步骤三：清洗接口返回结果：
仅处理接口状态为成功的结果，并读取 expanded_entities
保留以下五种实体类型：
➢ Applicable Object
➢ Applicable Scenario
➢ Functional Features
➢ Purchase Motivation
➢ Target User Group
每个实体保留：
● 实体类型 value
● 实体词 value
● 置信度 confidence
{
data: {
expanded_entities: {
Applicable Object: [{
basis: Title explicitly states rebar marking; aligns with steel marking,
confidence: high,
risk: None,
source: [product_text, confirmed_entity],
value: rebar
}, {
basis: Marks on steel implies broader metal surface applicability,
confidence: medium,
risk: Generalization beyond steel may overreach,
source: [product_text, product_logic],
value: metal
}, {
basis: Marks on wood; timber is a common construction wood object,
confidence: medium,
risk: Subcategory wording not explicitly used,
source: [product_text, product_logic],
value: timber
}, {
basis: Concrete mentioned; construction site imagery suggests slab/floor layout marking,
confidence: medium,
risk: "Specific form slab not explicitly named",
source: [product_text, product_image],
value: concrete slab
}, {
basis: "Visual text says colored stone pencils; chalk often used on stone",
confidence: low,
risk: Product title/main text focus on industrial/steel/concrete; stone not confirmed,
source: [product_image, product_logic],
value: stone
}, {
basis: Industrial marking chalk commonly used to mark metal pipes,
confidence: low,
risk: Not explicitly supported by current text or images,
source: [product_logic],
value: pipes
}],
Applicable Scenario: [{
basis: Image shows construction workers on-site; text emphasizes industrial marking tasks,
confidence: high,
risk: Assumes primary use on construction rather than other industrial settings,
source: [product_image, product_text, product_logic],
value: construction site
}, {
basis: Text mentions rebar and steel marking; typical setting is fabrication or welding prep,
confidence: medium,
risk: Specific shop environment not explicitly named,
source: [product_text, product_logic],
value: metal fabrication shop
}, {
basis: Product marks on wood and provides clear layout lines,
confidence: medium,
risk: May overlap with general woodworking outside heavy industry,
source: [product_text, product_logic],
value: carpentry layout
}, {
basis: Temporary chalk labeling fits industrial warehouses,
confidence: low,
risk: Not directly stated; environment inference could be too broad,
source: [product_logic],
value: warehouse labeling
}],
Functional Features: [{
basis: Chalk markings are typically non-permanent and erasable,
confidence: medium,
risk: Reversibility not explicitly stated in text,
source: [product_logic],
value: temporary marks
}, {
basis: Described as large size producing clear, visible lines; stick breadth suggests wide strokes,
confidence: medium,
risk: Exact width or edge design not specified,
source: [product_text, product_image],
value: wide marking edge
}, {
basis: Concrete and wood are rough; chalk suitable for rough texture,
confidence: medium,
risk: Performance language could overclaim without direct text,
source: [product_text, product_logic],
value: works rough
}, {
basis: Chalk sticks are designed to be held and used manually,
confidence: high,
risk: Generic but accurate,
source: [product_text, product_image],
value: handheld use
}],
Purchase Motivation: [{
basis: Text highlights clear, visible, precise lines,
confidence: high,
risk: None,
source: [product_text],
value: clear layout lines
}, {
basis: Resistant to breakage and reinforced design,
confidence: high,
risk: Paraphrase but consistent,
source: [product_text],
value: won’t easily
}, {
basis: Sufficient pieces in one box; 18 pcs,
confidence: high,
risk: Value perception varies,
source: [product_text],
value: bulk value pack
}, {
basis: Marks on steel, concrete, wood; industrial environments,
confidence: medium,
risk: Broad phrasing,
source: [product_text],
value: works tough
}, {
basis: Smudge-resistant/reinforced design to prevent smudging,
confidence: high,
risk: None,
source: [product_text],
value: less smudging
}],
Target User Group: [{
basis: Image depicts construction crew; industrial marking for concrete/rebar aligns,
confidence: high,
risk: Assumes primary buyer within construction,
source: [product_image, product_text],
value: construction workers
}, {
basis: Rebar and steel marking suggests fabrication and welding prep users,
confidence: medium,
risk: Not explicitly named,
source: [product_text, product_logic],
value: metal fabricators
}, {
basis: Marks on wood for layout lines fits carpentry tasks,
confidence: medium,
risk: Could overlap with general DIY woodworkers,
source: [product_text, product_logic],
value: carpenters
}, {
basis: Chalk often used for temporary site marks during layout,
confidence: low,
risk: Not supported by text or images,
source: [product_logic],
value: surveyors
}]
},
product_analysis: {
core_product: Industrial marking chalk sticks (white, large size) for making lines on hard surfaces such as steel, concrete, and wood,
key_constraints: [Industrial-oriented use emphasized, White color only, Chalk-based, temporary markings],
solved_problem: Provides clear, visible, durable marks for layout and labeling in tough work environments without frequent breakage or smudging,
use_actions: [marking lines, labeling parts, layout marking, temporary surface marking],
visible_image_evidence: [Multiple white rectangular chalk sticks, Workers on a construction site wearing hard hats and vests]
},
rejected_candidates: [{
entity_type: Functional Features,
reason: No evidence of water-resistant or wet-surface performance,
value: wet surface marking
}, {
entity_type: Applicable Scenario,
reason: Product is industrial marking chalk for hard surfaces, not classroom chalkboards,
value: blackboard use
}, {
entity_type: Applicable Scenario,
reason: No fabric/sewing context; material and text focus on steel/concrete/wood,
value: tailor marking
}, {
entity_type: Target User Group,
reason: DIY audience not explicitly supported; industrial focus,
value: DIY enthusiasts
}, {
entity_type: Functional Features,
reason: Chalk is generally temporary; permanence not claimed,
value: permanent marking
}]
},
input_text: {
competitor_data: [],
competitor_image_urls: [],
confirmed_entities: {
Applicable Object: [concrete, steel, wood],
Applicable Scenario: [industrial applications, industrial use],
Functional Features: [abundant supply, accurate marking, breakage resistant, clear visible lines, durable, long-lasting performance, marks on concrete, marks on steel, marks on wood, precise marking, reliable performance, resistant to breakage, smudge resistant, smudge-resistant, suitable for industrial use, wear resistant]
},
featureList: "The Industrial Marking Chalk offers several key features that make it a standout choice. |Its large size allows for precise and accurate marking, ensuring your lines are clear and easy to follow. |The use of stable material guarantees durability, making it resistant to breakage even in demanding work environments. |The reinforced design ensures that the chalk can use with ease, preventing any accidental smudging or breaking. |With sufficient pieces in one box, youll have plenty of chalk to tackle any marking task.",
image_urls: [https://irobotbox-hd1.oss-cn-hangzhou.aliyuncs.com/1/ProductImages/19162867/2024/06/07/e82e200dd4934dcda71ff3c1a3f43ac5/967860fa-7c92-40ed-ab25-f8f5e627eefc.jpg, https://irobotbox-hd1.oss-cn-hangzhou.aliyuncs.com/1/ProductImages/19162867/2024/06/07/e2c500e4442d42cc9cf13e24218c11f4/7b842dde-c4fd-4992-a961-bedbcbffed9b.jpg],
productDesc: "DescriptionnThe Marking Chalk for Industrial is a versatile tool for precise marking and labeling. Its large size allows for clear and visible lines, making it perfect for industrial use. Made from durable material, this chalk is resistant to breakage and ensures long-lasting performance. The reinforced design prevents the chalk from easily broken up, providing a reliable marking solution. With bulk pieces in one box, youll have an abundant supply to meet your marking needs. nFeaturesn-Color:Whiten-Material:Chalkn-Size:10.00X1.20X0.40cm/3.93X0.47X0.16inn-The Industrial Marking Chalk offers several key features that make it a standout choice. n-Its large size allows for precise and accurate marking, ensuring your lines are clear and easy to follow. n-The use of stable material guarantees durability, making it resistant to breakage even in demanding work environments. n-The reinforced design ensures that the chalk can use with ease, preventing any accidental smudging or breaking. n-With sufficient pieces in one box, youll have plenty of chalk to tackle any marking task.",
productName: 18Pcs Industrial Marking Chalk Large Marking Tool Industrial Mark Chalk White Chalk Rebar Marking Chalk,
product_chinese_title: 记号画笔彩色石笔工业滑石大号划线钢筋粉笔（常用款一盒（100*12*4）毫米共18支）,
product_entity_name: 石笔粉笔
},
message: 资料外实体拓展成功,基础数据来源为请求输入,
site_code: US,
sku: ,
status: success,
task_id: 19162867
}
● 步骤四：实体入库：
● 已存在相同实体时不重复新增，但更新置信度
● 新实体正常新增
● 标记来源类型为 实体生成来源
● 不得覆盖全套资料或FBA出单资料来源的实体
● 执行时机：
○ 首次初始化
➢ 按US执行全量初始化；
○ 增量触发
➢ 全套资料实体集合变化，参考需求：https://pm.zhcxkj.com/zentao/story-view-34074.html
➢ 统一触发链路：全套资料抽取成功 → FBA资料抽取成功或无数据成功 → 实体生成
需支持的服务功能调整
针对以下两个服务进行功能扩展，使其支持上述 2 个来源的实体处理：
1. 指定 listing 刊登（含 SKU）
2. 指定 listing 刷新（含 SKU）
五、应用文档：
https://alidocs.dingtalk.com/i/nodes/o14dA3GK8g50BqLnFn2P0EL9V9ekBD76?utm_scene=person_space
全套资料的抽取，描述部分也要按照这里的描述清洗逻辑来执行
1. 将转义的换行符还原为真实换行；
2. 解码HTML实体；
3. 移除HTML标签并保留段落换行；
4. 将不间断空格转换为普通空格；
5. 合并连续空白字符；
6. 删除空行及首尾空格；
7. 输出干净的纯文本描述。