任务描述

1.问题分析

智能刊登慢sql top 10, 大量的这种慢sql


2.解决方案

2.1分析发现sql有使用索引,导致慢的原因是并发高和表本身数据比较多,索引不是最优的,从业务上考虑数据逻辑闭环,将同步成功status=1的数据,删除或归档处理


select count(0) from M_Mercadolibre_Listing_ProductId                                     -- 82292489
select count(0) from M_Mercadolibre_Listing_ProductId where CustomerID=1 and  `Status`=1  -- 20928532  确认业务:删除或归档处理


2.2 看这个业务应该是每次批量获取50条数据,进行同步:

     优化:可以去掉排序和Limit 0,50 改为ID查询(`Id` > 557424057  and `Id` < 557424057  + 50)也是批量获取数据,默认是ID升序的,优化后速度会非常快,从100秒+到 3秒,因ID连续但这个查询条件查询出的ID是递增但是不连续,有可能=50查不出数据,建议+ 100000(`Id` > 557424057  and `Id` < 557424057  + 100000),这个查询条件一共只有14632条数据,也可以一次查询出来,具体业务根据实际情况定,这个方案让sql走了最优索引查询的快,做一个指引 如:



SELECT
`CustomerId`,
`OrderSourceId`,
`ProductId`,
`Status`,
`FailureLog`,
`Date`,
`Time`,
`SiteStatus`,
`SiteProductId`,
`GlobalStatus`,
`LastGlobalFinishedTime`,
`MarketStatus`,
`LastMarketFinishedTime`,
`Remark`,
`LastModifierId`,
`LastModifier`,
`LastModificationTime`,
`CreatorId`,
`Creator`,
`CreationTime`,
`Id` 
FROM
`M_Mercadolibre_Listing_ProductId` 
WHERE
((((((
`Id` > 557424057  and `Id` < 557424057  + 50
) 
AND ( `CustomerId` = 1 )) 
AND ( `OrderSourceId` = 11744 )) 
AND ( `SiteStatus` = 'success' )) 
AND ( `GlobalStatus` = 0 )) 
AND ( `Time` < TIMESTAMP ( '2023-04-07 08:39:32.546924' ) )) 


3.sql

SELECT
`CustomerId`,
`OrderSourceId`,
`ProductId`,
`Status`,
`FailureLog`,
`Date`,
`Time`,
`SiteStatus`,
`SiteProductId`,
`GlobalStatus`,
`LastGlobalFinishedTime`,
`MarketStatus`,
`LastMarketFinishedTime`,
`Remark`,
`LastModifierId`,
`LastModifier`,
`LastModificationTime`,
`CreatorId`,
`Creator`,
`CreationTime`,
`Id`
FROM
`M_Mercadolibre_Listing_ProductId`
WHERE
((((((
`Id` > 557424057
)
AND ( `CustomerId` = 1 ))
AND ( `OrderSourceId` = 11744 ))
AND ( `SiteStatus` = 'success' ))
AND ( `GlobalStatus` = 0 ))
AND ( `Time` < TIMESTAMP ( '2023-04-07 08:39:32.546924' ) ))
ORDER BY
Id ASC
LIMIT 0,
50

历史记录  
  1. 2023-04-07 14:16:14, 由 陈少军 创建。
  2. 2023-04-07 14:35:02, 由 陈少军 编辑。
    修改了 任务描述,区别为:
    005- 分析发现sql有使用索引,导致慢的原因是并发高和表本身数据比较多,将同步成功status=1的数据,删除或归档处理
    005+ 2.1分析发现sql有使用索引,导致慢的原因是并发高和表本身数据比较多,将同步成功status=1的数据,删除或归档处理
    007- select count(0) from M_Mercadolibre_Listing_ProductId -- 82292489 select count(0) from M_Mercadolibre_Listing_ProductId where CustomerID=1 and `Status`=1 -- 20928532 确认业务:删除或归档处理
    007+ select count(0) from M_Mercadolibre_Listing_ProductId -- 82292489 select count(0) from M_Mercadolibre_Listing_ProductId where CustomerID=1 and `Status`=1 -- 20928532 确认业务:删除或归档处理
    009- 3.sql
    009+ 2.2 看看业务是否可以去掉排序和Limit 0.50 改为ID查询,速度会非常快,从100秒+到 3秒 如:
    010- SELECT `CustomerId`, `OrderSourceId`, `ProductId`, `Status`, `FailureLog`, `Date`, `Time`, `SiteStatus`, `SiteProductId`, `GlobalStatus`, `LastGlobalFinishedTime`, `MarketStatus`, `LastMarketFinishedTime`, `Remark`, `LastModifierId`, `LastModifier`, `LastModificationTime`, `CreatorId`, `Creator`, `CreationTime`, `Id` FROM `M_Mercadolibre_Listing_ProductId` WHERE (((((( `Id` &gt; 557424057 ) AND ( `CustomerId` = 1 )) AND ( `OrderSourceId` = 11744 )) AND ( `SiteStatus` = 'success' )) AND ( `GlobalStatus` = 0 )) AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) )) ORDER BY Id ASC LIMIT 0, 50
    010+
    011+ SELECT `CustomerId`, `OrderSourceId`, `ProductId`, `Status`, `FailureLog`, `Date`, `Time`, `SiteStatus`, `SiteProductId`, `GlobalStatus`, `LastGlobalFinishedTime`, `MarketStatus`, `LastMarketFinishedTime`, `Remark`, `LastModifierId`, `LastModifier`, `LastModificationTime`, `CreatorId`, `Creator`, `CreationTime`, `Id`FROM `M_Mercadolibre_Listing_ProductId`WHERE (((((( `Id` &gt; 557424057 and `Id` &lt; 557424057 + 50 ) AND ( `CustomerId` = 1 )) AND ( `OrderSourceId` = 11744 )) AND ( `SiteStatus` = 'success' )) AND ( `GlobalStatus` = 0 )) AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) ))
    012+
    013+
    014+ 3.sql
    015+ SELECT `CustomerId`, `OrderSourceId`, `ProductId`, `Status`, `FailureLog`, `Date`, `Time`, `SiteStatus`, `SiteProductId`, `GlobalStatus`, `LastGlobalFinishedTime`, `MarketStatus`, `LastMarketFinishedTime`, `Remark`, `LastModifierId`, `LastModifier`, `LastModificationTime`, `CreatorId`, `Creator`, `CreationTime`, `Id` FROM `M_Mercadolibre_Listing_ProductId` WHERE (((((( `Id` &gt; 557424057 ) AND ( `CustomerId` = 1 )) AND ( `OrderSourceId` = 11744 )) AND ( `SiteStatus` = 'success' )) AND ( `GlobalStatus` = 0 )) AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) )) ORDER BY Id ASC LIMIT 0, 50
    005- <p>分析发现sql有使用索引,导致慢的原因是并发高和表本身数据比较多,将同步成功status=1的数据,删除或归档处理</p>
    005+ <p>2.1分析发现sql有使用索引,导致慢的原因是并发高和表本身数据比较多,将同步成功status=1的数据,删除或归档处理</p>
    007- <p><span> </span>select count(0) from M_Mercadolibre_Listing_ProductId -- 82292489<br /><span> </span>select count(0) from M_Mercadolibre_Listing_ProductId where CustomerID=1 and `Status`=1 -- 20928532 确认业务:<span>删除或归档处理</span><br /></p>
    007+ <p><span> </span>select count(0) from M_Mercadolibre_Listing_ProductId -- 82292489<br /><span> </span>select count(0) from M_Mercadolibre_Listing_ProductId where CustomerID=1 and `Status`=1 -- 20928532 确认业务:<span>删除或归档处理</span></p>
    009- <p>3.sql</p>
    009+ <p>2.2 看看业务是否可以去掉排序和Limit 0.50 改为ID查询,速度会非常快,从100秒+到 3秒 如:</p>
    010- <p>SELECT<br /> `CustomerId`,<br /> `OrderSourceId`,<br /> `ProductId`,<br /> `Status`,<br /> `FailureLog`,<br /> `Date`,<br /> `Time`,<br /> `SiteStatus`,<br /> `SiteProductId`,<br /> `GlobalStatus`,<br /> `LastGlobalFinishedTime`,<br /> `MarketStatus`,<br /> `LastMarketFinishedTime`,<br /> `Remark`,<br /> `LastModifierId`,<br /> `LastModifier`,<br /> `LastModificationTime`,<br /> `CreatorId`,<br /> `Creator`,<br /> `CreationTime`,<br /> `Id` <br />FROM<br /> `M_Mercadolibre_Listing_ProductId` <br />WHERE<br /> ((((((<br /> `Id` &gt; 557424057 <br /> ) <br /> AND ( `CustomerId` = 1 )) <br /> AND ( `OrderSourceId` = 11744 )) <br /> AND ( `SiteStatus` = 'success' )) <br /> AND ( `GlobalStatus` = 0 )) <br /> AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) )) <br />ORDER BY<br /> Id ASC <br /> LIMIT 0,<br /> 50</p>
    010+ <p><br /></p>
    011+ <p><br />SELECT<br /><span> </span>`CustomerId`,<br /><span> </span>`OrderSourceId`,<br /><span> </span>`ProductId`,<br /><span> </span>`Status`,<br /><span> </span>`FailureLog`,<br /><span> </span>`Date`,<br /><span> </span>`Time`,<br /><span> </span>`SiteStatus`,<br /><span> </span>`SiteProductId`,<br /><span> </span>`GlobalStatus`,<br /><span> </span>`LastGlobalFinishedTime`,<br /><span> </span>`MarketStatus`,<br /><span> </span>`LastMarketFinishedTime`,<br /><span> </span>`Remark`,<br /><span> </span>`LastModifierId`,<br /><span> </span>`LastModifier`,<br /><span> </span>`LastModificationTime`,<br /><span> </span>`CreatorId`,<br /><span> </span>`Creator`,<br /><span> </span>`CreationTime`,<br /><span> </span>`Id`<br />FROM<br /><span> </span>`M_Mercadolibre_Listing_ProductId`<br />WHERE<br /><span> </span>((((((<br /><span> </span>`Id` &gt; 557424057 and `Id` &lt; <span>557424057 + 50</span><br /><span> </span>)<br /><span> </span>AND ( `CustomerId` = 1 ))<br /><span> </span>AND ( `OrderSourceId` = 11744 ))<br /><span> </span>AND ( `SiteStatus` = 'success' ))<br /><span> </span>AND ( `GlobalStatus` = 0 ))<br /><span> </span>AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) ))<br /></p>
    012+ <p></p>
    013+ <p><br /></p>
    014+ <p>3.sql</p>
    015+ <p>SELECT<br /> `CustomerId`,<br /> `OrderSourceId`,<br /> `ProductId`,<br /> `Status`,<br /> `FailureLog`,<br /> `Date`,<br /> `Time`,<br /> `SiteStatus`,<br /> `SiteProductId`,<br /> `GlobalStatus`,<br /> `LastGlobalFinishedTime`,<br /> `MarketStatus`,<br /> `LastMarketFinishedTime`,<br /> `Remark`,<br /> `LastModifierId`,<br /> `LastModifier`,<br /> `LastModificationTime`,<br /> `CreatorId`,<br /> `Creator`,<br /> `CreationTime`,<br /> `Id` <br />FROM<br /> `M_Mercadolibre_Listing_ProductId` <br />WHERE<br /> ((((((<br /> `Id` &gt; 557424057 <br /> ) <br /> AND ( `CustomerId` = 1 )) <br /> AND ( `OrderSourceId` = 11744 )) <br /> AND ( `SiteStatus` = 'success' )) <br /> AND ( `GlobalStatus` = 0 )) <br /> AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) )) <br />ORDER BY<br /> Id ASC <br /> LIMIT 0,<br /> 50</p>
  3. 2023-04-07 14:35:45, 由 陈少军 编辑。
    修改了 任务描述,区别为:
    009- 2.2 看看业务是否可以去掉排序和Limit 0.50 改为ID查询,速度会非常快,从100秒+到 3秒 如:
    009+ 2.2 看看业务是否可以去掉排序和Limit 0.50 改为ID查询(`Id` &gt; 557424057 and `Id` &lt;557424057 + 50),速度会非常快,从100秒+到 3秒 如:
    011- SELECT `CustomerId`, `OrderSourceId`, `ProductId`, `Status`, `FailureLog`, `Date`, `Time`, `SiteStatus`, `SiteProductId`, `GlobalStatus`, `LastGlobalFinishedTime`, `MarketStatus`, `LastMarketFinishedTime`, `Remark`, `LastModifierId`, `LastModifier`, `LastModificationTime`, `CreatorId`, `Creator`, `CreationTime`, `Id`FROM `M_Mercadolibre_Listing_ProductId`WHERE (((((( `Id` &gt; 557424057 and `Id` &lt; 557424057 + 50 ) AND ( `CustomerId` = 1 )) AND ( `OrderSourceId` = 11744 )) AND ( `SiteStatus` = 'success' )) AND ( `GlobalStatus` = 0 )) AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) ))
    011+ SELECT `CustomerId`, `OrderSourceId`, `ProductId`, `Status`, `FailureLog`, `Date`, `Time`, `SiteStatus`, `SiteProductId`, `GlobalStatus`, `LastGlobalFinishedTime`, `MarketStatus`, `LastMarketFinishedTime`, `Remark`, `LastModifierId`, `LastModifier`, `LastModificationTime`, `CreatorId`, `Creator`, `CreationTime`, `Id`FROM `M_Mercadolibre_Listing_ProductId`WHERE (((((( `Id` &gt; 557424057 and `Id` &lt; 557424057 + 50 ) AND ( `CustomerId` = 1 )) AND ( `OrderSourceId` = 11744 )) AND ( `SiteStatus` = 'success' )) AND ( `GlobalStatus` = 0 )) AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) ))
    012-
    012+
    013-
    013+ 3.sql
    014- 3.sql
    014+ SELECT `CustomerId`, `OrderSourceId`, `ProductId`, `Status`, `FailureLog`, `Date`, `Time`, `SiteStatus`, `SiteProductId`, `GlobalStatus`, `LastGlobalFinishedTime`, `MarketStatus`, `LastMarketFinishedTime`, `Remark`, `LastModifierId`, `LastModifier`, `LastModificationTime`, `CreatorId`, `Creator`, `CreationTime`, `Id` FROM `M_Mercadolibre_Listing_ProductId` WHERE (((((( `Id` &gt; 557424057 ) AND ( `CustomerId` = 1 )) AND ( `OrderSourceId` = 11744 )) AND ( `SiteStatus` = 'success' )) AND ( `GlobalStatus` = 0 )) AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) )) ORDER BY Id ASC LIMIT 0, 50
    015- SELECT `CustomerId`, `OrderSourceId`, `ProductId`, `Status`, `FailureLog`, `Date`, `Time`, `SiteStatus`, `SiteProductId`, `GlobalStatus`, `LastGlobalFinishedTime`, `MarketStatus`, `LastMarketFinishedTime`, `Remark`, `LastModifierId`, `LastModifier`, `LastModificationTime`, `CreatorId`, `Creator`, `CreationTime`, `Id` FROM `M_Mercadolibre_Listing_ProductId` WHERE (((((( `Id` &gt; 557424057 ) AND ( `CustomerId` = 1 )) AND ( `OrderSourceId` = 11744 )) AND ( `SiteStatus` = 'success' )) AND ( `GlobalStatus` = 0 )) AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) )) ORDER BY Id ASC LIMIT 0, 50
    009- <p>2.2 看看业务是否可以去掉排序和Limit 0.50 改为ID查询,速度会非常快,从100秒+到 3秒 如:</p>
    009+ <p>2.2 看看业务是否可以去掉排序和Limit 0.50 改为ID查询(<span>`Id` &gt; 557424057 and `Id` &lt;</span><span>557424057 + 50</span>),速度会非常快,从100秒+到 3秒 如:</p>
    011- <p><br />SELECT<br /><span> </span>`CustomerId`,<br /><span> </span>`OrderSourceId`,<br /><span> </span>`ProductId`,<br /><span> </span>`Status`,<br /><span> </span>`FailureLog`,<br /><span> </span>`Date`,<br /><span> </span>`Time`,<br /><span> </span>`SiteStatus`,<br /><span> </span>`SiteProductId`,<br /><span> </span>`GlobalStatus`,<br /><span> </span>`LastGlobalFinishedTime`,<br /><span> </span>`MarketStatus`,<br /><span> </span>`LastMarketFinishedTime`,<br /><span> </span>`Remark`,<br /><span> </span>`LastModifierId`,<br /><span> </span>`LastModifier`,<br /><span> </span>`LastModificationTime`,<br /><span> </span>`CreatorId`,<br /><span> </span>`Creator`,<br /><span> </span>`CreationTime`,<br /><span> </span>`Id`<br />FROM<br /><span> </span>`M_Mercadolibre_Listing_ProductId`<br />WHERE<br /><span> </span>((((((<br /><span> </span>`Id` &gt; 557424057 and `Id` &lt; <span>557424057 + 50</span><br /><span> </span>)<br /><span> </span>AND ( `CustomerId` = 1 ))<br /><span> </span>AND ( `OrderSourceId` = 11744 ))<br /><span> </span>AND ( `SiteStatus` = 'success' ))<br /><span> </span>AND ( `GlobalStatus` = 0 ))<br /><span> </span>AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) ))<br /></p>
    011+ <p><br />SELECT<br /><span> </span>`CustomerId`,<br /><span> </span>`OrderSourceId`,<br /><span> </span>`ProductId`,<br /><span> </span>`Status`,<br /><span> </span>`FailureLog`,<br /><span> </span>`Date`,<br /><span> </span>`Time`,<br /><span> </span>`SiteStatus`,<br /><span> </span>`SiteProductId`,<br /><span> </span>`GlobalStatus`,<br /><span> </span>`LastGlobalFinishedTime`,<br /><span> </span>`MarketStatus`,<br /><span> </span>`LastMarketFinishedTime`,<br /><span> </span>`Remark`,<br /><span> </span>`LastModifierId`,<br /><span> </span>`LastModifier`,<br /><span> </span>`LastModificationTime`,<br /><span> </span>`CreatorId`,<br /><span> </span>`Creator`,<br /><span> </span>`CreationTime`,<br /><span> </span>`Id`<br />FROM<br /><span> </span>`M_Mercadolibre_Listing_ProductId`<br />WHERE<br /><span> </span>((((((<br /><span> </span>`Id` &gt; 557424057 and `Id` &lt; <span>557424057 + 50</span><br /><span> </span>)<br /><span> </span>AND ( `CustomerId` = 1 ))<br /><span> </span>AND ( `OrderSourceId` = 11744 ))<br /><span> </span>AND ( `SiteStatus` = 'success' ))<br /><span> </span>AND ( `GlobalStatus` = 0 ))<br /><span> </span>AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) ))</p>
    012- <p></p>
    012+ <p><br /></p>
    013- <p><br /></p>
    013+ <p>3.sql</p>
    014- <p>3.sql</p>
    014+ <p>SELECT<br /> `CustomerId`,<br /> `OrderSourceId`,<br /> `ProductId`,<br /> `Status`,<br /> `FailureLog`,<br /> `Date`,<br /> `Time`,<br /> `SiteStatus`,<br /> `SiteProductId`,<br /> `GlobalStatus`,<br /> `LastGlobalFinishedTime`,<br /> `MarketStatus`,<br /> `LastMarketFinishedTime`,<br /> `Remark`,<br /> `LastModifierId`,<br /> `LastModifier`,<br /> `LastModificationTime`,<br /> `CreatorId`,<br /> `Creator`,<br /> `CreationTime`,<br /> `Id` <br />FROM<br /> `M_Mercadolibre_Listing_ProductId` <br />WHERE<br /> ((((((<br /> `Id` &gt; 557424057 <br /> ) <br /> AND ( `CustomerId` = 1 )) <br /> AND ( `OrderSourceId` = 11744 )) <br /> AND ( `SiteStatus` = 'success' )) <br /> AND ( `GlobalStatus` = 0 )) <br /> AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) )) <br />ORDER BY<br /> Id ASC <br /> LIMIT 0,<br /> 50</p>
    015- <p>SELECT<br /> `CustomerId`,<br /> `OrderSourceId`,<br /> `ProductId`,<br /> `Status`,<br /> `FailureLog`,<br /> `Date`,<br /> `Time`,<br /> `SiteStatus`,<br /> `SiteProductId`,<br /> `GlobalStatus`,<br /> `LastGlobalFinishedTime`,<br /> `MarketStatus`,<br /> `LastMarketFinishedTime`,<br /> `Remark`,<br /> `LastModifierId`,<br /> `LastModifier`,<br /> `LastModificationTime`,<br /> `CreatorId`,<br /> `Creator`,<br /> `CreationTime`,<br /> `Id` <br />FROM<br /> `M_Mercadolibre_Listing_ProductId` <br />WHERE<br /> ((((((<br /> `Id` &gt; 557424057 <br /> ) <br /> AND ( `CustomerId` = 1 )) <br /> AND ( `OrderSourceId` = 11744 )) <br /> AND ( `SiteStatus` = 'success' )) <br /> AND ( `GlobalStatus` = 0 )) <br /> AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) )) <br />ORDER BY<br /> Id ASC <br /> LIMIT 0,<br /> 50</p>
  4. 2023-04-07 14:37:59, 由 陈少军 编辑。
    修改了 任务描述,区别为:
    009- 2.2 看看业务是否可以去掉排序和Limit 0.50 改为ID查询(`Id` &gt; 557424057 and `Id` &lt;557424057 + 50),速度会非常快,从100秒+到 3秒 如:
    009+ 2.2 看这个业务应该是每次批量获取50条数据,进行同步:
    010-
    010+ 优化:可以去掉排序和Limit 0,50 改为ID查询(`Id` &gt; 557424057 and `Id` &lt;557424057 + 50)也是批量获取数据,优化后速度会非常快,从100秒+到 3秒 如:
    011- SELECT `CustomerId`, `OrderSourceId`, `ProductId`, `Status`, `FailureLog`, `Date`, `Time`, `SiteStatus`, `SiteProductId`, `GlobalStatus`, `LastGlobalFinishedTime`, `MarketStatus`, `LastMarketFinishedTime`, `Remark`, `LastModifierId`, `LastModifier`, `LastModificationTime`, `CreatorId`, `Creator`, `CreationTime`, `Id`FROM `M_Mercadolibre_Listing_ProductId`WHERE (((((( `Id` &gt; 557424057 and `Id` &lt; 557424057 + 50 ) AND ( `CustomerId` = 1 )) AND ( `OrderSourceId` = 11744 )) AND ( `SiteStatus` = 'success' )) AND ( `GlobalStatus` = 0 )) AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) ))
    011+
    012-
    012+ SELECT `CustomerId`, `OrderSourceId`, `ProductId`, `Status`, `FailureLog`, `Date`, `Time`, `SiteStatus`, `SiteProductId`, `GlobalStatus`, `LastGlobalFinishedTime`, `MarketStatus`, `LastMarketFinishedTime`, `Remark`, `LastModifierId`, `LastModifier`, `LastModificationTime`, `CreatorId`, `Creator`, `CreationTime`, `Id`FROM `M_Mercadolibre_Listing_ProductId`WHERE (((((( `Id` &gt; 557424057 and `Id` &lt; 557424057 + 50 ) AND ( `CustomerId` = 1 )) AND ( `OrderSourceId` = 11744 )) AND ( `SiteStatus` = 'success' )) AND ( `GlobalStatus` = 0 )) AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) ))
    013- 3.sql
    013+
    014- SELECT `CustomerId`, `OrderSourceId`, `ProductId`, `Status`, `FailureLog`, `Date`, `Time`, `SiteStatus`, `SiteProductId`, `GlobalStatus`, `LastGlobalFinishedTime`, `MarketStatus`, `LastMarketFinishedTime`, `Remark`, `LastModifierId`, `LastModifier`, `LastModificationTime`, `CreatorId`, `Creator`, `CreationTime`, `Id` FROM `M_Mercadolibre_Listing_ProductId` WHERE (((((( `Id` &gt; 557424057 ) AND ( `CustomerId` = 1 )) AND ( `OrderSourceId` = 11744 )) AND ( `SiteStatus` = 'success' )) AND ( `GlobalStatus` = 0 )) AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) )) ORDER BY Id ASC LIMIT 0, 50
    014+ 3.sql
    015+ SELECT `CustomerId`, `OrderSourceId`, `ProductId`, `Status`, `FailureLog`, `Date`, `Time`, `SiteStatus`, `SiteProductId`, `GlobalStatus`, `LastGlobalFinishedTime`, `MarketStatus`, `LastMarketFinishedTime`, `Remark`, `LastModifierId`, `LastModifier`, `LastModificationTime`, `CreatorId`, `Creator`, `CreationTime`, `Id` FROM `M_Mercadolibre_Listing_ProductId` WHERE (((((( `Id` &gt; 557424057 ) AND ( `CustomerId` = 1 )) AND ( `OrderSourceId` = 11744 )) AND ( `SiteStatus` = 'success' )) AND ( `GlobalStatus` = 0 )) AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) )) ORDER BY Id ASC LIMIT 0, 50
    009- <p>2.2 看看业务是否可以去掉排序和Limit 0.50 改为ID查询(<span>`Id` &gt; 557424057 and `Id` &lt;</span><span>557424057 + 50</span>),速度会非常快,从100秒+到 3秒 如:</p>
    009+ <p>2.2 看这个业务应该是每次批量获取50条数据,进行同步:</p>
    010- <p><br /></p>
    010+ <p> 优化:可以去掉排序和Limit 0,50 改为ID查询(<span>`Id` &gt; 557424057 and `Id` &lt;</span><span>557424057 + 50</span>)也是批量获取数据,优化后速度会非常快,从100秒+到 3秒 如:</p>
    011- <p><br />SELECT<br /><span> </span>`CustomerId`,<br /><span> </span>`OrderSourceId`,<br /><span> </span>`ProductId`,<br /><span> </span>`Status`,<br /><span> </span>`FailureLog`,<br /><span> </span>`Date`,<br /><span> </span>`Time`,<br /><span> </span>`SiteStatus`,<br /><span> </span>`SiteProductId`,<br /><span> </span>`GlobalStatus`,<br /><span> </span>`LastGlobalFinishedTime`,<br /><span> </span>`MarketStatus`,<br /><span> </span>`LastMarketFinishedTime`,<br /><span> </span>`Remark`,<br /><span> </span>`LastModifierId`,<br /><span> </span>`LastModifier`,<br /><span> </span>`LastModificationTime`,<br /><span> </span>`CreatorId`,<br /><span> </span>`Creator`,<br /><span> </span>`CreationTime`,<br /><span> </span>`Id`<br />FROM<br /><span> </span>`M_Mercadolibre_Listing_ProductId`<br />WHERE<br /><span> </span>((((((<br /><span> </span>`Id` &gt; 557424057 and `Id` &lt; <span>557424057 + 50</span><br /><span> </span>)<br /><span> </span>AND ( `CustomerId` = 1 ))<br /><span> </span>AND ( `OrderSourceId` = 11744 ))<br /><span> </span>AND ( `SiteStatus` = 'success' ))<br /><span> </span>AND ( `GlobalStatus` = 0 ))<br /><span> </span>AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) ))</p>
    011+ <p><br /></p>
    012- <p><br /></p>
    012+ <p><br />SELECT<br /><span> </span>`CustomerId`,<br /><span> </span>`OrderSourceId`,<br /><span> </span>`ProductId`,<br /><span> </span>`Status`,<br /><span> </span>`FailureLog`,<br /><span> </span>`Date`,<br /><span> </span>`Time`,<br /><span> </span>`SiteStatus`,<br /><span> </span>`SiteProductId`,<br /><span> </span>`GlobalStatus`,<br /><span> </span>`LastGlobalFinishedTime`,<br /><span> </span>`MarketStatus`,<br /><span> </span>`LastMarketFinishedTime`,<br /><span> </span>`Remark`,<br /><span> </span>`LastModifierId`,<br /><span> </span>`LastModifier`,<br /><span> </span>`LastModificationTime`,<br /><span> </span>`CreatorId`,<br /><span> </span>`Creator`,<br /><span> </span>`CreationTime`,<br /><span> </span>`Id`<br />FROM<br /><span> </span>`M_Mercadolibre_Listing_ProductId`<br />WHERE<br /><span> </span>((((((<br /><span> </span>`Id` &gt; 557424057 and `Id` &lt; <span>557424057 + 50</span><br /><span> </span>)<br /><span> </span>AND ( `CustomerId` = 1 ))<br /><span> </span>AND ( `OrderSourceId` = 11744 ))<br /><span> </span>AND ( `SiteStatus` = 'success' ))<br /><span> </span>AND ( `GlobalStatus` = 0 ))<br /><span> </span>AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) ))</p>
    013- <p>3.sql</p>
    013+ <p><br /></p>
    014- <p>SELECT<br /> `CustomerId`,<br /> `OrderSourceId`,<br /> `ProductId`,<br /> `Status`,<br /> `FailureLog`,<br /> `Date`,<br /> `Time`,<br /> `SiteStatus`,<br /> `SiteProductId`,<br /> `GlobalStatus`,<br /> `LastGlobalFinishedTime`,<br /> `MarketStatus`,<br /> `LastMarketFinishedTime`,<br /> `Remark`,<br /> `LastModifierId`,<br /> `LastModifier`,<br /> `LastModificationTime`,<br /> `CreatorId`,<br /> `Creator`,<br /> `CreationTime`,<br /> `Id` <br />FROM<br /> `M_Mercadolibre_Listing_ProductId` <br />WHERE<br /> ((((((<br /> `Id` &gt; 557424057 <br /> ) <br /> AND ( `CustomerId` = 1 )) <br /> AND ( `OrderSourceId` = 11744 )) <br /> AND ( `SiteStatus` = 'success' )) <br /> AND ( `GlobalStatus` = 0 )) <br /> AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) )) <br />ORDER BY<br /> Id ASC <br /> LIMIT 0,<br /> 50</p>
    014+ <p>3.sql</p>
    015+ <p>SELECT<br /> `CustomerId`,<br /> `OrderSourceId`,<br /> `ProductId`,<br /> `Status`,<br /> `FailureLog`,<br /> `Date`,<br /> `Time`,<br /> `SiteStatus`,<br /> `SiteProductId`,<br /> `GlobalStatus`,<br /> `LastGlobalFinishedTime`,<br /> `MarketStatus`,<br /> `LastMarketFinishedTime`,<br /> `Remark`,<br /> `LastModifierId`,<br /> `LastModifier`,<br /> `LastModificationTime`,<br /> `CreatorId`,<br /> `Creator`,<br /> `CreationTime`,<br /> `Id` <br />FROM<br /> `M_Mercadolibre_Listing_ProductId` <br />WHERE<br /> ((((((<br /> `Id` &gt; 557424057 <br /> ) <br /> AND ( `CustomerId` = 1 )) <br /> AND ( `OrderSourceId` = 11744 )) <br /> AND ( `SiteStatus` = 'success' )) <br /> AND ( `GlobalStatus` = 0 )) <br /> AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) )) <br />ORDER BY<br /> Id ASC <br /> LIMIT 0,<br /> 50</p>
  5. 2023-04-07 14:56:40, 由 陈少军 编辑。
    修改了 任务描述,区别为:
    005- 2.1分析发现sql有使用索引,导致慢的原因是并发高和表本身数据比较多,将同步成功status=1的数据,删除或归档处理
    005+ 2.1分析发现sql有使用索引,导致慢的原因是并发高和表本身数据比较多,索引不是最优的,从业务上考虑数据逻辑闭环,将同步成功status=1的数据,删除或归档处理
    005- <p>2.1分析发现sql有使用索引,导致慢的原因是并发高和表本身数据比较多,将同步成功status=1的数据,删除或归档处理</p>
    005+ <p>2.1分析发现sql有使用索引,导致慢的原因是并发高和表本身数据比较多,索引<span>不是最优的,从业务上考虑数据逻辑闭环,</span>将同步成功status=1的数据,删除或归档处理</p>
  6. 2023-04-07 15:09:38, 由 陈少军 编辑。
    修改了 任务描述,区别为:
    013-
    013+ ORDER BYId ASC
    014- 3.sql
    014+
    015- SELECT `CustomerId`, `OrderSourceId`, `ProductId`, `Status`, `FailureLog`, `Date`, `Time`, `SiteStatus`, `SiteProductId`, `GlobalStatus`, `LastGlobalFinishedTime`, `MarketStatus`, `LastMarketFinishedTime`, `Remark`, `LastModifierId`, `LastModifier`, `LastModificationTime`, `CreatorId`, `Creator`, `CreationTime`, `Id` FROM `M_Mercadolibre_Listing_ProductId` WHERE (((((( `Id` &gt; 557424057 ) AND ( `CustomerId` = 1 )) AND ( `OrderSourceId` = 11744 )) AND ( `SiteStatus` = 'success' )) AND ( `GlobalStatus` = 0 )) AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) )) ORDER BY Id ASC LIMIT 0, 50
    015+ 3.sql
    016+ SELECT `CustomerId`, `OrderSourceId`, `ProductId`, `Status`, `FailureLog`, `Date`, `Time`, `SiteStatus`, `SiteProductId`, `GlobalStatus`, `LastGlobalFinishedTime`, `MarketStatus`, `LastMarketFinishedTime`, `Remark`, `LastModifierId`, `LastModifier`, `LastModificationTime`, `CreatorId`, `Creator`, `CreationTime`, `Id` FROM `M_Mercadolibre_Listing_ProductId` WHERE (((((( `Id` &gt; 557424057 ) AND ( `CustomerId` = 1 )) AND ( `OrderSourceId` = 11744 )) AND ( `SiteStatus` = 'success' )) AND ( `GlobalStatus` = 0 )) AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) )) ORDER BY Id ASC LIMIT 0, 50
    013- <p><br /></p>
    013+ <p><span style="font-size:14px;background-color:#FFFFFF;">ORDER BY</span><br /><span style="font-size:14px;background-color:#FFFFFF;">Id ASC</span></p>
    014- <p>3.sql</p>
    014+ <p></p>
    015- <p>SELECT<br /> `CustomerId`,<br /> `OrderSourceId`,<br /> `ProductId`,<br /> `Status`,<br /> `FailureLog`,<br /> `Date`,<br /> `Time`,<br /> `SiteStatus`,<br /> `SiteProductId`,<br /> `GlobalStatus`,<br /> `LastGlobalFinishedTime`,<br /> `MarketStatus`,<br /> `LastMarketFinishedTime`,<br /> `Remark`,<br /> `LastModifierId`,<br /> `LastModifier`,<br /> `LastModificationTime`,<br /> `CreatorId`,<br /> `Creator`,<br /> `CreationTime`,<br /> `Id` <br />FROM<br /> `M_Mercadolibre_Listing_ProductId` <br />WHERE<br /> ((((((<br /> `Id` &gt; 557424057 <br /> ) <br /> AND ( `CustomerId` = 1 )) <br /> AND ( `OrderSourceId` = 11744 )) <br /> AND ( `SiteStatus` = 'success' )) <br /> AND ( `GlobalStatus` = 0 )) <br /> AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) )) <br />ORDER BY<br /> Id ASC <br /> LIMIT 0,<br /> 50</p>
    015+ <p>3.sql</p>
    016+ <p>SELECT<br /> `CustomerId`,<br /> `OrderSourceId`,<br /> `ProductId`,<br /> `Status`,<br /> `FailureLog`,<br /> `Date`,<br /> `Time`,<br /> `SiteStatus`,<br /> `SiteProductId`,<br /> `GlobalStatus`,<br /> `LastGlobalFinishedTime`,<br /> `MarketStatus`,<br /> `LastMarketFinishedTime`,<br /> `Remark`,<br /> `LastModifierId`,<br /> `LastModifier`,<br /> `LastModificationTime`,<br /> `CreatorId`,<br /> `Creator`,<br /> `CreationTime`,<br /> `Id` <br />FROM<br /> `M_Mercadolibre_Listing_ProductId` <br />WHERE<br /> ((((((<br /> `Id` &gt; 557424057 <br /> ) <br /> AND ( `CustomerId` = 1 )) <br /> AND ( `OrderSourceId` = 11744 )) <br /> AND ( `SiteStatus` = 'success' )) <br /> AND ( `GlobalStatus` = 0 )) <br /> AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) )) <br />ORDER BY<br /> Id ASC <br /> LIMIT 0,<br /> 50</p>
  7. 2023-04-07 15:15:05, 由 陈少军 编辑。
    修改了 任务描述,区别为:
    010- 优化:可以去掉排序和Limit 0,50 改为ID查询(`Id` &gt; 557424057 and `Id` &lt;557424057 + 50)也是批量获取数据,优化后速度会非常快,从100秒+到 3秒 如:
    010+ 优化:可以去掉排序和Limit 0,50 改为ID查询(`Id` &gt; 557424057 and `Id` &lt;557424057 + 50)也是批量获取数据,默认是ID升序的,优化后速度会非常快,从100秒+到 3秒 如:
    013- ORDER BYId ASC
    013+
    014-
    014+ 3.sql
    015- 3.sql
    015+ SELECT `CustomerId`, `OrderSourceId`, `ProductId`, `Status`, `FailureLog`, `Date`, `Time`, `SiteStatus`, `SiteProductId`, `GlobalStatus`, `LastGlobalFinishedTime`, `MarketStatus`, `LastMarketFinishedTime`, `Remark`, `LastModifierId`, `LastModifier`, `LastModificationTime`, `CreatorId`, `Creator`, `CreationTime`, `Id` FROM `M_Mercadolibre_Listing_ProductId` WHERE (((((( `Id` &gt; 557424057 ) AND ( `CustomerId` = 1 )) AND ( `OrderSourceId` = 11744 )) AND ( `SiteStatus` = 'success' )) AND ( `GlobalStatus` = 0 )) AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) )) ORDER BY Id ASC LIMIT 0, 50
    016- SELECT `CustomerId`, `OrderSourceId`, `ProductId`, `Status`, `FailureLog`, `Date`, `Time`, `SiteStatus`, `SiteProductId`, `GlobalStatus`, `LastGlobalFinishedTime`, `MarketStatus`, `LastMarketFinishedTime`, `Remark`, `LastModifierId`, `LastModifier`, `LastModificationTime`, `CreatorId`, `Creator`, `CreationTime`, `Id` FROM `M_Mercadolibre_Listing_ProductId` WHERE (((((( `Id` &gt; 557424057 ) AND ( `CustomerId` = 1 )) AND ( `OrderSourceId` = 11744 )) AND ( `SiteStatus` = 'success' )) AND ( `GlobalStatus` = 0 )) AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) )) ORDER BY Id ASC LIMIT 0, 50
    010- <p> 优化:可以去掉排序和Limit 0,50 改为ID查询(<span>`Id` &gt; 557424057 and `Id` &lt;</span><span>557424057 + 50</span>)也是批量获取数据,优化后速度会非常快,从100秒+到 3秒 如:</p>
    010+ <p> 优化:可以去掉排序和Limit 0,50 改为ID查询(<span>`Id` &gt; 557424057 and `Id` &lt;</span><span>557424057 + 50</span>)也是批量获取数据,默认是ID升序的,优化后速度会非常快,从100秒+到 3秒 如:</p>
    013- <p><span style="font-size:14px;background-color:#FFFFFF;">ORDER BY</span><br /><span style="font-size:14px;background-color:#FFFFFF;">Id ASC</span></p>
    013+ <p><br /></p>
    014- <p></p>
    014+ <p>3.sql</p>
    015- <p>3.sql</p>
    015+ <p>SELECT<br /> `CustomerId`,<br /> `OrderSourceId`,<br /> `ProductId`,<br /> `Status`,<br /> `FailureLog`,<br /> `Date`,<br /> `Time`,<br /> `SiteStatus`,<br /> `SiteProductId`,<br /> `GlobalStatus`,<br /> `LastGlobalFinishedTime`,<br /> `MarketStatus`,<br /> `LastMarketFinishedTime`,<br /> `Remark`,<br /> `LastModifierId`,<br /> `LastModifier`,<br /> `LastModificationTime`,<br /> `CreatorId`,<br /> `Creator`,<br /> `CreationTime`,<br /> `Id` <br />FROM<br /> `M_Mercadolibre_Listing_ProductId` <br />WHERE<br /> ((((((<br /> `Id` &gt; 557424057 <br /> ) <br /> AND ( `CustomerId` = 1 )) <br /> AND ( `OrderSourceId` = 11744 )) <br /> AND ( `SiteStatus` = 'success' )) <br /> AND ( `GlobalStatus` = 0 )) <br /> AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) )) <br />ORDER BY<br /> Id ASC <br /> LIMIT 0,<br /> 50</p>
    016- <p>SELECT<br /> `CustomerId`,<br /> `OrderSourceId`,<br /> `ProductId`,<br /> `Status`,<br /> `FailureLog`,<br /> `Date`,<br /> `Time`,<br /> `SiteStatus`,<br /> `SiteProductId`,<br /> `GlobalStatus`,<br /> `LastGlobalFinishedTime`,<br /> `MarketStatus`,<br /> `LastMarketFinishedTime`,<br /> `Remark`,<br /> `LastModifierId`,<br /> `LastModifier`,<br /> `LastModificationTime`,<br /> `CreatorId`,<br /> `Creator`,<br /> `CreationTime`,<br /> `Id` <br />FROM<br /> `M_Mercadolibre_Listing_ProductId` <br />WHERE<br /> ((((((<br /> `Id` &gt; 557424057 <br /> ) <br /> AND ( `CustomerId` = 1 )) <br /> AND ( `OrderSourceId` = 11744 )) <br /> AND ( `SiteStatus` = 'success' )) <br /> AND ( `GlobalStatus` = 0 )) <br /> AND ( `Time` &lt; TIMESTAMP ( '2023-04-07 08:39:32.546924' ) )) <br />ORDER BY<br /> Id ASC <br /> LIMIT 0,<br /> 50</p>
  8. 2023-04-07 15:54:33, 由 陈少军 编辑。
    修改了 任务描述,区别为:
    010- 优化:可以去掉排序和Limit 0,50 改为ID查询(`Id` &gt; 557424057 and `Id` &lt;557424057 + 50)也是批量获取数据,默认是ID升序的,优化后速度会非常快,从100秒+到 3秒 如:
    010+ 优化:可以去掉排序和Limit 0,50 改为ID查询(`Id` &gt; 557424057 and `Id` &lt;557424057 + 50)也是批量获取数据,默认是ID升序的,优化后速度会非常快,从100秒+到 3秒,因ID连续但这个查询条件查询出的ID是递增但是不连续,有可能=50查不出数据,建议+ 100000(`Id` &gt; 557424057 and `Id` &lt;557424057 + 100000),这个查询条件一共只有14632条数据,也可以一次查询出来,具体业务根据实际情况定,这个方案让sql走了最优索引查询的快,做一个指引如:
    010- <p> 优化:可以去掉排序和Limit 0,50 改为ID查询(<span>`Id` &gt; 557424057 and `Id` &lt;</span><span>557424057 + 50</span>)也是批量获取数据,默认是ID升序的,优化后速度会非常快,从100秒+到 3秒 如:</p>
    010+ <p> 优化:可以去掉排序和Limit 0,50 改为ID查询(<span>`Id` &gt; 557424057 and `Id` &lt;</span><span>557424057 + 50</span>)也是批量获取数据,默认是ID升序的,优化后速度会非常快,从100秒+到 3秒,因ID连续但这个查询条件查询出的ID是递增但是不连续,有可能=50查不出数据,建议+ 100000<span>(</span><span>`Id` &gt; 557424057 and `Id` &lt;</span><span>557424057 + 100000</span><span>),这个查询条件一共只有14632条数据,也可以一次查询出来,具体业务根据实际情况定,这个方案让sql走了最优索引查询的快,做一个指引</span>如:</p>
  9. 2023-04-18 16:21:17, 由 陈少军 完成。
    修改了 总计消耗,旧值为 "0",新值为 "1"。
    修改了 指派给 ,旧值为 "chengwei",新值为 "chenshaojun"。
    修改了 实际完成,旧值为 "",新值为 "2023-04-18 16:20:42"。
    修改了 任务状态,旧值为 "wait",新值为 "done"。
    修改了 实际开始,旧值为 "0000-00-00 00:00:00",新值为 "2023-04-18 16:21:17"。
    修改了 由谁完成,旧值为 "",新值为 "chenshaojun"。
    开发已从sql上添加 强制索引暂时解决
  10. 2023-10-31 14:53:16, 由 周杰 关闭。
    修改了 任务状态,旧值为 "done",新值为 "closed"。
    修改了 指派给 ,旧值为 "chenshaojun",新值为 "closed"。
    修改了 由谁关闭,旧值为 "",新值为 "zhoujie"。
    修改了 关闭时间,旧值为 "0000-00-00 00:00:00",新值为 "2023-10-31 14:53:16"。
    修改了 关闭原因,旧值为 "",新值为 "done"。
所属迭代 系统管理部 - 系统服务优化
所属模块 /
相关需求 暂无
指派给 Closed 于 2023-10-31 14:53:16
任务类型 后端
任务标签 初始
开发组 有品开发组
任务状态 已关闭
进度 100%
优先级 P2
关键词
抄送给
由谁创建 陈少军 于 2023-04-07 14:16:14
由谁完成 陈少军 于 2023-04-18 16:20:42
由谁取消 暂无
由谁关闭 周杰 于 2023-10-31 14:53:16
关闭原因 已完成
最后编辑 周杰 于 2023-10-31 14:53:16
团队 最初预计 总计消耗 预计剩余
工时信息
最初预计 0工时
总计消耗 1工时
预计剩余 0工时
预计开始 2023-04-07
实际开始 2023-04-18
截止日期 2023-04-10