WO2021208228A1

WO2021208228A1 - Data processing method and apparatus, and computer device and storage medium

Info

Publication number: WO2021208228A1
Application number: PCT/CN2020/096997
Authority: WO
Inventors: 邰华运; 吕勇; 万鹏程
Original assignee: 苏宁易购集团股份有限公司; 苏宁云计算有限公司
Priority date: 2020-04-14
Filing date: 2020-06-19
Publication date: 2021-10-21
Also published as: CN111222089A; CA3177218A1; CN111222089B

Abstract

A data processing method and apparatus for commodity service data, and a computer device and a storage medium. The method comprises: obtaining first service input-parameter data of a target commodity service from full data of the target commodity service (S100); obtaining, according to the first service input-parameter data, first commodity service data of the target commodity service from service data of a database (S300), and obtaining, according to the first service input-parameter data, second commodity service data of the target commodity service from cache service data (S500); when the first commodity service data is different from the second commodity service data, updating, according to the first service input-parameter data, second service input-parameter data in a pre-warming table of the target commodity service; and synchronizing, according to the updated second service input-parameter data in the pre-warming table, the service data of the database corresponding to the target commodity service with the cache service data (S900). The method can automatically monitor the process of updating and caching the commodity service data.

Description

Data processing method, device, computer equipment and storage medium

Technical field

This application relates to the technical field of commodity service data processing, in particular to a data processing method, device, computer equipment and storage medium of commodity service data.

Background technique

The main value of the Internet commodity service management platform is to provide commodity information data services, including data distribution and reception, storage, cache preheating, data services, downstream system service push notifications and other basic data services. However, due to the distinction of services in commodity service data, the platform stores the service data in multi-dimensional database and table according to business scenarios. Therefore, it is easy to cause the platform to monitor the data processing of the commodity service data is not precise enough. In particular, there is a lack of monitoring of the data processing process of synchronizing the service data in the database to the cache service data. For this data processing process, the traditional monitoring method is to manually detect that there is a data inconsistency between the service data of the database and the cached service data in the production environment, manually input the corresponding input parameters, and synchronize the service data in the database to the corresponding input parameters according to the corresponding input parameters. Cache service data. The manual monitoring and manual operation methods result in low timeliness of data processing of commodity service data.

Summary of the invention

Based on this, it is necessary to address the above technical problems to provide a data processing method, device, computer equipment and storage medium for commodity service data that can automatically monitor the processing process of updating the cache of commodity service data and improve the timeliness of updating the cached commodity service data .

A data processing method for commodity service data, the method comprising:

Obtain the first service entry data of the target product service from the full data of the target product service; obtain the first product service data of the target product service from the service data of the database according to the first service entry data; According to the first service entry parameter The data obtains the second commodity service data of the target commodity service from the cached service data; when the first commodity service data is not the same as the second commodity service data, update the warm-up table of the target commodity service according to the first service input data The second service input parameter data, the warm-up table is used to store the corresponding service input parameter data when the service data of the database is synchronized to the cache service data, and the second service input parameter data is the service corresponding to the second commodity service data in the warm-up table Input parameter data; according to the updated second service input parameter data in the warm-up table, the service data of the database corresponding to the target commodity service is synchronized to the cache service data.

In one of the embodiments, there are multiple first service input parameter data, and obtaining the first commodity service data of the target commodity service from the service data of the database according to the first service input parameter data includes:

Acquire multiple first commodity service data from the service data of the database according to the respective first service input data;

Obtaining the second commodity service data of the target commodity service from the cached service data according to the first service input parameter includes:

Acquire multiple second commodity service data of the target commodity service from the cached service data according to the respective first service input parameters;

When the first product service data is different from the second product service data, update the second service entry data in the warm-up table of the target product service according to the first service entry data, including:

When the first product service data corresponding to any one of the first service input parameters in the multiple first service input parameters is different from the second product service data, update the warm-up table of the target product service according to any of the first service input parameters The corresponding input parameter data of the second service.

In one of the embodiments, synchronizing the service data of the database corresponding to the target commodity service to the cache service data according to the updated second service entry data in the warm-up table includes;

Read the updated second service input parameter data in the warm-up table according to a preset period; and synchronize the service data of the database to the cache service data according to the read second service input parameter data.

In one of the embodiments, the method further includes:

Obtain the incremental data of the target commodity service; obtain the third service input parameter data of the target commodity service from the incremental data; obtain the third commodity service data of the target commodity service from the service data of the database according to the third service input data, And obtain the fourth commodity service data of the target commodity service from the cached service data according to the third service input parameter; when the third commodity service data is different from the fourth commodity service data, update the target commodity service according to the third service input parameter data The fourth service entry data in the warm-up table, the fourth service entry is the service entry corresponding to the fourth commodity service data in the warm-up table; according to the updated fourth service entry database in the warm-up table The service data is synchronized to the cache service data.

In one of the embodiments, obtaining incremental data of the target commodity service includes:

A sliding time window is created according to the update time of the commodity service data of the target commodity service; the incremental data is obtained from the commodity data of the stored target commodity service through the sliding time window.

In one of the embodiments, the method further includes:

Write cache service data into the search service platform; develop the data query interface and abnormal data statistics interface of the search service platform; call the data query interface through the front-end display page to query the product service data in the search service platform through the data query interface, and The front-end display page displays the commodity service data in the search service platform; the abnormal data statistics interface is called through the front-end display page to query the abnormal commodity service data counted in the search service platform through the abnormal data statistics interface, and the search service is displayed on the front-end display page Abnormal commodity service data counted in the platform.

In one of the embodiments, the method further includes:

When the first product service data is different from the second product service data, generate a data log according to the first service input data; write the data log to the search service platform; call the data query interface through the front-end display page to pass the data query interface Query the data log in the search service platform, and display the data log in the search service platform on the front-end display page.

A data processing device for commodity service data, the device comprising:

The first obtaining module is used to obtain the first service input parameter data of the target commodity service from the full data of the target commodity service; the second obtaining module is used to obtain the target commodity from the service data of the database according to the first service input data The first commodity service data of the service; the third obtaining module is used to obtain the second commodity service data of the target commodity service from the cached service data according to the first service input parameter data; the update module is used to compare the first commodity service data with When the second commodity service data is not the same, update the second service input parameter data in the warm-up table of the target commodity service according to the first service input parameter data. The warm-up table is used to store and synchronize the service data of the database to the cached service data. The corresponding service input parameter data, the second service input parameter data is the service input parameter data corresponding to the second commodity service data in the warm-up table; the synchronization module is used to change the second service input parameter data according to the updated second service input data in the warm-up table The service data of the database corresponding to the target commodity service is synchronized to the cache service data.

A computer device includes a memory, a processor, and a computer program that is stored on the memory and can run on the processor. The processor implements the steps of the method in any of the foregoing embodiments when the processor executes the computer program.

A computer-readable storage medium with a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method in any one of the above-mentioned embodiments are implemented.

The above-mentioned data processing method, device, computer equipment and storage medium of the commodity service data obtain the first service entry data of the target commodity service from the full data of the target commodity service, and respectively obtain the service data from the database according to the first service entry data Obtain the first commodity service data and obtain the second commodity service data of the target commodity service from the cached service data. When the comparison of the first commodity service data and the second commodity service data is different, the second service entry data in the warm-up table of the target commodity service is updated according to the first service entry data. Further, the service data of the database is synchronized to the cache service data according to the updated second service entry in the warm-up table. Therefore, the data processing method of the commodity service data does not need to manually update the cached service data, and automatically corresponds to the target commodity service in the commodity service data and the cached service data in the database through the update of the service input parameter data in the warm-up table The comparison of commodity service data is monitored, and the service data in the database is automatically synchronized to the cache service data through the updated service input data in the warm-up table, and the cache update of the commodity service data corresponding to the target commodity service is completed, reducing The cost of artificial maintenance of commodity service data is reduced, the number of manual interventions is reduced, and the timeliness of data processing of commodity service data is improved.

Description of the drawings

Figure 1 is an application environment diagram of a data processing method for commodity service data in an embodiment;

2 is a schematic flowchart of a data processing method for commodity service data in an embodiment;

3 is a schematic flowchart of a data processing method for commodity service data in another embodiment;

FIG. 4 is a schematic diagram of a partial flow of a data processing method for commodity service data in another embodiment;

FIG. 5 is a schematic flowchart of a part of a data processing method for commodity service data in another embodiment;

FIG. 6 is a schematic diagram of a partial flow of a data processing method for commodity service data in another embodiment;

Figure 7 is a structural block diagram of a data processing device for commodity service data in an embodiment;

Fig. 8 is an internal structure diagram of a computer device in an embodiment.

Detailed ways

In order to make the purpose, technical solutions, and advantages of this application clearer and clearer, the following further describes the application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

The data processing method of commodity service data provided in this application can be applied to the application environment as shown in FIG. 1. The database 10 stores a variety of commodity service data, and each commodity service data may be separately stored in the database 10 in the form of a commodity service data storage table. The cache device 30 stores commodity service data used for cache processing in the database 10. Generally, it is necessary to filter out part of the commodity service data from the commodity service data of the database 10, and synchronously cache it in the storage and cache device 30, and perform pre-heat treatment on the commodity service data, so as to improve the reading speed of this part of commodity service data. The server 50 obtains the first service entry data of the target product service from the full data of the target product service in the database 10, and obtains the first product service data of the target product service and the first product service data from the database 10 according to the first service entry data. Obtain the second commodity service data of the target commodity service from the cached service data. When it is determined that the first product service data is different from the second product service data, the second service entry data in the warm-up table of the target product service in the database 10 is updated according to the first service entry data. Among them, the warm-up table of the target commodity service in the database 10 is used to store the corresponding service input parameters when the service data of the database 10 is synchronized to the cached service data in the cache device 30, and the second service-input parameter data is the first in the warm-up table. 2. The service entry data corresponding to the commodity service data. Finally, the server 50 synchronizes the service data of the database corresponding to the target commodity service to the cache service data according to the updated second service entry data in the warm-up table. Among them, the server 50 may be implemented by an independent server or a server cluster composed of multiple servers.

In one embodiment, as shown in FIG. 2, a data processing method for commodity service data is provided. Taking the method applied to the server in FIG. 1 as an example for description, the method includes the following steps:

S100: Obtain the first service entry data of the target commodity service from the full amount of data of the target commodity service.

In this embodiment, the server filters out the corresponding first service entry data from the full data of the target commodity service. The full data of the target commodity service includes the commodity service data of the target commodity service and the service entry data used to read the commodity service data. Among them, the full amount of data of the target commodity service may be the full amount of data stored in the database. The full amount of data can include offline data of the platform. The first service entry data of the target commodity service may be one or more. Specifically, the task computing capability of the big data platform is used to extract the offline full amount of data of the target commodity service to the temporary table of the big data platform, and the full amount of commodity service data of each dimension of the target commodity service is calculated to obtain the first service entry data.

S300: Acquire the first commodity service data of the target commodity service from the service data of the database according to the first service entry data.

When the commodity service management platform receives the commodity service data, it stores the commodity service data in the database. When the server receives the service acquisition request containing the service input parameter data, it retrieves the corresponding commodity service data from the database according to the service input parameter data. In this embodiment, the server obtains the first commodity service data of the target commodity service from the service data of the database. Specifically, the server stores the service entry data of the target commodity service and the correspondence relationship of the commodity service data, and the correspondence relationship between the two is stored in the server in the form of a commodity service information storage table. The server can obtain the corresponding first commodity service data according to the first service input parameter data.

S500: Acquire second commodity service data of the target commodity service from the cached service data according to the first service input parameter data.

In this embodiment, the commodity service management platform pre-heats part of the commodity service data in the database according to business requirements, and stores the part of the service data as cache service data in the cache device. The server reads the second commodity service data of the target commodity service from the cached service data according to the first service input parameter data. Wherein, the second commodity service data is commodity service data obtained by synchronizing the first commodity service data from the database to the caching device for caching. However, the commodity service data of the database will be updated according to the received new commodity service data, and there is a difference between the updated second commodity service data and the first commodity service data.

S700: When the first commodity service data is different from the second commodity service data, update the second service input parameter data in the warm-up table of the target commodity service according to the first service input parameter data, and the warm-up table is used to store the database The service data of is synchronized to the corresponding service entry data when the service data is cached, and the second service entry data is the service entry data corresponding to the second commodity service data in the warm-up table.

In this embodiment, the server sets a corresponding warm-up table according to the characteristics of the target commodity service. The warm-up table stores the corresponding service input data when synchronizing the service data of the database to the corresponding service input data when the service data is cached. That is, the server performs a warm-up operation on the commodity service data of the target commodity service according to the service entry data in the warm-up table. When the server detects that the first commodity service data is different from the second commodity service data, it updates the second service entry data in the warm-up table of the target commodity service according to the first service entry data. Specifically, a message format unsorted comparison algorithm is used to calculate the first commodity service data and the second commodity service data. When the two are inconsistent, write back the warm-up table and replace the first service entry data with the second service entry data in the warm-up table, thus effectively reducing the labor maintenance cost caused by the inconsistency of commodity service data, and improving The accuracy of production environment data is reduced, and the number of complaints caused by merchants and users not being updated in time for product and service data is reduced.

S900: Synchronize the service data of the database corresponding to the target commodity service to the cache service data according to the updated second service entry data in the warm-up table.

In this embodiment, when the server performs data warm-up on the commodity service data of the target commodity service, it reads the updated second service input parameter data from the warm-up table. The corresponding commodity service data is read from the service data of the database through the second service input data, and the corresponding commodity service data is synchronized to the cache service data. Therefore, it is possible to automatically detect the update of the product service data in the database corresponding to the target product service without manual monitoring, and synchronize the product service data in the database to the cache service data when the data is updated.

In an embodiment, S900 includes: reading the updated second service input parameter data in the warm-up table according to a preset period, and synchronizing the service data of the database to the cache service data according to the read second service input parameter data.

The above-mentioned data processing method of commodity service data obtains the first service input parameter data of the target commodity service from the full data of the target commodity service, and obtains the first commodity service data from the service data in the database according to the first service input parameter data, and Obtain the second commodity service data of the target commodity service from the cached service data. When the comparison of the first commodity service data and the second commodity service data is different, the second service entry data in the warm-up table of the target commodity service is updated according to the first service entry data. Further, the service data of the database is synchronized to the cache service data according to the updated second service entry in the warm-up table. Therefore, the data processing method of the commodity service data does not need to manually update the cached service data, and automatically corresponds to the target commodity service in the commodity service data and the cached service data in the database through the update of the service input parameter data in the warm-up table The comparison of commodity service data is monitored, and the service data in the database is automatically synchronized to the cache service data through the updated service input data in the warm-up table, and the cache update of the commodity service data corresponding to the target commodity service is completed, reducing The cost of artificial maintenance of commodity service data is reduced, the number of manual interventions is reduced, and the timeliness of data processing of commodity service data is improved.

In one embodiment, there are multiple first service entry data in the full data of the target commodity service. As shown in Figure 3, S300 includes steps:

S301: Acquire multiple first commodity service data from the service data of the database according to the respective first service input data.

S500 includes steps:

S501: Acquire multiple second commodity service data of the target commodity service from the cached service data according to each first service input parameter.

S700 includes steps:

S701: When the first product service data corresponding to any one of the first service input parameters in the plurality of first service input parameters is different from the second product service data, update the warm-up of the target product service according to any one of the first service input parameters The corresponding entry data of the second service in the table.

In this embodiment, the server obtains multiple first service input parameter data of the target commodity service from the full data of the target commodity service, and obtains multiple first service input parameter data from the service data of the database according to each first service input parameter data. The commodity service data and a plurality of second commodity service data of the target commodity service obtained from the cached service data. Further, each corresponding first commodity service data and second commodity service data are respectively compared. When any comparison result shows that the first product service data and the second product service data are different, the first service entry data corresponding to the result is used to update the corresponding second service entry data in the warm-up table. At this time, there are also multiple input parameter data for the second service. Finally, the service data of the database is synchronized to the cache service data according to all the updated second service entries in the warm-up table. Therefore, it is possible to automatically monitor the update status of all product service data corresponding to the target product service, and automatically synchronize the service data of the database to the cache service data according to the update result.

In a specific implementation, the temporary table of the big data platform is designed, and the full amount of basic data of the database is extracted to the temporary table through database middleware, such as Mycat middleware. Among them, the table structure of different temporary tables is designed according to the dimensions of different goods and services and their associated data, which is different from the difference between the data types of temporary tables and database tables, and the specific field types in the temporary tables can be mapped. In an implementation process, the target commodity service is a commodity basic information service, and the first service entry data is a commodity code. The commodity code is extracted from the commodity basic information table of the commodity basic information service. The merchant code is used as the key field for service query and cache query. The temporary table creation statement is designed as follows:

CREATE TABLE`hive_product_base_info`(

`partnumber`string COMMENT' commodity code',

`suppliercode`string COMMENT'merchant code',

COMMENT'Commodity basic information service key field table',

ROW FORMAT SERDE

′Org.apache.hadoop.hive.ql.io.orc.OrcSerde′

STORED AS INPUTFORMAT

‘Org.apache.hadoop.hive.ql.io.orc.OrcInputFormat’

OUTPUTFORMAT

‘Org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat’);

The above can create a temporary table, through the big data platform, extract the required field data in the basic information table of the Mycat data source product, and then on the big data platform, through the data exploration tool, you can query the data integrity in the hive_product_base_info table to verify The correctness of the service data extraction process.

After the above steps are completed, use the computing power of the big data platform to develop tasks to calculate the temporary table data, and use the message format unsorted comparison algorithm to compare the commodity service data in the database with the cached service data. If the data is inconsistent, the service will be added to the reference. Data, such as commodity codes, are written into the new temporary table. After the calculation is completed, the service input data in the new temporary table is exchanged through data, and the service input parameters in the warm-up table are written back. In the process, the monitoring logic is designed, and the data log is generated and written into the real-time data stream. During specific implementation, a spark package is created in the project, and a big data task calculation class for a specific service is created. For example, product basic information service, create a new SparkWarmProductBaseInfoHivejava class, write a main function, create SparkConf, JavaSparkContext, and HiveContext in turn to implement configuration read and load initialization, HiveContext instance sqlContext, support Spark SQL query, extract Hive table hive_product_base_info, generate Dataset<Row> The data source is transformed into a javaRDD object, the table data is traversed, each row of data is obtained in the loop body, the basic information service data of the goods in the cache and the database are respectively queried, and the message format unsorted comparison algorithm is used to compare whether the service data in the cache and the database are Completely consistent, write inconsistent service data into the newly created comparison result Hive table. Pack the SparkWarmProductBaseInfoHive class to generate a jar file, upload it to the big data platform, get the Hive table of the comparison result after running, and then write the service input parameter data into the database warm-up table in the form of data exchange, monitor all service data in the process, and generate Logs are written into the real-time data stream. After implementation, you can select part of the data through the Hive table for verification, and verify that the service data in the cache and the database are consistent.

In an embodiment, as shown in FIG. 4, after S900, the method further includes the following steps:

S1000: Obtain incremental data of the target commodity service.

S1200: Obtain the third service entry data of the target commodity service from the incremental data.

S1400: Obtain third commodity service data of the target commodity service from the service data of the database according to the third service input parameter data, and obtain fourth commodity service data of the target commodity service from the cached service data according to the third service input parameter.

S1600: When the third product service data is different from the fourth product service data, update the fourth service entry data in the warm-up table of the target product service according to the third service entry data, and the fourth service entry is the warm-up The service entry corresponding to the fourth commodity service data in the table.

S1800: Synchronize the service data of the database to the cache service data according to the updated fourth service entry in the warm-up table.

In this embodiment, after the server executes the above S100 to S900 according to the full data of the target commodity service, the subsequent server only needs to monitor the incremental data of the target commodity service when monitoring the commodity service data of the target commodity service. . Specifically, obtain the third service entry data from the incremental data, obtain the third product service data of the target product service from the service data of the database according to the third service entry data, and obtain the target product service data from the cached service data The fourth commodity service data. Wherein, the third commodity service data may be commodity service data in incremental data. Further, when the third product service data is different from the fourth product service data, the fourth service entry data in the warm-up table of the target product service is updated according to the third service entry data. Finally, the service data of the database is synchronized to the cache service data according to the updated fourth service entry in the warm-up table. Therefore, it is possible to monitor the update of the commodity service data in the incremental data of the subsequent target commodity service in the platform, and automatically update the corresponding cache service data.

In an implementation of this embodiment, S1000 includes: creating a sliding time window according to the update time of the commodity service data of the target commodity service, and obtaining incremental data from the commodity data of the stored target commodity service through the sliding time window.

In this embodiment, the incremental data of the commodity service data of the target commodity service is obtained through a sliding time window. Specifically, a sliding time window is created, and the sliding time window can be dynamically configured. For example, the value range of the sliding time window is determined according to the update time of the commodity service data table of the target commodity service. The value range of the sliding time window is expressed as: start time ±N*window time, end time ±N*window time. The specific way to obtain incremental data from the commodity service storage table storing the commodity service data of the target commodity service through the sliding time window is: obtain the commodity service data of the target commodity service from the commodity service storage table corresponding to the target commodity and service in the database. Table update time, the update time is within the range of [start time±N*window time, end time±N*window time) to obtain the commodity data of the target commodity service, and the obtained commodity data is the incremental data.

In one implementation of this embodiment, for the incremental data of the target commodity service, the warm-up logic is: store the inconsistent service entry data into the warm-up table of the target commodity service, and the service entry parameters in the warm-up table The data triggers real-time calculations through timed tasks. During calculation, the commodity service data corresponding to the service input parameter data in the database is taken in real time to update the currently cached commodity service data. At the same time, the update time in the warm-up table is added as the version number of the cached data , In order to obtain the inconsistency of the product and service data through the comparison result of the version number when the data is compared next time, reduce the execution time of the comparison algorithm, and improve the comparison efficiency.

Therefore, the commodity service data of the target commodity service is obtained through a sliding time window, and the consistency of the commodity service data in the cache and the database is compared. When inconsistent, the service input data is extracted and written back to re-warm, in order to realize the commodity service data Operation and maintenance monitoring. When commodity service data is updated due to delays, physical machine downtime, database master-slave unsynchronization, etc., inconsistencies are caused, through this technical means, the process is monitored, the commodity service data is operated and maintained, and the commodity service data accurate service is provided. .

For this embodiment, a specific implementation is provided as follows:

Develop the sliding time window time slice secondary warm-up task abstract class of the development service, and the secondary warm-up task abstract class implements data source definition, current service definition, service data query by time slice, cache key generation, cache query, database table General methods such as service data query, message format unsorted comparison algorithm, process status monitoring method, etc. In the specific implementation, create a new rewarmer package, create a new base package under it, and create a new AbstractRewarmer abstract class, which completes the automatic assembly instantiation of the Mycat data source, and defines the following abstract methods:

getService, used to obtain the service to which the secondary warm-up belongs;

queryChangeData, used to query the change data in the specific service table;

generalCacheKey, used to call the cache CacheKey template generator to generate the cache key;

generalDBValue, used to call the Mycat data source to query the basic information of a specific service;

compareData, an algorithm used to compare packet formats without ordering consistency;

insert2Warmer, used to write parameter data into the warm-up table in batches;

getInitParams, used to obtain the initial parameter list, including the time range startTime, endTime;

historyCacheKeyGenerator, used to generate historical information record cache key;

runHistoryTimeCacheKeyGenerator, used to generate historical run time range cache key;

stateCacheKeyGenerator, used to generate the state cache key for the second preheating job of the service;

totalCacheKeyGenerator, used to generate the cache key for the total preheated data volume of the service's secondary warm-up job history every day;

getRunningState, used to get the running status of the job;

Define the instance method:

run, used for job execution, acquiring status locks, locking and releasing locks, and exception capture;

execute, used for overall job execution and business processing.

After the abstract class definition is complete, you need to create a new implementation class. For example, the target commodity service is commodity basic information service: Create a new RewarmerProductBaseInfo under the revarmer package to implement the abstract methods in the abstract class and the basic definition of the realization class. The implemented business logic queries the commodity service data within the agreed time range (such as the last 5 minutes) of the current job operation from the commodity basic information table, and queries the commodity service data from the cache for comparison according to the CacheKey. If the product and service data is inconsistent, the service data is written into the warm-up table, and the current service status of the cache is updated at the same time. The execution time range of the next stage Job will move forward for 5 minutes.

After the task definition is completed, configure the Spring Quartz timer setting tool, start the job configured in the above steps, complete the time slice secondary warm-up function of each service, and implement the service self-operation and maintenance strategy. In specific implementation, you can configure Quartz's timer CronExpression, which starts at 0 minute and 0 second every hour and triggers every 5 minutes. After starting the project, check the running status of the job and verify that if the goods and service data in the cache and the database are inconsistent within the corresponding time range, whether the data will be written into the warm-up table after calculation, perform the second warm-up of the service, and verify the job When the second time is triggered, whether the range time slice will move 5 minutes later, compare the service data within 5 minutes after the query, and write the inconsistent service data after the comparison into the warm-up table.

After the task is completed, an abstract class of preheating tasks is developed to realize general methods such as reading of preheating tables, updating and writing of cache, querying of database service data, and business logic calculation methods of commodity service data. For specific implementation, create a warmer package, a base package under the warmer package, and a new AbstractWarmer abstract class. Define the above abstract methods. After the abstract class is completed, the AbstractWarmer abstract class is inherited under the warmer package according to different services to realize the preheating function of each specific service. After completion, the timed task is configured with the above steps to complete the preheating function of the service data.

In an embodiment, as shown in FIG. 5, before S100, the method further includes:

S101: Write the cache service data to the search service platform.

S103: Develop a data query interface and an abnormal data statistics interface of the search service platform.

S105: Call the data query interface through the front-end display page to query the product service data in the search service platform through the data query interface, and display the product service data in the search service platform on the front-end display page.

S107: Call the abnormal data statistics interface through the front-end display page to query the abnormal commodity service data counted in the search service platform through the abnormal data statistics interface, and display the commodity abnormal service data counted in the search service platform on the front-end display page.

In this embodiment, the server writes the cached service data as the commodity basic data of the target commodity service into the search service platform. In addition, the server increases the monitoring of the receipt and storage of the basic commodity data of the target commodity service, and the basic commodity data received is generated into a log and written into the real-time data stream. Further, it is verified whether the real-time data stream is written into the search service platform by middleware consumption, and the normal writing ends verification. At the same time, a data query interface and an abnormal data statistics interface are developed based on the search service platform for the front-end display page to call. Display the commodity service data in the search service platform and display the commodity abnormal service data counted in the search service platform through the front-end display page.

In a specific embodiment, monitoring is added to the receipt and storage of commodity basic data of the target commodity service, and the received data is generated into a log and written into the real-time data stream. During specific implementation, a system configuration switch can be added to the receive data method class execute of the abstract class AbstractMessageReceiveService for XML message reception processing, and the specific switch name can be set to log.switch. The switch is a disaster recovery method to control abnormal conditions. Disaster recovery methods such as abnormalities or sharp increases in data volume. Use the switch to realize the analysis of the received message and the conversion of the object format, and then encapsulate it into the message format data for real-time data stream transmission specified by the protocol. Set the transmission object name to MonitorLogDto, including fields: serial number, service type, service code, processing time, number of message items, creation time, abnormal information, message content, etc. The data transmission method is not limited to bytes The form of flow. You can define the real-time data stream as system_monitor_log, write the tool class of production data, and then write the encapsulated monitoring data into the real-time data stream queue. After writing, you can simulate consumption on the real-time data stream management platform to verify whether the data and format are correct. After completing the step of writing monitoring data into the real-time data stream, continue to verify whether the real-time data stream data is consumed and written into the search service platform. In the implementation, you need to configure middleware to consume the data in the real-time data stream queue. Here, the data stream GroupId is defined as groupid_system_monitor_log. After the configuration is completed, you can check the configured index and type on the search service platform to see if there is data written to the platform by consumption, and the verification is complete if there is data written and the format is correct.

After verifying the above steps, it is necessary to develop a data query interface and an abnormal data statistics interface based on the search service platform for the front-end monitoring page to call. During the specific implementation, configure the project project, use ElasticSearch (hereinafter referred to as es) as the search service platform, modify the pom.xml configuration file, and add the dependency groupId to org.elasticsearch.client and artifactId to transport. Add the tool package tool to the project, create the es client tool class ElasticsearchConfig, configure the cluster name, ip address and port port number, and create the es client connection to implement the es basic tool class. Create a new controller package and service package to realize the paging query and statistics function of es on the monitoring business data interface. At this time, the QueryBuilders provided by the es client jar package are used, including wildcardQuery fuzzy query, termQuery precise query, boolQuery combined query, fliter filter query, etc., and AggregationBuilders, including sum sum, avg average, count count, dateHistogram aggregation query Wait. From the monitoring perspective, multi-dimensional data query and statistics functions are realized. The front-end monitoring page verifies whether the controller interface designed in Spring rest style can normally call the data, and multiple interfaces are designed to complete the data needs of the front-end page. The details are as follows (the following interface Both support Http request in Get mode):

queryData, according to the input service code and time range, query the monitoring data of es by time range aggregation;

queryGroupData, according to the input service code and time range, query the monitoring data of es grouped by time range;

queryTodayData, according to the entered service code, query all the service data of the monitoring and statistics of the day;

queryExceptionData, according to the input service code, time range, paging condition (large amount of data, support paging query), message keyword, query the monitored service exception data.

In an embodiment, as shown in FIG. 6, after S900, the method further includes the following steps:

S901: When the first commodity service data is different from the second commodity service data, generate a data log according to the first service input data.

S903: Write the data log to the search service platform.

S905: Call the data query interface through the front-end display page to query the data log in the search service platform through the data query interface, and display the data log in the search service platform on the front-end display page.

Specifically, the data log refers to the data log used for statistical analysis, and the dimensions are also specific to the dimensions of each commodity and service. When it is monitored that the first commodity service data is the same as the second commodity service data, the data is normally written into the database. However, there will be performance bottlenecks when doing statistical analysis of data (for example, the query is too slow, and the statistical analysis of billion-level service data cannot be obtained), so the data is recorded in the form of logs, written into the real-time data stream, and the search service platform is the result data Provide retrieval and statistical analysis, so that the front-end monitoring page can display the monitoring results more friendly. Therefore, the monitoring of the commodity service data of the target commodity service is more refined, which is beneficial to improve the timeliness of the data processing of the commodity service data of the target commodity service.

It should be understood that although the various steps in the flowchart are displayed in sequence as indicated by the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless specifically stated in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The order of execution of these sub-steps or stages It is not necessarily performed sequentially, but may be performed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

In one embodiment, as shown in FIG. 7, a data processing device for commodity service data is provided, which includes a first acquisition module 11, a second acquisition module 13, a third acquisition module 15, an update module 17, and a synchronization module 19. .

The first obtaining module 11 is configured to obtain the first service entry data of the target commodity service from the full amount of data of the target commodity service.

The second obtaining module 13 is configured to obtain the first commodity service data of the target commodity service from the service data of the database according to the first service input data.

The third obtaining module 15 is configured to obtain the second commodity service data of the target commodity service from the cached service data according to the first service input parameter data.

The update module 17 is used to update the second service input parameter data in the warm-up table of the target merchandise service according to the first service input parameter data when the first commodity service data is different from the second commodity service data. To store the corresponding service input parameter data when synchronizing the service data of the database to the cache service data, the second service input parameter data is the service input parameter data corresponding to the second commodity service data in the warm-up table.

The synchronization module 19 is configured to synchronize the service data of the database corresponding to the target commodity service to the cache service data according to the updated second service input data in the preheating table.

In one of the embodiments, there are multiple input parameter data for the first service, and the second acquiring module 13 may include (not shown in FIG. 7):

The second obtaining unit is configured to obtain a plurality of first commodity service data from the service data of the database according to the respective first service input data.

The third acquisition module 15 includes:

The third obtaining unit is configured to obtain multiple second commodity service data of the target commodity service from the cached service data according to the respective first service input parameters.

Update module 17, including:

The updating unit is configured to update the target commodity service according to any one of the first service input parameters when the first commodity service data corresponding to any one of the first service input parameters is different from the second commodity service data The corresponding second service input parameter data in the warm-up table of.

In one of the embodiments, the synchronization module 19 includes (not shown in Fig. 7):

The reading unit is configured to read the updated second service input parameter data in the warm-up table according to a preset period.

The update unit is used to synchronize the service data of the database to the cache service data according to the read second service input parameter data.

In one of the embodiments, a data processing device for commodity service data may include (not shown in Fig. 7):

The fourth acquiring module is used to acquire incremental data of the target commodity service.

The fifth acquiring module is used to acquire the third service entry data of the target commodity service from the incremental data.

The sixth obtaining module is used to obtain the third commodity service data of the target commodity service from the service data of the database according to the third service input parameter data, and obtain the fourth commodity service data of the target commodity service from the cached service data according to the third service input parameter Commodity service data.

The replacement module is used to update the fourth service entry data in the warm-up table of the target product service according to the third service entry data when the third product service data is different from the fourth product service data. Enter the service parameter corresponding to the fourth commodity service data in the warm-up table.

The preheating module is used for synchronizing the service data of the database to the cached service data according to the updated fourth service entry in the preheating table.

In one of the embodiments, the fourth acquiring module may include (not shown in FIG. 7):

The creation unit is used to create a sliding time window according to the update time of the commodity service data of the target commodity service;

The fourth acquiring unit is configured to acquire incremental data from the product data of the stored target product service through a sliding time window.

The cache module is used to write cache service data into the search service platform.

The development module is used to develop the data query interface and abnormal data statistics interface of the search service platform.

The first display module is configured to call the data query interface through the front-end display page to query the product service data in the search service platform through the data query interface, and display the product service data in the search service platform on the front-end display page.

The second display module is used to call the abnormal data statistics interface through the front-end display page to query the abnormal commodity service data counted in the search service platform through the abnormal data statistics interface, and display the abnormality counted in the search service platform on the front-end display page Commodity service data.

The generating module is used to generate a data log according to the first service input data when the first commodity service data is different from the second commodity service data.

The writing module is used to write the data log to the search service platform.

The third display module is used to call the data query interface through the front-end display page to query the data log in the search service platform through the data query interface, and display the data log in the search service platform on the front-end display page.

For the specific definition of a data processing device for commodity service data, please refer to the above definition of a data processing method for commodity service data, which will not be repeated here. Each module in the above-mentioned data processing device for commodity service data can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules may be embedded in the form of hardware or independent of the processor in the computer equipment, or may be stored in the memory of the computer equipment in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. 8. The computer equipment includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer equipment is used to store relevant data of the target commodity service, such as service entry data. The network interface of the computer device is used to communicate with an external terminal through a network connection. The computer program is executed by the processor to realize a data processing method of commodity service data.

Those skilled in the art can understand that the structure shown in FIG. 8 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may Including more or fewer parts than shown in the figure, or combining some parts, or having a different arrangement of parts.

In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the following steps when the processor executes the computer program:

In an embodiment, there are multiple first service input parameter data, and when the processor executes the computer program to implement the above-mentioned step of obtaining the first product service data of the target product service from the service data of the database according to the first service input parameter data, The following steps are specifically implemented: obtaining multiple first commodity service data from the service data in the database according to each first service input parameter data; the processor executes the computer program to realize the above-mentioned obtaining target from the cache service data according to the first service input parameter In the second commodity service data step of commodity service, the following steps are specifically implemented: obtain multiple second commodity service data of the target commodity service from the cached service data according to each first service input parameter; the processor executes the computer program to realize the above When the first commodity service data is different from the second commodity service data, when the second service entry data step in the warm-up table of the target commodity service is updated according to the first service entry data, the following steps are specifically implemented: When the first product service data corresponding to any one of the first service input parameters in the first service input parameter is different from the second product service data, the corresponding first product service data in the warm-up table of the target product service is updated according to any first service input parameter. 2. Service entry data.

In one embodiment, when the processor executes the computer program to realize the step of synchronizing the service data of the database corresponding to the target commodity service to the cache service data according to the updated second service entry data in the warm-up table, the following steps are specifically implemented : Read the updated second service input parameter data in the warm-up table according to the preset period; synchronize the service data of the database to the cache service data according to the read second service input parameter data.

In one embodiment, the processor executes the computer program to implement the following steps: obtain incremental data of the target commodity service; obtain the third service entry data of the target commodity service from the incremental data; obtain the third service entry data from the database according to the third service entry data Obtain the third commodity service data of the target commodity service from the service data, and obtain the fourth commodity service data of the target commodity service from the cached service data according to the third service input parameter; when the third commodity service data and the fourth commodity service data When they are not the same, update the fourth service entry data in the warm-up table of the target product service according to the third service entry data, and the fourth service entry is the service entry corresponding to the fourth product service data in the warm-up table; The updated fourth service entry in the warm-up table synchronizes the service data of the database to the cache service data.

In one embodiment, when the processor executes the computer program to implement the above step of obtaining the incremental data of the target commodity service, it specifically implements the following steps: creating a sliding time window according to the update time of the commodity service data of the target commodity service; using the sliding time window Obtain incremental data from the product data of the storage target product service.

In one embodiment, the processor executes the computer program to implement the following steps: write cache service data into the search service platform; develop the data query interface and abnormal data statistics interface of the search service platform; call the data query interface through the front-end display page to pass The data query interface queries the product service data in the search service platform, and displays the product service data in the search service platform on the front-end display page; calls the abnormal data statistics interface through the front-end display page to query the statistics in the search service platform through the abnormal data statistics interface The abnormal commodity and service data that is generated, and the abnormal commodity and service data counted in the search service platform are displayed on the front-end display page.

In one embodiment, the processor executes the computer program to implement the following steps: when the first commodity service data is different from the second commodity service data, generate a data log according to the first service input data; write the data log into the search Service platform: call the data query interface through the front-end display page to query the data log in the search service platform through the data query interface, and display the data log in the search service platform on the front-end display page.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the data processing method for commodity service data described in any of the above embodiments is implemented.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

In one embodiment, there are multiple first service input parameter data, and the computer program is executed by the processor to realize the above-mentioned step of obtaining the first product service data of the target product service from the service data of the database according to the first service input parameter data. , Specifically implement the following steps: obtain a plurality of first commodity service data from the service data of the database according to each first service input parameter data; the computer program is executed by the processor to realize the above-mentioned cached service data according to the first service input parameter In the step of obtaining the second commodity service data of the target commodity service, the following steps are specifically implemented: obtaining multiple second commodity service data of the target commodity service from the cached service data according to each first service input parameter; the computer program is executed by the processor When the above-mentioned step of updating the second service entry data in the warm-up table of the target product service according to the first service entry data when the first product service data is different from the second product service data is implemented, the following steps are specifically implemented: When the first product service data corresponding to any one of the first service input parameters in the multiple first service input parameters is different from the second product service data, update the warm-up table of the target product service according to any of the first service input parameters The corresponding input parameter data of the second service.

In one embodiment, when the computer program is executed by the processor to realize the step of synchronizing the service data of the database corresponding to the target commodity service to the cache service data according to the updated second service input parameter data in the warm-up table, the following is specifically realized Steps: reading the updated second service input parameter data in the warm-up table according to the preset period; and synchronizing the service data of the database to the cache service data according to the read second service input parameter data.

In one embodiment, the computer program is executed by the processor to achieve the following steps: obtain incremental data of the target commodity service; obtain the third service entry data of the target commodity service from the incremental data; Obtain the third commodity service data of the target commodity service from the service data of the database, and obtain the fourth commodity service data of the target commodity service from the cached service data according to the third service input parameter; when the third commodity service data and the fourth commodity service data When the data is not the same, update the fourth service entry data in the warm-up table of the target product service according to the third service entry data, and the fourth service entry is the service entry corresponding to the fourth product service data in the warm-up table; The service data of the database is synchronized to the cache service data according to the updated fourth service entry in the warm-up table.

In one embodiment, when the computer program is executed by the processor to achieve the above step of obtaining the incremental data of the target commodity service, the following steps are specifically implemented: creating a sliding time window according to the update time of the commodity service data of the target commodity service; The window obtains incremental data from the product data of the stored target product service.

In one embodiment, the computer program is executed by the processor to implement the following steps: write cache service data into the search service platform; develop the data query interface and abnormal data statistics interface of the search service platform; call the data query interface through the front-end display page to Query the product service data in the search service platform through the data query interface, and display the product service data in the search service platform on the front-end display page; call the abnormal data statistics interface through the front-end display page to query the search service platform through the abnormal data statistics interface Statistics of abnormal commodity and service data, and display the abnormal commodity and service data counted in the search service platform on the front-end display page.

In one embodiment, the computer program is executed by the processor to implement the following steps: when the first commodity service data is different from the second commodity service data, generate a data log according to the first service input data; write the data log to the Search service platform; call the data query interface through the front-end display page to query the data log in the search service platform through the data query interface, and display the data log in the search service platform on the front-end display page.

A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage. In the medium, when the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database, or other media used in the embodiments provided in this application may include non-volatile and/or volatile memory. Non-volatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not a limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Channel (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be It is considered as the range described in this specification.

The above-mentioned embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the invention patent. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can be made, and these all fall within the protection scope of this application. Therefore, the scope of protection of the patent of this application shall be subject to the appended claims.

Claims

A data processing method for commodity service data, the method comprising:

Acquiring the first service entry data of the target commodity service from the full data of the target commodity service;

Acquiring the first commodity service data of the target commodity service from the service data of the database according to the first service entry data;

Acquiring second commodity service data of the target commodity service from cached service data according to the first service entry parameter data;

When the first commodity service data is different from the second commodity service data, update the second service entry data in the warm-up table of the target commodity service according to the first service entry data, and the The warm-up table is used to store the corresponding service input parameter data when the service data of the database is synchronized to the cached service data, and the second service input parameter data is the second commodity service data in the warm-up table Corresponding service entry data;

Synchronizing the service data of the database corresponding to the target commodity service to the cache service data according to the updated second service entry data in the warm-up table.
The method according to claim 1, wherein the first service entry data is multiple, and the first service entry data of the target commodity service is obtained from the service data of the database according to the first service entry data. 1. Commodity and service data, including:

Acquiring a plurality of first commodity service data from the service data of the database according to each of the first service input data;

The obtaining the second commodity service data of the target commodity service from cached service data according to the first service input parameter includes:

Acquiring a plurality of second commodity service data of the target commodity service from cached service data according to each of the first service input parameters;

When the first commodity service data is different from the second commodity service data, updating the second service entry data in the warm-up table of the target commodity service according to the first service entry data, include:

When the first product service data corresponding to any one of the first service input parameters among the plurality of first service input parameters is different from the second product service data, according to any one of the first service input parameters The service input parameter updates the corresponding second service input parameter data in the warm-up table of the target commodity service.
The method according to claim 1, wherein the service data of the database corresponding to the target commodity service is synchronized to the cache according to the updated second service entry data in the warm-up table Service data, including;

Read the updated second service input parameter data in the warm-up table according to a preset period;

Synchronize the service data of the database to the cache service data according to the read second service input parameter data.
The method according to claim 1, wherein the method further comprises:

Acquiring incremental data of the target commodity service;

Acquiring the third service entry data of the target commodity service from the incremental data;

Obtain the third commodity service data of the target commodity service from the service data of the database according to the third service input parameter data, and obtain the target from the cache service data according to the third service input parameter The fourth commodity service data of commodity service;

When the third commodity service data is different from the fourth commodity service data, the fourth service entry data in the warm-up table of the target commodity service is updated according to the third service entry data, and the The fourth service entry is the service entry corresponding to the fourth commodity service data in the warm-up table;

Synchronizing the service data of the database to the cache service data according to the updated fourth service entry in the warm-up table.
The method according to claim 4, wherein said acquiring incremental data of said target commodity service comprises:

Creating a sliding time window according to the update time of the commodity service data of the target commodity service;

The incremental data is obtained from the product data of the stored target product service through a sliding time window.
The method according to any one of claims 1 to 4, wherein the method further comprises:

Writing the cache service data to the search service platform;

Develop the data query interface and abnormal data statistics interface of the search service platform;

Calling the data query interface through the front-end display page to query the product service data in the search service platform through the data query interface, and display the product service data in the search service platform on the front-end display page;

Call the abnormal data statistics interface through the front-end display page to query the abnormal commodity service data counted in the search service platform through the abnormal data statistics interface, and display the statistics in the search service platform on the front-end display page Abnormal commodity service data.
The method according to claim 6, wherein the method further comprises:

When the first commodity service data is different from the second commodity service data, generating a data log according to the first service entry data;

Writing the data log to the search service platform;

Call the data query interface through the front-end display page to query the data log in the search service platform through the data query interface, and display the data log in the search service platform on the front-end display page .
A data processing device for commodity service data, characterized in that the device comprises:

The first obtaining module is configured to obtain the first service entry data of the target commodity service from the full data of the target commodity service;

The second obtaining module is configured to obtain the first commodity service data of the target commodity service from the service data of the database according to the first service entry data;

The third obtaining module is configured to obtain the second commodity service data of the target commodity service from the cached service data according to the first service input parameter data;

The update module is configured to update the second service entry in the warm-up table of the target product service according to the first service entry data when the first product service data is different from the second product service data Parameter data, the warm-up table is used to store the corresponding service input parameter data when synchronizing the service data of the database to the cached service data, and the second service input parameter data is the one in the warm-up table Service entry data corresponding to the second commodity service data;

The synchronization module is configured to synchronize the service data of the database corresponding to the target commodity service to the cache service data according to the updated second service input data in the warm-up table.
A computer device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, wherein the processor implements any one of claims 1 to 7 when the computer program is executed The steps of the method.
A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the method according to any one of claims 1 to 7 when the computer program is executed by a processor.