CN111222089B - Data processing method, data processing device, computer equipment and storage medium - Google Patents

Data processing method, data processing device, computer equipment and storage medium Download PDF

Info

Publication number
CN111222089B
CN111222089B CN202010287873.0A CN202010287873A CN111222089B CN 111222089 B CN111222089 B CN 111222089B CN 202010287873 A CN202010287873 A CN 202010287873A CN 111222089 B CN111222089 B CN 111222089B
Authority
CN
China
Prior art keywords
service
data
commodity
parameter
service data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010287873.0A
Other languages
Chinese (zh)
Other versions
CN111222089A (en
Inventor
邰华运
吕勇
万鹏程
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NANJING SUNING ELECTRONIC INFORMATION TECHNOLOGY Co.,Ltd.
Original Assignee
Suning Cloud Computing Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suning Cloud Computing Co Ltd filed Critical Suning Cloud Computing Co Ltd
Priority to CN202010287873.0A priority Critical patent/CN111222089B/en
Publication of CN111222089A publication Critical patent/CN111222089A/en
Priority to PCT/CN2020/096997 priority patent/WO2021208228A1/en
Priority to CA3177218A priority patent/CA3177218A1/en
Application granted granted Critical
Publication of CN111222089B publication Critical patent/CN111222089B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/14—Fourier, Walsh or analogous domain transformations, e.g. Laplace, Hilbert, Karhunen-Loeve, transforms
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01R—MEASURING ELECTRIC VARIABLES; MEASURING MAGNETIC VARIABLES
    • G01R29/00—Arrangements for measuring or indicating electric quantities not covered by groups G01R19/00 - G01R27/00
    • G01R29/26—Measuring noise figure; Measuring signal-to-noise ratio
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G06F2218/02—Preprocessing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G06F2218/08—Feature extraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G06F2218/12—Classification; Matching

Abstract

The present application relates to the technical field of commodity service data processing, and in particular, to a data processing method and apparatus for commodity service data, a computer device, and a storage medium. The method comprises the following steps: acquiring first service parameter data of the target commodity service from the full data of the target commodity service; acquiring first commodity service data of the target commodity service from the service data of the database according to the first service parameter data, and acquiring second commodity service data of the target commodity service from the cache service data; when the first commodity service data is different from the second commodity service data, updating second service entry parameter data in a preheating table of the target commodity service according to the first service entry parameter data; and synchronizing the service data of the database corresponding to the target commodity service to the cache service data according to the updated second service entry parameter data in the preheating table. The method can automatically monitor the updating and caching process of the commodity service data.

Description

Data processing method, data processing device, computer equipment and storage medium
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data processing method and apparatus for commodity service data, a computer device, and a storage medium.
Background
The internet commodity service management platform is mainly used for providing data services of commodity information, wherein the data services comprise data distribution and receiving, warehousing, cache preheating, data services, downstream system service push notification and other basic functions. However, the commodity service data are differentiated by service, and the platform stores the service data according to the service scene in a multi-dimensional database and table division mode. Therefore, the monitoring of the data processing of the commodity service data by the platform is not fine enough. In particular, there is a lack of monitoring of the data processing process for synchronizing service data in a database to cached service data. For the data processing process, the traditional monitoring mode is that when the condition that the service data of the database in the production environment is inconsistent with the cache service data is manually detected, corresponding entry parameters are manually input, and the service data in the database is synchronized to the cache service data according to the corresponding entry parameters. The manual monitoring and manual operation mode results in low timeliness of data processing of commodity service data.
Disclosure of Invention
In view of the above, it is necessary to provide a data processing method, an apparatus, a computer device and a storage medium for merchandise service data, which can automatically monitor the processing procedure of merchandise service data update cache and improve the timeliness of updating the cached merchandise service data.
A data processing method of commodity service data comprises the following steps:
acquiring first service parameter data of the target commodity service from the full data of the target commodity service; acquiring first commodity service data of a target commodity service from service data of a database according to the first service access data; acquiring second commodity service data of the target commodity service from the cache service data according to the first service entry parameter data; when the first commodity service data is different from the second commodity service data, updating second service parameter entering data in a preheating table of the target commodity service according to the first service parameter entering data, wherein the preheating table is used for storing corresponding service parameter entering data when the service data of the database is synchronized to the cache service data, and the second service parameter entering data is service parameter entering data corresponding to the second commodity service data in the preheating table; and synchronizing the service data of the database corresponding to the target commodity service to the cache service data according to the updated second service entry parameter data in the preheating table.
In one embodiment, the obtaining of the first commodity service data of the target commodity service from the service data of the database according to the first service parameter data includes:
acquiring a plurality of first commodity service data from the service data of the database according to each first service parameter data;
acquiring second commodity service data of the target commodity service from the cache service data according to the first service entry parameter, wherein the second commodity service data comprises:
respectively acquiring a plurality of second commodity service data of the target commodity service from the cache service data according to each first service entry parameter;
when the first commodity service data is different from the second commodity service data, updating second service entry parameter data in a preheating table of the target commodity service according to the first service entry parameter data, wherein the second service entry parameter data comprises:
and when the first commodity service data corresponding to any one of the first service access parameters is different from the second commodity service data, updating the corresponding second service access parameter data in the preheating table of the target commodity service according to any one of the first service access parameters.
In one embodiment, synchronizing service data of a database corresponding to the target commodity service to cache service data according to the updated second service entry data in the preheating table includes:
reading second service access parameter data updated in the preheating table according to a preset period; and synchronizing the service data of the database to the cache service data according to the read second service access data.
In one embodiment, the method further comprises:
obtaining incremental data of target commodity service; acquiring third service parameter data of the target commodity service from the incremental data; acquiring third commodity service data of the target commodity service from the service data of the database according to the third service parameter, and acquiring fourth commodity service data of the target commodity service from the cache service data according to the third service parameter; when the third commodity service data is different from the fourth commodity service data, updating fourth service participation data in a preheating table of the target commodity service according to the third service participation parameter data, wherein the fourth service participation is service participation corresponding to the fourth commodity service data in the preheating table; and synchronizing the service data of the database to the cache service data according to the updated fourth service entry in the preheating table.
In one embodiment, obtaining incremental data for a target commodity service includes:
creating a sliding time window according to the updating time of the commodity service data of the target commodity service; and acquiring incremental data from the commodity data of the storage target commodity service through a sliding time window.
In one embodiment, the method further comprises:
writing the cache service data into a search service platform; developing a data query interface and an abnormal data statistical interface of a search service platform; calling a data query interface through a front-end display page to query the commodity service data in the search service platform through the data query interface, and displaying the commodity service data in the search service platform on the front-end display page; and calling an abnormal data statistical interface through the front-end display page so as to inquire the abnormal commodity service data counted in the search service platform through the abnormal data statistical interface and display the abnormal commodity service data counted in the search service platform on the front-end display page.
In one embodiment, the method further comprises:
when the first commodity service data is different from the second commodity service data, generating a data log according to the first service input parameter data; writing the data log into a search service platform; and calling a data query interface through the front-end display page so as to query the data logs in the search service platform through the data query interface, and displaying the data logs in the search service platform on the front-end display page.
A data processing apparatus for commodity service data, the apparatus comprising:
the first acquisition module is used for acquiring first service parameter data of the target commodity service from the full data of the target commodity service; the second acquisition module is used for acquiring first commodity service data of the target commodity service from the service data of the database according to the first service access data; the third acquisition module is used for acquiring second commodity service data of the target commodity service from the cache service data according to the first service access parameter data; the updating module is used for updating second service parameter entering data in a preheating table of target commodity service according to the first service parameter entering data when the first commodity service data is different from the second commodity service data, the preheating table is used for storing corresponding service parameter entering data when the service data of the database is synchronized to the cache service data, and the second service parameter entering data is service parameter entering data corresponding to the second commodity service data in the preheating table; and the synchronization module is used for synchronizing the service data of the database corresponding to the target commodity service to the cache service data according to the updated second service access data in the preheating table.
A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the steps of the method of any of the above embodiments being performed by the processor when the computer program is executed.
A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method of any of the above embodiments.
According to the data processing method, the data processing device, the computer equipment and the storage medium of the commodity service data, first service parameter entering data of the target commodity service are obtained from the full data of the target commodity service, first commodity service data are obtained from the service data of the database according to the first service parameter entering data, and second commodity service data of the target commodity service are obtained from the cache service data. And when the first commodity service data is different from the second commodity service data, updating second service entry parameter data in the preheating table of the target commodity service according to the first service entry parameter data. Further, the service data of the database is synchronized to the cache service data according to the updated second service entry in the preheating table. Therefore, according to the data processing method of the commodity service data, the cache service data does not need to be updated manually, the comparison condition of the commodity service data in the database corresponding to the target commodity service and the commodity service data in the cache service data is automatically monitored through the updating condition of the service entry parameter data in the preheating table, the service data in the database is automatically synchronized to the cache service data through the service entry parameter data updated in the preheating table, the cache updating of the commodity service data corresponding to the target commodity service is completed, the cost for manually maintaining the commodity service data is reduced, the times of manual intervention are reduced, and the timeliness of data processing of the commodity service data is improved.
Drawings
FIG. 1 is a diagram illustrating an exemplary embodiment of a data processing method for merchandise service data;
FIG. 2 is a flow chart illustrating a data processing method for merchandise service data according to an embodiment;
FIG. 3 is a flow chart illustrating a data processing method for merchandise service data according to another embodiment;
FIG. 4 is a partial flow chart illustrating a data processing method for merchandise service data according to still another embodiment;
FIG. 5 is a partial flow chart illustrating a method for processing merchandise service data according to another embodiment;
FIG. 6 is a partial flow chart illustrating a method for processing merchandise service data according to another embodiment;
FIG. 7 is a block diagram of a data processing apparatus for merchandise service data according to an embodiment;
FIG. 8 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The data processing method of the commodity service data can be applied to the application environment shown in fig. 1. The database 10 stores a plurality of kinds of commodity service data, and each kind of commodity service data may be stored in the database 10 in the form of a commodity service data storage table, respectively. The cache device 30 stores commodity service data for caching in the database 10. Generally, it is necessary to screen out a part of the commodity service data from the commodity service data in the database 10, synchronously cache the part of the commodity service data in the storage cache device 30, and perform a pre-heating process on the commodity service data so as to increase the reading speed of the part of the commodity service data. The server 50 obtains first service parameter data of the target commodity service from the total data of the target commodity service in the database 10, and obtains first commodity service data of the target commodity service from the database 10 and second commodity service data of the target commodity service from the cache service data according to the first service parameter data. And when the first commodity service data is determined to be different from the second commodity service data, updating the second service entry parameter data in the preheating table of the target commodity service in the database 10 according to the first service entry parameter data. The preheating table of the target commodity service in the database 10 is used for storing the service parameter corresponding to the service data synchronized to the cache service data in the cache device 30, and the second service parameter data is the service parameter data corresponding to the second commodity service data in the preheating table. Finally, the server 50 synchronizes the service data of the database corresponding to the target commodity service to the cache service data according to the updated second service entry data in the preheating table. The server 50 may be implemented as a stand-alone server or a server cluster composed of a plurality of servers.
In one embodiment, as shown in fig. 2, a data processing method for commodity service data is provided, which is described by taking the method as an example applied to the server in fig. 1, and includes the following steps:
and S100, acquiring first service parameter data of the target commodity service from the total data of the target commodity service.
In this embodiment, the server screens out corresponding first service parameter data from the total data of the target commodity service. The total data of the target commodity service comprises commodity service data of the target commodity service and service parameter data used for reading the commodity service data. Wherein the full amount data of the target commodity service may be the full amount data stored in the database. The full amount of data may include offline data of the platform. The first serving parameter of the target goods service may be one or more. Specifically, offline full data of the target commodity service is extracted to a temporary table of the big data platform by using task computing capacity of the big data platform, and all dimension commodity service data of the target commodity service are computed in full to obtain first service access data.
S300, acquiring first commodity service data of the target commodity service from the service data of the database according to the first service access data.
And when the commodity service management platform receives the commodity service data, storing the commodity service data into the database. And when receiving a service acquisition request containing the service entry data, the server calls corresponding commodity service data from the database according to the service entry data. In this embodiment, the server acquires first commodity service data of the target commodity service from the service data of the database. Specifically, the server stores the service-in parameter data of the target commodity service and the corresponding relation of the commodity service data, and the corresponding relation of the two is stored in the server in the form of a commodity service information storage table. The server can obtain corresponding first commodity service data according to the first service access parameter.
And S500, acquiring second commodity service data of the target commodity service from the cache service data according to the first service parameter data.
In this embodiment, the commodity service management platform performs pre-heating processing on part of commodity service data in the database according to the service requirement, and stores the part of service data as cache service data in the cache device. And the server reads out second commodity service data of the target commodity service from the cache service data according to the first service parameter data. The second commodity service data is commodity service data which synchronizes the first commodity service data from the database to the cache device for caching. However, the commodity service data in the database is updated according to the received new commodity service data, and the updated second commodity service data is different from the first commodity service data.
S700, when the first commodity service data is different from the second commodity service data, updating second service parameter entering data in a preheating table of the target commodity service according to the first service parameter entering data, wherein the preheating table is used for storing corresponding service parameter entering data when the service data of the database is synchronized to the cache service data, and the second service parameter entering data is service parameter entering data corresponding to the second commodity service data in the preheating table.
In the embodiment, the server sets a corresponding preheating table according to the characteristics of the target commodity service. And the preheating table stores service entry parameter data corresponding to service entry parameter data when the service data of the database is synchronized to the cache service data. That is, the server performs the preheating operation on the commodity service data of the target commodity service according to the service entry data in the preheating table. And when the server detects that the first commodity service data is different from the second commodity service data, updating second service entry parameter data in the preheating table of the target commodity service according to the first service entry parameter data. Specifically, a message format no-ordering comparison algorithm is adopted to calculate the first commodity service data and the second commodity service data. When the two are inconsistent, the preheating table is written back, and the first service entry parameter data is used for replacing the second service entry parameter data in the preheating table, so that the labor maintenance cost caused by the inconsistency of the commodity service data is effectively reduced, the precision of the production environment data is improved, and the complaint times of merchants and users caused by the fact that the commodity service data is not updated in time are reduced.
And S900, synchronizing the service data of the database corresponding to the target commodity service to the cache service data according to the updated second service access data in the preheating table.
In this embodiment, when the server performs data preheating on the commodity service data of the target commodity service, the updated second service entry parameter data is read from the preheating table. And reading corresponding commodity service data from the service data of the database through the second service parameter data, and synchronizing the corresponding commodity service data into the cache service data. Therefore, the commodity service data in the database corresponding to the target commodity service can be automatically detected to be updated without manual monitoring, and the commodity service data in the database is synchronized into the cache service data when the data is updated.
In one embodiment, S900 includes: and reading the updated second service entry data in the preheating table according to a preset period, and synchronizing the service data of the database to the cache service data according to the read second service entry data.
According to the data processing method of the commodity service data, first service parameter entering data of the target commodity service are obtained from the full data of the target commodity service, first commodity service data are obtained from the service data of the database according to the first service parameter entering data, and second commodity service data of the target commodity service are obtained from the cache service data. And when the first commodity service data is different from the second commodity service data, updating second service entry parameter data in the preheating table of the target commodity service according to the first service entry parameter data. Further, the service data of the database is synchronized to the cache service data according to the updated second service entry in the preheating table. Therefore, according to the data processing method of the commodity service data, the cache service data does not need to be updated manually, the comparison condition of the commodity service data in the database corresponding to the target commodity service and the commodity service data in the cache service data is automatically monitored through the updating condition of the service entry parameter data in the preheating table, the service data in the database is automatically synchronized to the cache service data through the service entry parameter data updated in the preheating table, the cache updating of the commodity service data corresponding to the target commodity service is completed, the cost for manually maintaining the commodity service data is reduced, the times of manual intervention are reduced, and the timeliness of data processing of the commodity service data is improved.
In one embodiment, the first service participation data in the total data of the target commodity service is a plurality of. As shown in fig. 3, S300 includes the steps of:
s301: and acquiring a plurality of first commodity service data from the service data of the database according to the first service access data respectively.
S500 includes the steps of:
s501: and acquiring a plurality of second commodity service data of the target commodity service from the cache service data according to the first service access parameters.
S700 includes the steps of:
s701: and when the first commodity service data corresponding to any one of the first service access parameters is different from the second commodity service data, updating the corresponding second service access parameter data in the preheating table of the target commodity service according to any one of the first service access parameters.
In this embodiment, the server obtains a plurality of first service parameter data of the target commodity service from the full data of the target commodity service, and obtains a plurality of first commodity service data from the service data of the database and a plurality of second commodity service data of the target commodity service from the cache service data according to each first service parameter. Further, the first commodity service data and the second commodity service data corresponding to each other are compared with each other. And when any comparison result shows that the first commodity service data and the second commodity service data are different, updating the corresponding second service entry parameter data in the preheating table by using the first service entry parameter data corresponding to the result. In this case, the number of the second service entry data is also plural. And finally, synchronizing the service data of the database to the cache service data according to all the updated second service entries in the preheating table. Therefore, the updating conditions of all commodity service data corresponding to the target commodity service can be automatically monitored, and the service data of the database can be automatically synchronized to the cache service data according to the updating result.
In a specific embodiment, a large data platform temporary table is designed, and the full amount of data of the basic data of the database is extracted to the temporary table through a database middleware, such as a Mycat middleware. Different table structures of the temporary table are designed according to dimensions of different commodity services and associated data thereof, and different from different data types of the temporary table and a database table, specific field types in the temporary table can be mapped. In an implementation process, the target commodity service is a commodity basic information service, and the first service reference data is a commodity code. Extracting a commodity code from a commodity basic information table of the commodity basic information service, wherein the commodity code is used as a key field for service query and cache query, and a temporary table establishing statement is designed as follows:
CREATE TABLE `hive_product_base_info`(
'part number' string CommENT 'commodity code',
"reproducing Commment' merchant code",
COMMENT 'Commodity basic information service key field table',
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'）；
the temporary table can be created, required field data in the Mycat data source commodity basic information table is extracted through the big data platform, then on the big data platform, the data integrity in the hive _ product _ base _ info table can be inquired through a data exploration tool, and the correctness of the service data extraction process is verified.
After the steps are completed, the computing capability of a big data platform is utilized, task computing temporary table data is developed, a message format no-ordering comparison algorithm is utilized, commodity service data in a database is compared with cache service data, service entry parameter data such as commodity codes are written into a new temporary table when the data are inconsistent, after the computation is completed, the service entry parameter data in the new temporary table are written back to a service entry parameter in a preheating table through data exchange, monitoring logic is designed in the process, a data log is generated and written into a real-time data stream, when the method is specifically implemented, a Spark packet is newly built in a project, a big data task computing class of specific service is newly built, for example, a commodity basic information service is created, Spark product progress and live view type, a Spark main function is created, Spark Conf, javas Spark Context and Hivee context are sequentially to achieve reading loading initialization of configuration, a Spark context instance, a Spark cache SQ L is supported to query, a Hivek table is extracted, a Spark Context, a Spark context is created, a Spark context is written into a Spark context, a Spark context is used to achieve reading loading initialization of a comparison result, whether a comparison result of a running context is created, a data log, whether the data log is written into a Spark context is completely verified, a data base data comparison result of a Spark context is obtained, a data exchange database, a Spark context is obtained, a data comparison result of a Spark context is obtained, a data, a Spark context is written into a Spark context, a Spark context is obtained, a Spark context is written into a Spark context database, a Spark context is verified, a data base, a data comparison result, a data base, a data in a data base, a data exchange result of a Spark context is verified, a data exchange result, a data base is verified, a data base is written into a data, a data comparison.
In an embodiment, as shown in fig. 4, after S900, the method further includes the steps of:
s1000: and acquiring incremental data of the target commodity service.
S1200: and acquiring third service parameter data of the target commodity service from the incremental data.
S1400: and acquiring third commodity service data of the target commodity service from the service data of the database according to the third service parameter, and acquiring fourth commodity service data of the target commodity service from the cache service data according to the third service parameter.
S1600: and when the third commodity service data is different from the fourth commodity service data, updating fourth service participation data in a preheating table of the target commodity service according to the third service participation parameter data, wherein the fourth service participation is service participation corresponding to the fourth commodity service data in the preheating table.
S1800: and synchronizing the service data of the database to the cache service data according to the updated fourth service entry in the preheating table.
In this embodiment, after the server executes the above S100 to S900 according to the total data of the target commodity service, when the subsequent server monitors the commodity service data of the target commodity service, it only needs to monitor the incremental data of the target commodity service. Specifically, third service parameter data is obtained from the incremental data, third commodity service data of the target commodity service is obtained from the service data of the database according to the third service parameter data, and fourth commodity service data of the target commodity service is obtained from the cached service data. Wherein the third commodity service data may be commodity service data in the incremental data. Further, when the third commodity service data is different from the fourth commodity service data, the fourth service entry parameter data in the preheating table of the target commodity service is updated according to the third service entry parameter data. And finally, synchronizing the service data of the database to the cache service data according to the updated fourth service entry in the preheating table. Therefore, the method and the device can monitor the commodity service data updating condition in the incremental data of the subsequent target commodity service in the platform and automatically update the corresponding cache service data.
In one implementation of this embodiment, S1000 includes: and creating a sliding time window according to the updating time of the commodity service data of the target commodity service, and acquiring incremental data from the commodity data of the stored target commodity service through the sliding time window.
In this embodiment, the incremental data of the goods service data of the target goods service is acquired through the sliding time window. Specifically, a sliding time window is created, which is dynamically configurable. And determining the value range of the sliding time window according to the commodity service data tabulation updating time of the target commodity service. The value range of the sliding time window is expressed as: start time ± N × window time, end time ± N × window time. The specific way of acquiring the incremental data from the commodity service storage table storing the commodity service data of the target commodity service through the sliding time window is as follows: and acquiring commodity service data of the target commodity service within the range of [ starting time +/-N window time and ending time +/-N window time ] of the update time, wherein the acquired commodity data is incremental data.
In one implementation of this embodiment, the preheating logic for the incremental data of the target commodity service is: the inconsistent service entry parameter data is stored in a preheating table of the target commodity service, the service entry parameter data in the preheating table is triggered to be calculated in real time through a timing task, the service entry parameter data in the database is acquired in real time during calculation to update the current cached commodity service data, the updating time in the preheating table is added to serve as the version number of the cached data, the inconsistency of the commodity service data is acquired through the comparison result of the version number during the next data comparison, the execution time of the comparison algorithm is reduced, and the comparison efficiency is improved.
Therefore, the commodity service data of the target commodity service is obtained through the sliding time window, the consistency of the cache and the commodity service data in the database is compared, and when the consistency is not consistent, the service entry parameter data is extracted, written back and preheated again, so that the operation and maintenance monitoring of the commodity service data can be realized. Under the condition that the commodity service data are inconsistent due to updating delay, physical downtime, asynchronism of a database master and a database slave and the like, the technical means is used for monitoring the process, operating and maintaining the commodity service data and providing accurate service of the commodity service data.
For this embodiment, a specific implementation is provided as follows:
the method comprises the following steps of developing a sliding time window time slice secondary preheating task abstract class of the service, wherein the secondary preheating task abstract class realizes data source definition, current service definition, service data query according to time slices, generation of a cache Key, cache query, database table service data query, a message format non-sequencing comparison algorithm, a process state monitoring method and other general methods. In specific implementation, a rewrmer package is newly built, a base package is newly built below the rewrmer package, and an abstract rewrmer abstract class is newly built, wherein the automatic assembly instantiation of the Mycat data source is completed, and the following abstract method is defined:
getService, which is used to obtain the service of the secondary preheating;
the queryChangeData is used for inquiring the change data in the specific service table;
the general cacheKey is used for calling the cache cacheKey template generator to generate a cache Key;
generalDBValue, which is used for calling the Mycat data source to inquire the basic information of the specific service;
the composedata is used for comparing algorithms of message formats without ordering consistency;
insert2Warmer, is used for writing parameter data into the preheating table in batches;
getInitParams, which is used for acquiring an initial parameter list including a time range startTime and endTime;
the historycachekey generator is used for generating a history information record cache key;
a runhistorytimecachekey generator for generating a historical runtime range cache key;
the stateCacheKeyGenerator is used for generating a service secondary preheating job state cache key;
the totalcachey key generator is used for generating a daily total preheating data amount cache key of the secondary preheating job history of the service;
getRning State, used for obtaining a job running state;
an example method is defined:
run, used for execution of job, obtaining state lock, locking and releasing lock, abnormal capture;
execute is used for executing the job as a whole and processing the service.
After the abstract class definition is completed, a new implementation class is required. For example, the target commodity service is a commodity basic information service: RewarerProduceBaseInfo is newly built under a rewrmer package to realize abstract methods in abstract classes and realize basic definition of the classes. The realized business logic inquires the commodity service data in the current jobrunning appointed time range (such as the latest 5 minutes) from the commodity basic information table, and inquires the commodity service data from the cache according to the CacheKey for comparison. And if the commodity service data are inconsistent, writing the service data into the preheating table, updating the current service state of the cache, and pushing the execution time range of the Job for 5 minutes in the next stage.
After the task definition is completed, a Spring Quartz timer setting tool is configured, the jobs configured in the steps are started, the time slice secondary preheating function of each service is completed, and the strategy of service self-operation and maintenance is realized. In specific implementation, a CronExpression timer of Quartz can be configured, starting at 0 minute and 0 second of each hour, and triggering once every 5 minutes. After the project is started, checking the operation condition of the jobs, verifying whether the data can be written into the preheating table after calculation or not in the corresponding time range and under the condition that the commodity service data of the cache and the database are inconsistent, performing secondary preheating of service, verifying whether the jobs can be pushed for 5 minutes later when triggering for the second time, comparing the service data within 5 minutes after inquiry, and writing the inconsistent service data into the preheating table after comparison.
After the task is completed, a preheating task abstract class is developed, and universal methods such as reading of a preheating table, updating and writing of a cache, query of database service data, a business logic calculation method of commodity service data and the like are realized. In specific implementation, a warmer package is newly built, a base package is newly built under the warmer package, and an abstract Warmer abstract class is newly built. And defining the abstract method, after the abstract class is finished, realizing the preheating function of each concrete service according to different service inheritance abstract Warmer abstract classes under a warmer package, and configuring a timing task to finish the preheating function of service data by the same steps after finishing the preheating function.
In an embodiment, as shown in fig. 5, before S100, the method further includes:
s101: and writing the cache service data into the search service platform.
S103: and developing a data query interface and an abnormal data statistic interface of the search service platform.
S105: and calling a data query interface through the front-end display page so as to query the commodity service data in the search service platform through the data query interface, and displaying the commodity service data in the search service platform on the front-end display page.
S107: and calling an abnormal data statistical interface through the front-end display page so as to inquire the abnormal commodity service data counted in the search service platform through the abnormal data statistical interface, and displaying the abnormal commodity service data counted in the search service platform on the front-end display page.
In this embodiment, the server writes the cache service data as commodity basic data of the target commodity service into the search service platform. In addition, the server additionally monitors the receiving and warehousing of the commodity basic data of the target commodity service, and the received commodity basic data generates a log and writes the log into a real-time data stream. And further, verifying whether the real-time data stream is consumed by the middleware and written into the search service platform, and finishing verification after normal writing. Meanwhile, a data query interface and an abnormal data statistics interface are developed based on the search service platform and are used for being called by a front-end display page. And displaying the commodity service data in the search service platform and displaying the commodity abnormal service data counted in the search service platform through the front-end display page.
In a specific implementation, a system configuration switch can be added in a data receiving method class execute of an XM L message receiving and processing abstract class Abstract MessageReceiveService, a specific switch name can be set to log.switch, which is a disaster tolerance means for controlling abnormal conditions, if abnormal conditions occur or the data volume increases sharply, the switch is used for analyzing a received message and converting an object format, and further packaging the message format data for real-time data stream transmission specified by a protocol, a transmission object name is set to Monitor L ogDto include fields of stream number, service type, service coding, number of message singles, creation time, abnormal information, message content and the like, the data transmission mode is not limited to a form of a word stream, and the data can be defined as a stream management _ Monitor log, if a data configuration tool for processing a message singles, creation time, abnormal information, message content and the like, whether the data is written in a real-time stream management queue, whether a log of a real-time data queue of a real-time monitoring tool is written, and whether a log is written in a real-time data queue, and whether a log is written in a real-time data stream management data queue, and whether a log is written in a real-time data-consumption-time data-consumption-stream-monitoring platform.
After the steps are verified, a data query interface and an abnormal data statistical interface need to be developed based on a search service platform for the front-end monitoring page to call. In specific implementation, project engineering is configured, an elastic search (hereinafter referred to as es) is used as a search service platform, a point. Adding tool class package tool in engineering, creating es client side tool class ElasticisearchConfig, configuring cluster name, ip address and port number, creating es client side connection, and realizing es basic tool class. And creating a controller packet and a service packet, and realizing the paging inquiry and statistical functions of es on the monitoring service data interface. At this time, QueryBuilders provided by an es client jar packet are used, and comprise wildcardQuery fuzzy query, termQuery precision query, coolquery combination query, fliter filtering query and the like, and AggregationBuilders, which comprise sum summation, avg averaging, count counting, dateHistogram aggregation query and the like. The method comprises the following steps of realizing multi-dimensional data query and statistical functions from a monitoring angle, verifying whether a controller interface designed by a Spring rest style can normally call data by a front-end monitoring page, and designing a plurality of interfaces to meet the data requirement displayed by the front-end page, wherein the following interfaces support Http requests in a Get mode:
the queryData inquires monitoring data which are aggregated and inquired by es according to the participating service codes and the time range;
the queryGroupData queries the monitoring data of the es grouped according to the participating service codes and the time range;
the queryTodayData inquires all service data monitored and counted on the day according to the participating service codes;
queryexceptionsata inquires monitored service abnormal data according to the participating service codes, the time range, the paging conditions (large data volume and supporting paging inquiry), the message keywords.
In an embodiment, as shown in fig. 6, after S900, the method further includes the steps of:
s901: and when the first commodity service data is different from the second commodity service data, generating a data log according to the first service input parameter data.
S903: and writing the data log into the search service platform.
S905: and calling a data query interface through the front-end display page so as to query the data logs in the search service platform through the data query interface, and displaying the data logs in the search service platform on the front-end display page.
In particular, the data log refers to the data log used for statistical analysis, and the dimension is also specific to each commodity service dimension. And monitoring that the first commodity service data is the same as the second commodity service data, and normally writing the data into the database. However, when data statistical analysis is performed, a performance bottleneck (for example, query is too slow, hundreds of millions of service data statistical analysis cannot be performed), so that data is recorded in a log form and written into a real-time data stream, and a search service platform provides retrieval and statistical analysis for result data, so that a front-end monitoring page can more friendly display a monitoring result. Therefore, the monitoring of the commodity service data of the target commodity service is more precise, and the timeliness of data processing of the commodity service data of the target commodity service is improved.
It should be understood that, although the steps in the flowchart are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the sub-steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
In one embodiment, as shown in fig. 7, a data processing apparatus for merchandise service data is provided, which includes a first obtaining module 11, a second obtaining module 13, a third obtaining module 15, an updating module 17 and a synchronization module 19.
The first obtaining module 11 is configured to obtain first service parameter data of the target commodity service from the total data of the target commodity service.
The second obtaining module 13 is configured to obtain first commodity service data of the target commodity service from the service data of the database according to the first service entry data.
And a third obtaining module 15, configured to obtain, according to the first service entry parameter data, second commodity service data of the target commodity service from the cache service data.
The updating module 17 is configured to update second service parameter entry data in a preheating table of the target commodity service according to the first service parameter entry data when the first commodity service data is different from the second commodity service data, where the preheating table is used to store service parameter entry data corresponding to service data of the database when the service data of the database is synchronized to the cache service data, and the second service parameter entry data is service parameter entry data corresponding to the second commodity service data in the preheating table.
And the synchronization module 19 is configured to synchronize the service data of the database corresponding to the target commodity service to the cache service data according to the updated second service entry data in the preheating table.
In one embodiment, the first service access data is multiple, and the second obtaining module 13 may include (not shown in fig. 7):
and the second acquisition unit is used for acquiring a plurality of first commodity service data from the service data of the database according to the first service access data.
A third acquisition module 15, comprising:
and the third acquisition unit is used for acquiring a plurality of second commodity service data of the target commodity service from the cache service data according to the first service access parameters.
An update module 17, comprising:
and the updating unit is used for updating the corresponding second service entry parameter data in the preheating table of the target commodity service according to any first service entry parameter when the first commodity service data corresponding to any first service entry parameter in the plurality of first service entry parameters is different from the second commodity service data.
In one embodiment, the synchronization module 19 comprises (not shown in fig. 7):
and the reading unit is used for reading the second service access parameter data updated in the preheating table according to a preset period.
And the updating unit is used for synchronizing the service data of the database to the cache service data according to the read second service access data.
In one embodiment, a data processing apparatus for merchandise service data may include (not shown in fig. 7):
and the fourth acquisition module is used for acquiring the incremental data of the target commodity service.
And the fifth acquisition module is used for acquiring third service parameter data of the target commodity service from the incremental data.
And the sixth acquisition module is used for acquiring third commodity service data of the target commodity service from the service data of the database according to the third service entry parameter and acquiring fourth commodity service data of the target commodity service from the cache service data according to the third service entry parameter.
And the replacing module is used for updating fourth service parameter entering data in the preheating table of the target commodity service according to the third service parameter entering data when the third commodity service data is different from the fourth commodity service data, wherein the fourth service parameter entering is a service parameter entering corresponding to the fourth commodity service data in the preheating table.
And the preheating module is used for synchronizing the service data of the database to the cache service data according to the updated fourth service entry in the preheating table.
In one embodiment, the fourth obtaining module may include (not shown in fig. 7):
a creating unit configured to create a sliding time window according to update time of commodity service data of a target commodity service;
and the fourth acquisition unit is used for acquiring the incremental data from the commodity data of the storage target commodity service through the sliding time window.
In one embodiment, a data processing apparatus for merchandise service data may include (not shown in fig. 7):
and the cache module is used for writing the cache service data into the search service platform.
And the development module is used for developing a data query interface and an abnormal data statistical interface of the search service platform.
The first display module is used for calling the data query interface through the front-end display page so as to query the commodity service data in the search service platform through the data query interface and displaying the commodity service data in the search service platform on the front-end display page.
And the second display module is used for calling the abnormal data statistical interface through the front-end display page so as to inquire the abnormal commodity service data counted in the search service platform through the abnormal data statistical interface and display the abnormal commodity service data counted in the search service platform on the front-end display page.
In one embodiment, a data processing apparatus for merchandise service data may include (not shown in fig. 7):
and the generating module is used for generating a data log according to the first service input parameter data when the first commodity service data is different from the second commodity service data.
And the writing module is used for writing the data log into the search service platform.
And the third display module is used for calling the data query interface through the front-end display page so as to query the data logs in the search service platform through the data query interface and displaying the data logs in the search service platform on the front-end display page.
For the specific limitation of the data processing device for commodity service data, reference may be made to the above limitation on the data processing method for commodity service data, and details are not described herein again. The modules in the data processing device for commodity service data can be wholly or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 8. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing relevant data of target commodity service, such as service access data and the like. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data processing method of commodity service data.
Those skilled in the art will appreciate that the architecture shown in fig. 8 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program:
acquiring first service parameter data of the target commodity service from the full data of the target commodity service; acquiring first commodity service data of a target commodity service from service data of a database according to the first service access data; acquiring second commodity service data of the target commodity service from the cache service data according to the first service entry parameter data; when the first commodity service data is different from the second commodity service data, updating second service parameter entering data in a preheating table of the target commodity service according to the first service parameter entering data, wherein the preheating table is used for storing corresponding service parameter entering data when the service data of the database is synchronized to the cache service data, and the second service parameter entering data is service parameter entering data corresponding to the second commodity service data in the preheating table; and synchronizing the service data of the database corresponding to the target commodity service to the cache service data according to the updated second service entry parameter data in the preheating table.
In one embodiment, when the processor executes the computer program to implement the step of obtaining the first commodity service data of the target commodity service from the service data of the database according to the first service parameter data, the following steps are specifically implemented: acquiring a plurality of first commodity service data from the service data of the database according to each first service parameter data; when the processor executes the computer program to realize the step of acquiring the second commodity service data of the target commodity service from the cache service data according to the first service entry parameter, the following steps are specifically realized: respectively acquiring a plurality of second commodity service data of the target commodity service from the cache service data according to each first service entry parameter; when the processor executes the computer program to realize the step of updating the second service entry parameter data in the preheating table of the target commodity service according to the first service entry parameter data when the first commodity service data is different from the second commodity service data, the following steps are specifically realized: and when the first commodity service data corresponding to any one of the first service access parameters is different from the second commodity service data, updating the corresponding second service access parameter data in the preheating table of the target commodity service according to any one of the first service access parameters.
In an embodiment, when the processor executes the computer program to implement the step of synchronizing the service data of the database corresponding to the target commodity service to the cache service data according to the updated second service entry data in the pre-heating table, the following steps are specifically implemented: reading second service access parameter data updated in the preheating table according to a preset period; and synchronizing the service data of the database to the cache service data according to the read second service access data.
In one embodiment, the processor executes the computer program to perform the steps of: obtaining incremental data of target commodity service; acquiring third service parameter data of the target commodity service from the incremental data; acquiring third commodity service data of the target commodity service from the service data of the database according to the third service parameter, and acquiring fourth commodity service data of the target commodity service from the cache service data according to the third service parameter; when the third commodity service data is different from the fourth commodity service data, updating fourth service participation data in a preheating table of the target commodity service according to the third service participation parameter data, wherein the fourth service participation is service participation corresponding to the fourth commodity service data in the preheating table; and synchronizing the service data of the database to the cache service data according to the updated fourth service entry in the preheating table.
In one embodiment, when the processor executes the computer program to implement the step of obtaining the incremental data of the target commodity service, the following steps are specifically implemented: creating a sliding time window according to the updating time of the commodity service data of the target commodity service; and acquiring incremental data from the commodity data of the storage target commodity service through a sliding time window.
In one embodiment, the processor executes the computer program to perform the steps of: writing the cache service data into a search service platform; developing a data query interface and an abnormal data statistical interface of a search service platform; calling a data query interface through a front-end display page to query the commodity service data in the search service platform through the data query interface, and displaying the commodity service data in the search service platform on the front-end display page; and calling an abnormal data statistical interface through the front-end display page so as to inquire the abnormal commodity service data counted in the search service platform through the abnormal data statistical interface and display the abnormal commodity service data counted in the search service platform on the front-end display page.
In one embodiment, the processor executes the computer program to perform the steps of: when the first commodity service data is different from the second commodity service data, generating a data log according to the first service input parameter data; writing a data log into the search service platform; and calling a data query interface through the front-end display page so as to query the data logs in the search service platform through the data query interface, and displaying the data logs in the search service platform on the front-end display page.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when being executed by a processor, implements a data processing method of goods service data according to any one of the above embodiments.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:
acquiring first service parameter data of the target commodity service from the full data of the target commodity service; acquiring first commodity service data of a target commodity service from service data of a database according to the first service access data; acquiring second commodity service data of the target commodity service from the cache service data according to the first service entry parameter data; when the first commodity service data is different from the second commodity service data, updating second service parameter entering data in a preheating table of the target commodity service according to the first service parameter entering data, wherein the preheating table is used for storing corresponding service parameter entering data when the service data of the database is synchronized to the cache service data, and the second service parameter entering data is service parameter entering data corresponding to the second commodity service data in the preheating table; and synchronizing the service data of the database corresponding to the target commodity service to the cache service data according to the updated second service entry parameter data in the preheating table.
In one embodiment, the first service parameter data is a plurality of data, and the computer program when executed by the processor performs the step of obtaining the first commodity service data of the target commodity service from the service data of the database according to the first service parameter data, specifically implements the following steps: acquiring a plurality of first commodity service data from the service data of the database according to each first service parameter data; when the computer program is executed by the processor to realize the step of acquiring the second commodity service data of the target commodity service from the cache service data according to the first service entry parameter, the following steps are specifically realized: respectively acquiring a plurality of second commodity service data of the target commodity service from the cache service data according to each first service entry parameter; when the computer program is executed by the processor to realize the step of updating the second service entry parameter data in the preheating table of the target commodity service according to the first service entry parameter data when the first commodity service data is different from the second commodity service data, the following steps are specifically realized: and when the first commodity service data corresponding to any one of the first service access parameters is different from the second commodity service data, updating the corresponding second service access parameter data in the preheating table of the target commodity service according to any one of the first service access parameters.
In an embodiment, when the computer program is executed by the processor to implement the step of synchronizing the service data of the database corresponding to the target commodity service to the cache service data according to the updated second service entry data in the pre-heating table, the following steps are specifically implemented: reading second service access parameter data updated in the preheating table according to a preset period; and synchronizing the service data of the database to the cache service data according to the read second service access data.
In one embodiment, the computer program is executed by a processor to implement the steps of: obtaining incremental data of target commodity service; acquiring third service parameter data of the target commodity service from the incremental data; acquiring third commodity service data of the target commodity service from the service data of the database according to the third service parameter, and acquiring fourth commodity service data of the target commodity service from the cache service data according to the third service parameter; when the third commodity service data is different from the fourth commodity service data, updating fourth service participation data in a preheating table of the target commodity service according to the third service participation parameter data, wherein the fourth service participation is service participation corresponding to the fourth commodity service data in the preheating table; and synchronizing the service data of the database to the cache service data according to the updated fourth service entry in the preheating table.
In one embodiment, when the processor executes the step of obtaining the incremental data of the target commodity service, the following steps are specifically implemented: creating a sliding time window according to the updating time of the commodity service data of the target commodity service; and acquiring incremental data from the commodity data of the storage target commodity service through a sliding time window.
In one embodiment, the computer program is executed by a processor to implement the steps of: writing the cache service data into a search service platform; developing a data query interface and an abnormal data statistical interface of a search service platform; calling a data query interface through a front-end display page to query the commodity service data in the search service platform through the data query interface, and displaying the commodity service data in the search service platform on the front-end display page; and calling an abnormal data statistical interface through the front-end display page so as to inquire the abnormal commodity service data counted in the search service platform through the abnormal data statistical interface and display the abnormal commodity service data counted in the search service platform on the front-end display page.
In one embodiment, the computer program is executed by a processor to implement the steps of: when the first commodity service data is different from the second commodity service data, generating a data log according to the first service input parameter data; writing a data log into the search service platform; and calling a data query interface through the front-end display page so as to query the data logs in the search service platform through the data query interface, and displaying the data logs in the search service platform on the front-end display page.
It will be understood by those of ordinary skill in the art that all or a portion of the processes of the methods of the embodiments described above may be implemented by a computer program that may be stored on a non-volatile computer-readable storage medium, which when executed, may include the processes of the embodiments of the methods described above, wherein any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims (10)

1. A data processing method of goods service data, the method comprising:
acquiring first service parameter data of a target commodity service from the full data of the target commodity service;
acquiring first commodity service data of the target commodity service from service data of a database according to the first service access data;
acquiring second commodity service data of the target commodity service from cache service data according to the first service input parameter;
when the first commodity service data is different from the second commodity service data, updating second service parameter entering data in a preheating table of the target commodity service according to the first service parameter entering data, wherein the preheating table is used for storing corresponding service parameter entering data when the service data of the database is synchronized to the cache service data, and the second service parameter entering data is corresponding to the second commodity service data in the preheating table;
and synchronizing the service data of the database corresponding to the target commodity service to the cache service data according to the updated second service access data in the preheating table.
2. The method according to claim 1, wherein the first service reference data is a plurality of data, and the obtaining the first commodity service data of the target commodity service from the service data of the database according to the first service reference data comprises:
acquiring a plurality of first commodity service data from the service data of the database according to the first service access data respectively;
the obtaining second commodity service data of the target commodity service from cache service data according to the first service entry parameter includes:
respectively acquiring a plurality of second commodity service data of the target commodity service from cache service data according to the first service input parameters;
when the first commodity service data is different from the second commodity service data, updating second service entry parameter data in a preheating table of the target commodity service according to the first service entry parameter data, wherein the updating comprises the following steps:
when any first commodity service data corresponding to any first service access parameter in the plurality of first service access parameters is different from the second commodity service data, updating corresponding second service access parameter data in a preheating table of the target commodity service according to any first service access parameter.
3. The method of claim 1, wherein the synchronizing the service data of the database corresponding to the target commodity service to the cache service data according to the updated second service entry data in the warming table comprises:
reading second service access parameter data updated in the preheating table according to a preset period;
and synchronizing the service data of the database to the cache service data according to the read second service access data.
4. The method of claim 1, further comprising:
obtaining incremental data of the target commodity service;
acquiring third service parameter data of the target commodity service from the incremental data;
acquiring third commodity service data of the target commodity service from the service data of the database according to the third service parameter data, and acquiring fourth commodity service data of the target commodity service from the cache service data according to the third service parameter data;
when the third commodity service data is different from the fourth commodity service data, updating fourth service parameter entering data in a preheating table of the target commodity service according to the third service parameter entering data, wherein the fourth service parameter entering data is service parameter entering data corresponding to the fourth commodity service data in the preheating table;
and synchronizing the service data of the database to the cache service data according to the updated fourth service entry data in the preheating table.
5. The method of claim 4, wherein the obtaining incremental data for the target commodity service comprises:
creating a sliding time window according to the updating time of the commodity service data of the target commodity service;
and acquiring the incremental data from the commodity data storing the target commodity service through a sliding time window.
6. The method of any one of claims 1 to 4, further comprising:
writing the cache service data into a search service platform;
developing a data query interface and an abnormal data statistical interface of the search service platform;
calling the data query interface through a front-end display page to query the commodity service data in the search service platform through the data query interface, and displaying the commodity service data in the search service platform on the front-end display page;
and calling the abnormal data statistics interface through a front-end display page so as to inquire the abnormal commodity service data counted in the search service platform through the abnormal data statistics interface, and displaying the abnormal commodity service data counted in the search service platform on the front-end display page.
7. The method of claim 6, further comprising:
when the first commodity service data is different from the second commodity service data, generating a data log according to the first service input parameter data;
writing the data log into the search service platform;
and calling the data query interface through a front-end display page so as to query the data logs in the search service platform through the data query interface, and displaying the data logs in the search service platform on the front-end display page.
8. A data processing apparatus for commodity service data, the apparatus comprising:
the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring first service parameter data of a target commodity service from full data of the target commodity service;
the second acquisition module is used for acquiring first commodity service data of the target commodity service from service data of a database according to the first service access data;
the third acquisition module is used for acquiring second commodity service data of the target commodity service from cache service data according to the first service access parameter;
an updating module, configured to update second service parameter entry data in a preheating table of the target commodity service according to the first service parameter entry data when the first commodity service data is different from the second commodity service data, where the preheating table is used to store service parameter entry data corresponding to service data of the database when the service data of the database is synchronized to the cache service data, and the second service parameter entry data is service parameter entry data corresponding to the second commodity service data in the preheating table;
and the synchronization module is used for synchronizing the service data of the database corresponding to the target commodity service to the cache service data according to the updated second service access data in the preheating table.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 7 are implemented when the computer program is executed by the processor.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
CN202010287873.0A 2020-04-14 2020-04-14 Data processing method, data processing device, computer equipment and storage medium Active CN111222089B (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
CN202010287873.0A CN111222089B (en) 2020-04-14 2020-04-14 Data processing method, data processing device, computer equipment and storage medium
PCT/CN2020/096997 WO2021208228A1 (en) 2020-04-14 2020-06-19 Data processing method and apparatus, and computer device and storage medium
CA3177218A CA3177218A1 (en) 2020-04-14 2020-06-19 Data processing method, device, computer equipment, and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010287873.0A CN111222089B (en) 2020-04-14 2020-04-14 Data processing method, data processing device, computer equipment and storage medium

Publications (2)

Publication Number Publication Date
CN111222089A CN111222089A (en) 2020-06-02
CN111222089B true CN111222089B (en) 2020-07-31

Family

ID=70830097

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010287873.0A Active CN111222089B (en) 2020-04-14 2020-04-14 Data processing method, data processing device, computer equipment and storage medium

Country Status (3)

Country Link
CN (1) CN111222089B (en)
CA (1) CA3177218A1 (en)
WO (1) WO2021208228A1 (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111222089B (en) * 2020-04-14 2020-07-31 苏宁云计算有限公司 Data processing method, data processing device, computer equipment and storage medium
CN112783901B (en) * 2021-03-01 2023-09-01 合沃物联技术（南京）有限公司 Internet of things time sequence big data processing method based on Internet of things middleware
CN115080662A (en) * 2022-06-21 2022-09-20 浙江网商银行股份有限公司 Database switching method and device
CN115460227A (en) * 2022-11-14 2022-12-09 成都怡康科技有限公司 Method, device and system for synchronizing data, computer equipment and storage medium

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101464895A (en) * 2009-01-21 2009-06-24 阿里巴巴集团控股有限公司 Method, system and apparatus for updating internal memory data
CN102331986B (en) * 2010-07-12 2014-07-16 阿里巴巴集团控股有限公司 Database cache management method and database server
CN102244900B (en) * 2011-07-15 2014-01-22 上海华为技术有限公司 Method, device and system for synchronizing cached data heat value, and distributed caching method
US9361334B2 (en) * 2013-08-23 2016-06-07 Cisco Technology, Inc. Addressing cache coherence in updates to a shared database in a network environment
CN107644071A (en) * 2017-09-18 2018-01-30 广州爱九游信息技术有限公司 Buffering updating method, device and data-storage system
CN108418851B (en) * 2018-01-12 2020-12-04 创新先进技术有限公司 Policy issuing system, method, device and equipment
CN110753099B (en) * 2019-10-12 2023-09-29 平安健康保险股份有限公司 Distributed cache system and cache data updating method
CN111222089B (en) * 2020-04-14 2020-07-31 苏宁云计算有限公司 Data processing method, data processing device, computer equipment and storage medium

Also Published As

Publication number Publication date
WO2021208228A1 (en) 2021-10-21
CA3177218A1 (en) 2021-10-21
CN111222089A (en) 2020-06-02

Similar Documents

Publication Publication Date Title
CN111222089B (en) Data processing method, data processing device, computer equipment and storage medium
CN107038162B (en) Real-time data query method and system based on database log
CN104809201B (en) A kind of method and apparatus of database synchronization
US9311664B2 (en) Systems and methods for automatically collection of performance data in a multi-tenant database system environment
CN113111129B (en) Data synchronization method, device, equipment and storage medium
CN112559475B (en) Data real-time capturing and transmitting method and system
CN107924345B (en) Data store for aggregated measurements of metrics
WO2022237506A1 (en) Method, apparatus, and device for monitoring online diagnosis service, and storage medium
US10545988B2 (en) System and method for data synchronization using revision control
CN106844694B (en) Method and apparatus for synchrodata
US20220044144A1 (en) Real time model cascades and derived feature hierarchy
CN113468143A (en) Data migration method, system, computing device and storage medium
CN112905676A (en) Data file importing method and device
CN117131059A (en) Report data processing method, device, equipment and storage medium
US10706073B1 (en) Partitioned batch processing for a usage analysis system
US9727621B2 (en) Systems and methods for servicing database events
US10348596B1 (en) Data integrity monitoring for a usage analysis system
CN114490865A (en) Database synchronization method, device, equipment and computer storage medium
CN113220530A (en) Data quality monitoring method and platform
CN113014347A (en) Server time synchronization method and device, computer equipment and storage medium
CN101894119B (en) Mass data storage system for monitoring
CN113783921A (en) Method and device for creating cache component
CN110674214A (en) Big data synchronization method and device, computer equipment and storage medium
CN117390040B (en) Service request processing method, device and storage medium based on real-time wide table
CN117349384B (en) Database synchronization method, system and equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20210506

Address after: 210042 no.1-9 Suning Avenue, Xuanwu District, Nanjing City, Jiangsu Province (Jiangsu Province)

Patentee after: NANJING SUNING ELECTRONIC INFORMATION TECHNOLOGY Co.,Ltd.

Address before: No.1-1 Suning Avenue, Xuzhuang Software Park, Xuanwu District, Nanjing, Jiangsu Province, 210000

Patentee before: Suning Cloud Computing Co.,Ltd.