CN112765187A - Commodity data synchronization method and commodity data transmission system - Google Patents
Commodity data synchronization method and commodity data transmission system Download PDFInfo
- Publication number
- CN112765187A CN112765187A CN202011607163.8A CN202011607163A CN112765187A CN 112765187 A CN112765187 A CN 112765187A CN 202011607163 A CN202011607163 A CN 202011607163A CN 112765187 A CN112765187 A CN 112765187A
- Authority
- CN
- China
- Prior art keywords
- data
- commodity
- module
- target
- synchronization
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 49
- 230000005540 biological transmission Effects 0.000 title claims abstract description 16
- 230000008859 change Effects 0.000 claims abstract description 61
- 230000009471 action Effects 0.000 claims abstract description 29
- 230000001360 synchronised effect Effects 0.000 claims abstract description 16
- 230000004044 response Effects 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 4
- 230000003068 static effect Effects 0.000 claims description 3
- 230000003993 interaction Effects 0.000 description 10
- 230000008569 process Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 8
- 238000012545 processing Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 230000000694 effects Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 230000006399 behavior Effects 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000737 periodic effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 102100036702 Glucosamine-6-phosphate isomerase 2 Human genes 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000012508 change request Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Computer Security & Cryptography (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The disclosure provides a commodity data synchronization method and a commodity data transmission system. The method comprises the following steps: responding to the commodity data change message, and acquiring one or more types of change data and one or more target identifiers; when the type of the changed data is a first type, updating the corresponding updating times of the target identifier in the first list, and triggering a synchronization action according to the target identifier recorded in the first list when a period synchronization time point is reached; when the type of the changed data is a second type, triggering a synchronous action according to the target identifier; the synchronization action comprises calling a service module to read a cache module to obtain changed data and updating a search database according to the changed data, wherein the commodity data corresponding to the first type is data with the updating times larger than or equal to a preset value in a preset time period, and the commodity data corresponding to the second type is data with the updating times smaller than the preset value in the preset time period. The method and the device can ensure that the data are updated in time and the system pressure is reduced.
Description
Technical Field
The present disclosure relates to the field of internet technologies, and in particular, to a commodity data synchronization method and a commodity data transmission system using the same.
Background
In the internet industry, in order to improve the Search efficiency and the accuracy of Search data, an Elastic Search (ES) Search service is used to provide brief commodity information for a user, and at this time, how to timely update the commodity information in a background database to an ES facing the user in the front end becomes an important problem.
It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure, and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
The present disclosure is directed to a commodity data synchronization method and a commodity data transmission system, which are used to overcome, at least to some extent, the problems of low real-time performance or low execution efficiency of data synchronization due to the limitations and disadvantages of the related art.
According to a first aspect of the embodiments of the present disclosure, there is provided a commodity data synchronization method, including: responding to the commodity data change message, and acquiring one or more types of change data and one or more target identifiers; when the type of the changed data is a first type, updating the corresponding updating times of the target identifier in a first list, and triggering a synchronization action according to the target identifier recorded in the first list when a cycle synchronization time point is reached; when the type of the changed data is a second type, triggering the synchronous action according to the target identifier; the synchronization action comprises calling a service module to read a cache module to obtain changed data, and updating and searching a database according to the changed data, wherein the data corresponding to the first type is data with the updating times larger than or equal to a preset value in a preset time period, and the data corresponding to the second type is data with the updating times smaller than the preset value in the preset time period.
In an exemplary embodiment of the present disclosure, the triggering a synchronization action according to a target identifier recorded in the first list includes: acquiring a target identifier with the maximum current corresponding updating times in the first list; generating a commodity data synchronization request based on the target identifier with the maximum current corresponding updating times and sending the commodity data synchronization request to the service module; and acquiring the changed data returned by the service module, and deleting the target identifier with the maximum current corresponding updating times and the updating times thereof in the first list.
In an exemplary embodiment of the present disclosure, the updating the search database according to the changed data includes: and updating the first commodity data in the changed data to the search database.
In an exemplary embodiment of the disclosure, the calling the service module to read the cache module to obtain the changed data includes: sending a commodity data synchronization request containing the target identifier to the service module so that the service module reads the cache module; when the service module finds the commodity data corresponding to the target identifier in the cache module, the service module acquires the commodity data corresponding to the target identifier as the changed data; and when the service module does not find the commodity data corresponding to the target identifier in the cache module, the service module searches the commodity data corresponding to the target identifier in a database as the changed data, and correspondingly writes the changed data and the target identifier into the cache module.
In an exemplary embodiment of the disclosure, the calling the service module to read the cache module to obtain the changed data includes: sending a commodity data synchronization request containing the target identifier to the service module so that the service module reads the cache module; when the service module finds the commodity data corresponding to the target identifier in the cache module, the service module acquires the commodity data corresponding to the target identifier as the changed data; and when the service module does not find the commodity data corresponding to the target identifier in the cache module, the service module searches the commodity data corresponding to the target identifier in a database as the changed data, and correspondingly writes the changed data and the target identifier into the cache module.
According to a second aspect of the present disclosure, there is provided a commodity data synchronization device including: a change response module configured to respond to the commodity data change message and acquire one or more types of change data and one or more target identifiers; the dynamic data synchronization module is set to update the corresponding updating times of the target identifier in a first list when the type of the changed data is a first type, and trigger a synchronization action according to the target identifier recorded in the first list when a period synchronization time point is reached; the static data synchronization module is set to trigger the synchronization action according to the target identifier when the type of the changed data is a second type; and the synchronous action module is set to call the service module to read the cache module so as to obtain changed data and update and search the database according to the changed data, the commodity data corresponding to the first type is data of which the update times in a preset time period are more than or equal to a preset value, and the commodity data corresponding to the second type is data of which the update times in the preset time period are less than the preset value.
According to a third aspect of the present disclosure, there is provided a commodity data transmission system including: the search engine is used for responding to user operation to generate a commodity data search request and displaying commodity data corresponding to the commodity data search request to a user; the search database is connected with the search engine and used for storing first commodity data and responding to the commodity data search request to return the first commodity data, and the first commodity data are data of which the change times in a preset time period are lower than a preset value; the cache module is used for storing the first commodity data and second commodity data, wherein the second commodity data is data of which the change times in a preset time period are more than or equal to the preset value; the database is connected with the cache module and used for storing the first commodity data and the second commodity data and emptying the commodity data corresponding to the target identifier of the changed data in the cache module when data change occurs; the data synchronization module is connected with the search database and the database and is used for monitoring commodity data change messages of the database so as to execute the commodity data synchronization method to synchronize data of the search database; and the service module is connected with the search engine, the cache module, the database and the data synchronization module and used for responding to the commodity data search request from the search engine or the commodity data synchronization request from the data synchronization module to read the cache module or the database so as to return commodity data.
In an exemplary embodiment of the disclosure, the service module is configured to: responding to a commodity data search request from the search engine, and sending a commodity data reading request corresponding to the identifier of the target commodity to the cache module; when the commodity data corresponding to the identifier of the target commodity exists in the cache module, sending the commodity data corresponding to the identifier of the target commodity to the search engine; when the commodity data corresponding to the identifier of the target commodity does not exist in the cache module, sending a commodity data reading request corresponding to the identifier of the target commodity to the database; and after acquiring target data corresponding to the identifier of the target commodity from the database, sending the target data to the search engine, and writing the target data into the cache module.
In an exemplary embodiment of the disclosure, the service module is configured to: responding to a commodity data synchronization request from the data synchronization module, and sending a commodity data reading request corresponding to a target identifier to the cache module; when the commodity data corresponding to the target identifier exists in the cache module, sending the commodity data corresponding to the target identifier to the search engine; when the commodity data corresponding to the target identifier does not exist in the cache module, sending a commodity data reading request corresponding to the target identifier to the database; and after the target data corresponding to the target identifier is obtained from the database, the target data is sent to the search engine, and the target data is written into the cache module.
According to a fourth aspect of the present disclosure, there is provided an electronic device comprising: a memory; and a processor coupled to the memory, the processor configured to perform the method of any of the above based on instructions stored in the memory.
According to a fifth aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon a program which, when executed by a processor, implements the merchandise data synchronization method according to any one of the above.
According to the method and the device, the synchronization strategy is determined according to the commodity data types, the data change with low updating frequency and important data change is synchronized in time, the data change with high updating frequency and less important data change is synchronized in timing, the pressure of the high-frequency change of the hot data on the system operation can be avoided, and meanwhile, the timely synchronization of the important data is guaranteed in time.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure. It is to be understood that the drawings in the following description are merely exemplary of the disclosure, and that other drawings may be derived from those drawings by one of ordinary skill in the art without the exercise of inventive faculty.
Fig. 1 is a schematic diagram of a commodity data transmission system according to an embodiment of the present disclosure.
Fig. 2 is an interaction diagram of the merchandise data transmission system 100 according to an embodiment of the present disclosure.
Fig. 3 is a flowchart of a merchandise data synchronization method in an exemplary embodiment of the present disclosure.
FIG. 4 is a sub-flowchart of step S12 in one embodiment of the present disclosure.
Fig. 5 is a block diagram of a merchandise data synchronization device in an exemplary embodiment of the present disclosure.
FIG. 6 is a block diagram of an electronic device in an exemplary embodiment of the present disclosure.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the subject matter of the present disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and the like. In other instances, well-known technical solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.
Further, the drawings are merely schematic illustrations of the present disclosure, in which the same reference numerals denote the same or similar parts, and thus, a repetitive description thereof will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.
The following detailed description of exemplary embodiments of the disclosure refers to the accompanying drawings.
Fig. 1 is a schematic diagram of a commodity data transmission system according to an embodiment of the present disclosure.
Referring to fig. 1, the goods data transmission system 100 may include:
the search engine 11 faces to the user, and is used for responding to the user operation to generate a commodity data search request and displaying commodity data corresponding to the commodity data search request to the user;
a search database 12 connected to the search engine 11, configured to store first commodity data, and return the first commodity data in response to the commodity data search request, where the first commodity data is data in which the change frequency within a preset time is lower than a preset value;
the cache module 13 is configured to store the first commodity data and second commodity data, where the second commodity data is data in which the change times within a preset time are greater than or equal to the preset value;
a database 14 connected to the cache module 13, configured to store the first commodity data and the second commodity data, and clear commodity data corresponding to a target identifier of changed data in the cache module 13 when a data change occurs;
a data synchronization module 15, connected to the search database 12 and the database 14, configured to monitor a commodity data change message of the database 14, so as to execute the commodity data synchronization method 300 provided in this disclosure to synchronize data of the search database 14;
the service module 16 is connected to the search engine 11, the cache module 13, the database 14, and the data synchronization module 15, and configured to respond to the commodity data search request from the search engine 11 or the commodity data synchronization request from the data synchronization module 15 to read the cache module 13 or the database 14 to return commodity data.
The Search database 12 may be, for example, an Elastic Search (ES) for storing some data (data displayed on a Search result page) that can be easily searched by a user, that is, the first commodity data. The first commodity data may include, for example, commodity names, commodity categories, commodity profiles, commodity prices, commodity sales volumes, and the like, for the user to sift through the commodities in the search interface. The change times of the data such as commodity price and commodity sales are high within the preset time, the influence on the screening behavior of the user is limited, and the data are called as first type data in the embodiment of the disclosure; the change times of the data such as the commodity name, the commodity type, the commodity profile and the like within the preset time are low, the influence on the screening behavior of the user is large, and the data are called as second type data in the embodiment of the disclosure.
The database 14 is used to store all the commodity data, including the first commodity data and the second commodity data described above. Wherein the second commodity data is, for example, commodity detail data, picture data, comment data, and the like, which are displayed when the user clicks to refer to the commodity details. The database 14 may be implemented by a common database such as MySQL, for example, and the disclosure is not limited thereto.
The cache module 13 may be implemented by a Redis cache, for example, and is configured to store some second commodity data with high reference frequency. The cache module 13 can avoid frequent database query, and effectively improve the data feedback efficiency. The cache module 13 is provided with a data expiration clearing mechanism, which can clear the second commodity data that is not read within a preset time period, and only retain the second commodity data that has been consulted recently.
In the embodiment of the present disclosure, the database 14 and the cache module 13 both store commodity data through a plurality of data tables, each data table is used to store one kind of data, and the types and the numbers of the data tables in the database 14 and the cache module 13 correspond to each other. For example, one data table may be used to store commodity sales data corresponding to a commodity identifier, and another data table may be used to store commodity name data corresponding to a commodity identifier.
The service module 16 includes a plurality of sub-services corresponding to each data table, which may be implemented by, for example, a microservice. Each sub-service is used for reading the corresponding commodity data table in the cache module 13 or the database 14 for data query, or writing data into the corresponding commodity data table in the cache module 13.
The data synchronization module 15 may be configured with a monitoring function, and implement data synchronization by constructing an index database, creating an index map, and monitoring the commodity data change message of the database 14. In one embodiment, the function of the database 14 may be changed first, so that it triggers a commodity data change message when a data change occurs, and simultaneously sends a data clearing instruction corresponding to a target identifier of the changed data to the cache module 13. At this time, the goods data change Message of the database 14 may be monitored and transmitted to the data synchronization module 15 through a rockmq (socket Message Queue). The function of listening to the database 14 may be implemented by canal, for example. canal is an open source project of the Alibara, is realized based on java, is a very mature database synchronization scheme, and only needs simple configuration for basic use. canal monitors binlog log of MySQL to acquire data by simulating slave mode of MySQL, and after the binlog is set to row mode, the canal can acquire each executed adding and deleting script and also can acquire data before and after modification, and based on the characteristic, canal can acquire commodity data change information of MySQL with high performance.
Fig. 2 is an interaction diagram of the merchandise data transmission system 100 according to an embodiment of the present disclosure.
Referring to fig. 2, the interaction process of the commodity data transmission system 100 may be divided into two types, namely, data synchronization interaction a and search interaction B.
The process of data synchronization interaction A comprises steps SA 1-SA 9.
At step SA1, the administrator makes a change to the product data in the database 14, and the database 14 obtains one or more object identifiers of the changed data;
at step SA2, the database 14 sends goods data clearing commands corresponding to one or more object identifiers to the cache module 13;
at step SA3, the data synchronization module 15 obtains a change message including one or more target identifiers of the changed data to the database 14;
after receiving the data change request, the data synchronization module 15 executes the commodity data synchronization method 300 as described later;
at step SA4, the data synchronization module 15 sends a merchandise data synchronization request to the service module 16, the merchandise data synchronization request including one or more object identifiers of the changed data;
at step SA5, the service module 16 sends a merchandise data read request to the cache module 13 according to the one or more target identifiers of the changed data, wherein the merchandise data read request includes the one or more target identifiers;
at step SA61, if there is commodity data corresponding to the one or more object identifiers in the cache module 13 (i.e. step SA5 happens right after the subsequent step SB 44), changed data corresponding to the one or more object identifiers is returned to the service module 16;
at step SA62, if the cache module 13 does not have the commodity data corresponding to the one or more target identifiers (this is the normal case, because the previous step SA2 has cleared the commodity data corresponding to the one or more target identifiers in the cache module 13), the service module 16 sends a commodity data reading request to the database 14;
at step SA63, the service module 16 obtains the changed data from the database 14;
in step SA64, the service module 16 writes the changed data to the cache module 13;
at step SA7, whether the changed data is from the cache module 13 or the database 14, the service module 16 returns the changed data corresponding to the one or more target identifiers to the data synchronization module 15 at this step, and in one embodiment, step SA64 and step SA7 may occur simultaneously;
at step SA8, the data synchronization module 15 filters the changed data, and synchronizes only the first commodity data in the changed data to the search database 12 at step SA 9.
The process of searching for interaction A includes steps SB1 through SB 6.
At step SB1, the search engine 11 sends a first commodity data search request to the search database 12 in response to a user operation, the first commodity data search request including a search condition;
at step SB2, the search database returns the first commodity data that meets the search condition to the search engine 11 for the user to filter the commodity information;
at step SB3, after the user determines the commodity that the user wants to further understand from the first commodity data, the user controls the search engine 11 to send a second commodity data search request to the service module 16 by clicking a commodity detail viewing button or the like, where the second commodity search request carries an identifier of the target commodity;
at step SB4, the service module 16 sends, to the corresponding product data table in the cache module 13 via the plurality of sub-service modules, a product data read request for reading the second product data corresponding to the identifier of the target product, based on the identifier of the target product;
in step SB51, the cache module 13 normally has data in the positions corresponding to the identifiers of the target products in the multiple data tables of the cache module 13, and at this time, the cache module 13 directly returns the second product data to the service module 16;
at step SB52, in some cases, for example, when step SB4 occurs between step SA2 and step SA64, the cache module 13 may not have the commodity data corresponding to an identifier, and the service module 16 continues to send the commodity data reading request to the database 14;
at step SB53, the service module 16 acquires the second commodity data from the database 14;
at step SB54, the service module 16 writes the second commodity data to the cache module 13;
at step SB6, whether the second item data is from the cache module 13 or the database 14, the service module 16 returns the second item data corresponding to the identifier of the target item to the search engine 11 at this step, and in one embodiment, steps SB54 and SB6 may occur simultaneously.
The data synchronization interaction A and the search interaction B are independent of each other and occur simultaneously.
Through the interaction process shown in fig. 2, the commodity data transmission system 100 can acquire the changed data in time and feed back the latest data to the user in time.
Fig. 3 is a flowchart of a merchandise data synchronization method in an exemplary embodiment of the present disclosure.
Referring to fig. 3, the merchandise data synchronization method 300 may include:
step S11, responding to the merchandise data change message, acquiring one or more types of change data and one or more object identifiers;
step S12, when the type of the changed data is a first type, updating the number of times corresponding to the target identifier in a first list, and when the period synchronization time point is reached, triggering a synchronization action according to the target identifier recorded in the first list;
step S13, when the type of the changed data is a second type, triggering the synchronization action according to the target identifier;
the synchronization action comprises calling a service module to read a cache module to obtain changed data, and updating and searching a database according to the changed data, wherein the data corresponding to the first type is data with the updating times larger than or equal to a preset value in a preset time period, and the data corresponding to the second type is data with the updating times smaller than the preset value in the preset time period.
According to the embodiment of the disclosure, the synchronization strategy is determined according to the data type, the data change with low update frequency and important data is synchronized timely, the data change with high update frequency and less important data is synchronized regularly, the pressure of the high-frequency change of the hot data on the system operation can be avoided, and the timely synchronization of the important data is ensured timely.
Next, each step of the product data synchronization method 300 will be described in detail.
In step S11, in response to the merchandise data change message, one or more types of change data and one or more object identifiers are acquired.
In the embodiment of the present disclosure, the commodity data in which the number of times of change in the preset time period is greater than or equal to the preset value is referred to as first kind of commodity data, such as commodity price and commodity sales; the commodity data in which the number of changes within the preset time period is smaller than the preset value is referred to as second kind of commodity data, such as a commodity name, a commodity kind, a commodity profile, and the like.
The target identifier of the change data may include, for example, a product identifier and an attribute identifier corresponding to the change data, and indicate which attribute of which data has been changed.
In step S12, when the type of the changed data is the first type, the update times corresponding to the object identifier are updated in the first list, and when the cycle synchronization time point is reached, a synchronization operation is triggered based on the object identifier recorded in the first list.
In the embodiment of the disclosure, the first kind of data which is changed more frequently is periodically updated in batches, so as to prevent the frequent update from stressing the system. A first list may be maintained, where the first list is used to record first type data that changes between two periodic synchronization time points and the change times corresponding to the first type data, so as to implement deduplication of hot spot data with high frequency change and avoid pressure on the system caused by frequent updates.
FIG. 4 is a sub-flowchart of step S12 in one embodiment of the present disclosure.
Referring to fig. 4, step S12 may include:
step S121, obtaining a target identifier with the maximum current corresponding updating times in the first list;
step S122, generating a commodity data synchronization request based on the target identifier with the maximum current corresponding updating times and sending the commodity data synchronization request to the service module;
step S123, obtaining the changed data returned by the service module, and deleting the target identifier with the largest current corresponding update time in the first list and the update time thereof.
In the embodiment of the present disclosure, when the first type of data is synchronized in batch, the data that is the more hot (i.e., the larger the number of updates between two periodic synchronization time points) is synchronized in sequence by the method that the synchronization sequence is the earlier, so as to implement timely synchronization of the hot data.
In step S13, when the type of the changed data is the second type, triggering the synchronization operation based on the target identifier;
the synchronization action comprises calling a service module to read a cache module to obtain changed data, and updating and searching a database according to the changed data, wherein the data corresponding to the first type is data with the updating times larger than or equal to a preset value in a preset time period, and the data corresponding to the second type is data with the updating times smaller than the preset value in the preset time period.
In the disclosed embodiment, the second kind of data is updated on the fly to ensure the accuracy of the data with lower update frequency but higher importance.
The synchronization action may refer to the embodiment shown in fig. 2 and its corresponding description. It should be noted that, although the search database 12 acquires changed data covering all kinds when data synchronization is performed, the search database 12 synchronizes only the first product data in the changed data to the search database 12.
The method 300 is described in detail below with reference to specific examples.
In one embodiment of the present disclosure, the synchronization process of the first type of data includes:
1. after the user purchases a commodity, the order purchase amount corresponding to the commodity in the commodity SKU (Stock Keeping Unit) in the database 14 and the cache module 13 is updated, and meanwhile, the sales increase change weight of the commodity identifier in the order queue of the cache module 13 is updated;
2. acquiring a change quantity set of each commodity order subjected to weight removal within a period of time by using a CronJOB (a timing task firstly checks whether the residual change quantity in a sku _ order _ read _ set synchronous set is not synchronous, and if the sku _ order _ read _ set synchronous set is empty, a rename operation of Redis is executed to rename the update set sku _ order _ write _ set to sku _ order _ read _ set);
3. searching a synchronous service to obtain a commodity identifier of a synchronous set, and obtaining the latest purchase quantity in real time by inquiring the purchase quantity of a Redis order;
4. and the search synchronization service calls an ES part synchronization interface to update the order purchase quantity of the commodity according to the dynamically configured rule engine if the purchase quantity participates in the search sequencing.
In one embodiment of the present disclosure, the synchronization process of the second type of data includes:
1. when the background operator changes the data table in the database, triggering the SKU data main table to update the update _ time (update time), and deleting the cache data corresponding to the changed data by the database 14;
2. the data synchronization module 15 monitors binlog of the SKU data master table through canal, takes the target identifier of the changed data, delivers the target identifier and the change notice to a RockettMQ message center;
3. the data synchronization module 15 obtains a target identifier of the changed data in the rockmq message center, and remotely calls the service module 16 (for example, an intranet detail query interface of the microservice);
4. the service module 16 calls Redis to inquire the associated data of the target identifier of the changed data and the order and activity related cache data, if the cache is not hit, the database is inquired, the cache is updated by using the changed data after the related data is obtained from the database, the changed data is packaged and the changed data is returned;
5. the data synchronization module 15 obtains the changed data (intranet interface detail data) returned by the service module 16, and updates the data participating in the ES index to the ES by using a dynamically configured rule engine (configuring and filtering the field of the ES (search database 12) index, and using a jsup to obtain the value of the Json specified node according to the Path).
In summary, the embodiment of the present disclosure avoids frequent updating of the ES index by performing deduplication and timing synchronization on the high-frequency change of the hot data; by setting the filtering engine, the reduction of the excessive query efficiency of the ES index field is avoided, and the index field is supported to expand and change through dynamic configuration; the newly accessed data is stored by using the Redis cache, so that the frequent query of a database by searching synchronization service full update and searching query service encapsulated data is avoided; by adopting multi-table cache in the cache module 13, the full-amount cache updating of the commodity cache query association table caused by the partial data change of the commodity is avoided, and the cache data corresponding to the change table is only needed to be updated when the partial data of the commodity is changed, such as real-time display of the purchase amount of a commodity order, activity tags calculated according to activity rules and the like. That is, the embodiment of the present disclosure introduces ES and Redis, which solves the real-time performance of data update, query rate and search data display, and has a simple, stable, extensible, and real-time data synchronization effect in a high-concurrency scenario.
Corresponding to the above method embodiment, the present disclosure further provides a commodity data synchronization apparatus, which may be used to execute the above method embodiment.
Fig. 5 schematically shows a block diagram of a commodity data synchronization apparatus in an exemplary embodiment of the present disclosure.
Referring to fig. 5, the goods data synchronization apparatus 500 may include:
a change response module 51 configured to respond to the commodity data change message and obtain one or more types of change data and one or more object identifiers;
a dynamic data synchronization module 52, configured to update the update times corresponding to the target identifier in a first list when the type of the changed data is a first type, and trigger a synchronization action according to the target identifier recorded in the first list when a period synchronization time point is reached;
a static data synchronization module 53 configured to trigger the synchronization action according to the target identifier when the type of the changed data is a second type;
and the synchronization action module 54 is configured to call the service module to read the cache module to obtain the changed data, and update and search the database according to the changed data, where the commodity data corresponding to the first type is data whose update times are greater than or equal to a preset value within a preset time period, and the commodity data corresponding to the second type is data whose update times are less than the preset value within the preset time period.
In an exemplary embodiment of the present disclosure, the dynamic data synchronization module 52 is configured to: acquiring a target identifier with the maximum current corresponding updating times in the first list; generating a commodity data synchronization request based on the target identifier with the maximum current corresponding updating times and sending the commodity data synchronization request to the service module; and acquiring the changed data returned by the service module, and deleting the target identifier with the maximum current corresponding updating times and the updating times thereof in the first list.
In an exemplary embodiment of the present disclosure, the synchronization action module 54 is configured to update the first item data in the changed data to the search database.
In an exemplary embodiment of the present disclosure, the synchronization action module 54 is configured to: sending a commodity data synchronization request containing the target identifier to the service module so that the service module reads the cache module; when the service module finds the commodity data corresponding to the target identifier in the cache module, the service module acquires the commodity data corresponding to the target identifier as the changed data; and when the service module does not find the commodity data corresponding to the target identifier in the cache module, the service module searches the commodity data corresponding to the target identifier in a database as the changed data, and correspondingly writes the changed data and the target identifier into the cache module.
In an exemplary embodiment of the disclosure, the synchronization action module 54 is configured to send a commodity data synchronization request containing the target identifier to the service module, so that the service module reads the cache module; when the service module finds the commodity data corresponding to the target identifier in the cache module, the service module acquires the commodity data corresponding to the target identifier as the changed data; and when the service module does not find the commodity data corresponding to the target identifier in the cache module, the service module searches the commodity data corresponding to the target identifier in a database as the changed data, and correspondingly writes the changed data and the target identifier into the cache module.
Since the functions of the apparatus 500 have been described in detail in the corresponding method embodiments, the disclosure is not repeated herein.
It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.
In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.
An electronic device 600 according to this embodiment of the invention is described below with reference to fig. 6. The electronic device 600 shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 6, the electronic device 600 is embodied in the form of a general purpose computing device. The components of the electronic device 600 may include, but are not limited to: the at least one processing unit 610, the at least one memory unit 620, and a bus 630 that couples the various system components including the memory unit 620 and the processing unit 610.
Wherein the storage unit stores program code that is executable by the processing unit 610 to cause the processing unit 610 to perform steps according to various exemplary embodiments of the present invention as described in the above section "exemplary methods" of the present specification. For example, the processing unit 610 may perform the steps as shown in fig. 3.
The storage unit 620 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)6201 and/or a cache memory unit 6202, and may further include a read-only memory unit (ROM) 6203.
The memory unit 620 may also include a program/utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 600, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 600 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 650. Also, the electronic device 600 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 660. As shown, the network adapter 660 communicates with the other modules of the electronic device 600 over the bus 630. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.
In an exemplary embodiment of the present disclosure, there is also provided a computer-readable storage medium having stored thereon a program product capable of implementing the above-described method of the present specification. In some possible embodiments, aspects of the invention may also be implemented in the form of a program product comprising program code means for causing a terminal device to carry out the steps according to various exemplary embodiments of the invention described in the above section "exemplary methods" of the present description, when said program product is run on the terminal device.
The program product for implementing the above method according to an embodiment of the present invention may employ a portable compact disc read only memory (CD-ROM) and include program codes, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A computer readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
Furthermore, the above-described figures are merely schematic illustrations of processes involved in methods according to exemplary embodiments of the invention, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, e.g., in multiple modules.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
Claims (10)
1. A commodity data synchronization method is characterized by comprising the following steps:
responding to the commodity data change message, and acquiring one or more types of change data and one or more target identifiers;
when the type of the changed data is a first type, updating the corresponding updating times of the target identifier in a first list, and triggering a synchronization action according to the target identifier recorded in the first list when a cycle synchronization time point is reached;
when the type of the changed data is a second type, triggering the synchronous action according to the target identifier;
the synchronization action comprises calling a service module to read a cache module to obtain changed data, and updating and searching a database according to the changed data, wherein the data corresponding to the first type is data with the updating times larger than or equal to a preset value in a preset time period, and the data corresponding to the second type is data with the updating times smaller than the preset value in the preset time period.
2. The merchandise data synchronization method of claim 1, wherein the triggering a synchronization action according to the target identifier recorded in the first list comprises:
acquiring a target identifier with the maximum current corresponding updating times in the first list;
generating a commodity data synchronization request based on the target identifier with the maximum current corresponding updating times and sending the commodity data synchronization request to the service module;
and acquiring the changed data returned by the service module, and deleting the target identifier with the maximum current corresponding updating times and the updating times thereof in the first list.
3. The merchandise data synchronization method of claim 1, wherein the updating the search database according to the changed data comprises:
and updating the first commodity data in the changed data to the search database.
4. The merchandise data synchronization method of claim 1, wherein the invoking the service module to read the cache module to obtain the changed data comprises:
sending a commodity data synchronization request containing the target identifier to the service module so that the service module reads the cache module;
when the service module finds the commodity data corresponding to the target identifier in the cache module, the service module acquires the commodity data corresponding to the target identifier as the changed data;
and when the service module does not find the commodity data corresponding to the target identifier in the cache module, the service module searches the commodity data corresponding to the target identifier in a database as the changed data, and correspondingly writes the changed data and the target identifier into the cache module.
5. A commodity data synchronizing device, comprising:
a change response module configured to respond to the commodity data change message and acquire one or more types of change data and one or more target identifiers;
the dynamic data synchronization module is set to update the corresponding updating times of the target identifier in a first list when the type of the changed data is a first type, and trigger a synchronization action according to the target identifier recorded in the first list when a period synchronization time point is reached;
the static data synchronization module is set to trigger the synchronization action according to the target identifier when the type of the changed data is a second type;
and the synchronous action module is set to call the service module to read the cache module so as to obtain changed data and update and search the database according to the changed data, the commodity data corresponding to the first type is data of which the update times in a preset time period are more than or equal to a preset value, and the commodity data corresponding to the second type is data of which the update times in the preset time period are less than the preset value.
6. A merchandise data transmission system, comprising:
the search engine is used for responding to user operation to generate a commodity data search request and displaying commodity data corresponding to the commodity data search request to a user;
the search database is connected with the search engine and used for storing first commodity data and responding to the commodity data search request to return the first commodity data, and the first commodity data are data of which the change times in a preset time period are lower than a preset value;
the cache module is used for storing the first commodity data and second commodity data, wherein the second commodity data is data of which the change times in a preset time period are more than or equal to the preset value;
the database is connected with the cache module and used for storing the first commodity data and the second commodity data and emptying the commodity data corresponding to the target identifier of the changed data in the cache module when data change occurs;
the data synchronization module is connected with the search database and the database and used for monitoring commodity data change messages of the database so as to execute the commodity data synchronization method according to any one of claims 1 to 5 to synchronize data of the search database;
and the service module is connected with the search engine, the cache module, the database and the data synchronization module and used for responding to the commodity data search request from the search engine or the commodity data synchronization request from the data synchronization module to read the cache module or the database so as to return commodity data.
7. The merchandise data transmission system of claim 6, wherein the service module is configured to:
responding to a commodity data search request from the search engine, and sending a commodity data reading request corresponding to the identifier of the target commodity to the cache module;
when the commodity data corresponding to the identifier of the target commodity exists in the cache module, sending the commodity data corresponding to the identifier of the target commodity to the search engine;
when the commodity data corresponding to the identifier of the target commodity does not exist in the cache module, sending a commodity data reading request corresponding to the identifier of the target commodity to the database;
and after acquiring target data corresponding to the identifier of the target commodity from the database, sending the target data to the search engine, and writing the target data into the cache module.
8. The merchandise data transmission system of claim 6, wherein the service module is configured to:
responding to a commodity data synchronization request from the data synchronization module, and sending a commodity data reading request corresponding to a target identifier to the cache module;
when the commodity data corresponding to the target identifier exists in the cache module, sending the commodity data corresponding to the target identifier to the search engine;
when the commodity data corresponding to the target identifier does not exist in the cache module, sending a commodity data reading request corresponding to the target identifier to the database;
and after the target data corresponding to the target identifier is obtained from the database, the target data is sent to the search engine, and the target data is written into the cache module.
9. An electronic device, comprising:
a memory; and
a processor coupled to the memory, the processor configured to perform the merchandise data synchronization method of any of claims 1-4 based on instructions stored in the memory.
10. A computer-readable storage medium on which a program is stored, the program, when executed by a processor, implementing the merchandise data synchronization method according to any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011607163.8A CN112765187A (en) | 2020-12-30 | 2020-12-30 | Commodity data synchronization method and commodity data transmission system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011607163.8A CN112765187A (en) | 2020-12-30 | 2020-12-30 | Commodity data synchronization method and commodity data transmission system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112765187A true CN112765187A (en) | 2021-05-07 |
Family
ID=75697467
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011607163.8A Pending CN112765187A (en) | 2020-12-30 | 2020-12-30 | Commodity data synchronization method and commodity data transmission system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112765187A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113204561A (en) * | 2021-05-21 | 2021-08-03 | 长沙市到家悠享家政服务有限公司 | Method for keeping consistency of data of multi-system and multi-system adopting method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101155018A (en) * | 2006-09-28 | 2008-04-02 | 华为技术有限公司 | Data synchronization method and its implementing device and system |
| US20110295874A1 (en) * | 2010-05-27 | 2011-12-01 | International Business Machines Corporation | Metadata cache management |
| CN106649145A (en) * | 2016-12-15 | 2017-05-10 | Tcl集团股份有限公司 | Self-adaptive cache strategy updating method and system |
| CN112084258A (en) * | 2020-08-18 | 2020-12-15 | 腾讯科技（深圳）有限公司 | Data synchronization method and device |
-
2020
- 2020-12-30 CN CN202011607163.8A patent/CN112765187A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101155018A (en) * | 2006-09-28 | 2008-04-02 | 华为技术有限公司 | Data synchronization method and its implementing device and system |
| US20110295874A1 (en) * | 2010-05-27 | 2011-12-01 | International Business Machines Corporation | Metadata cache management |
| CN106649145A (en) * | 2016-12-15 | 2017-05-10 | Tcl集团股份有限公司 | Self-adaptive cache strategy updating method and system |
| CN112084258A (en) * | 2020-08-18 | 2020-12-15 | 腾讯科技（深圳）有限公司 | Data synchronization method and device |
Non-Patent Citations (1)
| Title |
|---|
| 周京晖: "数据缓存按需同步的设计与应用", 《软件》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113204561A (en) * | 2021-05-21 | 2021-08-03 | 长沙市到家悠享家政服务有限公司 | Method for keeping consistency of data of multi-system and multi-system adopting method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111858248A (en) | Application monitoring method, device, equipment and storage medium | |
| CN110765206A (en) | Data synchronization system, method, device, equipment and storage medium | |
| CN112084208A (en) | Data visualization method and system, storage medium and electronic device | |
| US10452879B2 (en) | Memory structure for inventory management | |
| CN112765187A (en) | Commodity data synchronization method and commodity data transmission system | |
| CN112307300A (en) | Article information query method and device, electronic equipment and readable storage medium | |
| US9785599B2 (en) | Information processing apparatus and log output method | |
| CN110866158A (en) | Information updating method, device, system, storage medium and electronic equipment | |
| CN114637448A (en) | Data processing method and device, electronic equipment and storage medium | |
| US20220276845A1 (en) | Systems and methods for consistent feature flag evaluation | |
| CN111737344B (en) | Data synchronization method and device for tag type weighing equipment | |
| CN113760977A (en) | Information query method, device, equipment and storage medium | |
| CN110716911B (en) | Data processing method and device, electronic equipment and storage medium | |
| CN111159207A (en) | Information processing method and device | |
| CN113051510B (en) | Interactive processing method, device, front-end equipment, back-end equipment and storage medium | |
| CN112000723B (en) | Enterprise information management device and application thereof | |
| CN113344680A (en) | Order processing method, related device, equipment and storage medium | |
| US11120486B2 (en) | Systems and methods for distributing online advertisements | |
| CN112182078A (en) | Method, device, equipment and storage medium for issuing and using electronic product | |
| CN112084252A (en) | Multi-data linkage interaction method and device | |
| CN109299139B (en) | Information query method and device | |
| CN111695749A (en) | Method and device for generating grouping tasks | |
| JP2020024649A (en) | Product information instant outputting system in operation of a plurality of websites | |
| US11314796B2 (en) | Dimension-specific dynamic text interface for data analytics | |
| US20140297744A1 (en) | Real-time supplement of segmented data for user targeting |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210507 |