CN115712677A - Search data synchronization method and device, equipment, medium and product thereof - Google Patents

Abstract

The application discloses a search data synchronization method and a device, equipment, medium and product thereof, wherein the method comprises the following steps: responding to a search data increment event acting on a service database, acquiring a plurality of increment search data, and determining an e-commerce independent station corresponding to the increment search data; counting the incremental data flow of the E-commerce independent station, verifying whether the incremental data flow meets a data flow limiting rule, and if not, pushing incremental search data into a data version verifier; when the incremental data flow meets the data flow limit rule, the incremental search data is pushed to a flow limit pusher so as to drive the flow limit pusher to speed-limit and push the incremental search data to a data version checker based on the speed-limit push rule; and the drive data version checker checks the data version of each increment search data and pushes the increment search data with the later data version to the e-commerce search service. The method and the device can limit the transmission speed of the concurrent search data to improve the stability of the data synchronization service.

Description

Search data synchronization method and device, equipment, medium and product thereof

Technical Field

The present application relates to the field of network e-commerce, and in particular, to a method for synchronizing search data, and further, to a corresponding apparatus, device, non-volatile storage medium, and computer program product of the method.

Background

The existing network e-commerce independent station platform can provide search service for an e-commerce independent station in the platform, so that buyer users entering the e-commerce independent station to carry out e-commerce shopping can search corresponding goods or order information in the e-commerce independent station through the search service, and the search service provided by the network e-commerce independent station platform for most of the e-commerce independent stations in the platform is shared search service, namely, each e-commerce independent station in the platform commonly uses the search service maintained by the same server cluster, so that the platform can maintain the search service.

When any independent E-commerce station puts new commodities on shelves or generates new orders, commodity data information or order data information is stored in a corresponding business database, so that the platform synchronizes data to search services, and a buyer user can inquire the newly put commodities or order information corresponding to online shopping through the search services.

In view of the problems of the existing data synchronization service acting on the search service plane to process concurrent data, the present applicant has made corresponding researches in view of solving the problems.

Disclosure of Invention

The application aims to meet the requirements of users and provides a search data synchronization method, and further relates to a corresponding device, equipment, a non-volatile storage medium and a computer program product of the method.

In order to realize the purpose of the application, the following technical scheme is adopted:

a method for synchronizing search data, adapted to the purpose of the present application, comprising the steps of:

responding to a search data increment event acting on a service database, acquiring a plurality of increment search data corresponding to the search data increment event, and determining an e-commerce independent station corresponding to the increment search data;

counting the incremental data flow of the independent E-commerce station, checking whether the incremental data flow meets a preset data flow limiting rule, and if not, pushing each incremental search data to a data version checker;

when the incremental data traffic meets the data flow limiting rule, pushing each incremental search data into a flow limiting pusher so as to drive the flow limiting pusher to speed-limit and push each incremental search data into the data version checker based on a speed-limit pushing rule;

and driving the data version checker to check the data version of each increment search data, and pushing the increment search data with the later data version to the e-commerce search service.

In a further embodiment, the step of obtaining a plurality of incremental search data corresponding to the incremental search data event in response to the incremental search data event acting on the service database, and determining the independent e-commerce station corresponding to the incremental search data includes the following steps:

responding to the search data increment event, and acquiring a data operation log corresponding to the search data increment event;

acquiring incremental search data corresponding to the service data identifications from the service database according to the plurality of service data identifications contained in the data operation log;

and determining the independent E-commerce station corresponding to the data operation log, and pushing each incremental search data to a to-be-consumed data queue.

In a further embodiment, the step of counting incremental data traffic of the e-commerce independent station, checking whether the incremental data traffic meets a preset data flow limiting rule or pushing each incremental search data to a flow limiting pusher when the incremental data traffic meets the data flow limiting rule includes the following steps:

counting the quantity of incremental search data corresponding to the current e-commerce independent station in the data queue to be consumed in a preset counting time window;

judging whether the quantity of the incremental search data exceeds a preset current limiting data quantity or not, and if not, pushing the incremental search data corresponding to the current e-commerce independent station in the to-be-consumed data queue to a data version checker;

and when the quantity of the incremental search data exceeds the preset current limiting data quantity, pushing the incremental search data corresponding to the current E-business independent station in the to-be-consumed data queue to a current limiting pusher in a preset current limiting time window.

In a further embodiment, the step of driving the current limit pusher to speed-limit and push each increment search data to the data version verifier based on a speed-limit pushing rule includes the following steps:

storing the current latest acquired increment search data to a first storage position of a current limiting data queue;

pushing the incremental searching data at the tail storage position in the current limiting data queue to a data version checker;

and by analogy, the storage and the pushing of the increment search data acting on the current-limiting data queue are processed, and the pushing speed of the increment search data is controlled according to the preset data pushing speed.

In a further embodiment, the step of driving the data version verifier to verify the data version of each incremental search data and pushing the incremental search data with a later data version to the e-commerce search service includes the following steps:

determining a data storage time stamp and data characteristic information of newly acquired increment search data, and correspondingly storing the data storage time stamp, the data characteristic information and the increment search data into a version verification cache space, wherein the data characteristic information comprises an e-commerce independent station identifier and a data attribute identifier;

querying a plurality of target increment search data with the same data characteristic information in the version verification cache space, and comparing respective data storage time stamps of the target increment search data;

and pushing the target incremental search data with the later data storage time stamp to the E-commerce search service, and removing other target incremental search data from the version check cache space.

In a further embodiment, the method comprises the following post steps:

responding to the search data increment event, and acquiring a data operation log corresponding to the search data increment event;

acquiring incremental search data corresponding to the service data identifications from the service database according to the plurality of service data identifications contained in the data operation log;

and determining an e-commerce independent station corresponding to the data operation log, and pushing each increment search data to the current-limiting pusher when the e-commerce independent station is a current-limiting processing e-commerce independent station.

A search data synchronization apparatus proposed in conformity with an object of the present application includes:

the incremental data acquisition module is used for responding to a search data incremental event acting on a service database, acquiring a plurality of incremental search data corresponding to the search data incremental event and determining an e-commerce independent station corresponding to the incremental search data;

the data flow verification module is used for counting the incremental data flow of the E-commerce independent station, verifying whether the incremental data flow meets a preset data flow limiting rule or not, and if not, pushing each incremental search data to the data version verifier;

the data flow limiting and pushing module is used for pushing each increment search data to a flow limiting pusher when the increment data flow meets the data flow limiting rule so as to drive the flow limiting pusher to push each increment search data to the data version checker in a speed-limiting manner based on a speed-limiting pushing rule;

and the data version checking module is used for driving the data version checker to check the data version of each increment search data and pushing the increment search data with the later data version to the e-commerce search service.

In a further embodiment, the incremental data obtaining module includes:

the operation log obtaining sub-module is used for responding to the search data increment event and obtaining a data operation log corresponding to the search data increment event;

the service data acquisition submodule is used for acquiring incremental search data corresponding to the service data identifications from the service database according to the plurality of service data identifications contained in the data operation log;

and the incremental data storage submodule is used for determining the E-commerce independent station corresponding to the data operation log and pushing each incremental search data to a to-be-consumed data queue.

In a further embodiment, the data traffic checking module includes:

the data quantity counting submodule is used for counting the quantity of incremental search data corresponding to the current e-commerce independent station in the data queue to be consumed in a preset counting time window;

the data quantity checking submodule is used for judging whether the quantity of the incremental search data exceeds a preset current-limiting data quantity or not, and if not, pushing the incremental search data corresponding to the current e-commerce independent station in the data queue to be consumed to the data version checker;

and the current limiting data pushing submodule is used for pushing the incremental searching data corresponding to the current e-commerce independent station in the data queue to be consumed into the current limiting pusher in a preset current limiting time window when the quantity of the incremental searching data exceeds a preset current limiting data quantity.

In a further embodiment, the data stream limitation pushing module includes:

the data storage submodule is used for storing the currently and latest acquired increment search data to the first storage position of the current limiting data queue;

the data pushing submodule is used for pushing the increment searching data at the tail storage position in the current limiting data queue to the data version checker;

and the pushing speed control submodule is used for processing the storage and pushing of the incremental search data acting on the current-limiting data queue by analogy, and controlling the pushing speed of the incremental search data according to the preset data pushing speed.

In a further embodiment, the data version checking module includes:

the mapping data cache submodule is used for determining a data storage time stamp and data characteristic information of newly acquired increment search data, and correspondingly storing the data storage time stamp, the data characteristic information and the increment search data into a version verification cache space, wherein the data characteristic information comprises an e-commerce independent station identifier and a data attribute identifier;

the storage time comparison sub-module is used for inquiring a plurality of target increment search data with the same data characteristic information in the version verification cache space and comparing respective data storage time stamps of the target increment search data;

and the incremental data pushing submodule is used for pushing the target incremental search data with the later data storage time stamp to the E-commerce search service, and removing other target incremental search data from the version checking cache space.

In order to solve the above technical problem, an embodiment of the present invention further provides a computer device, including a memory and a processor, where the memory stores computer-readable instructions, and the computer-readable instructions, when executed by the processor, cause the processor to execute the steps of the search data synchronization method.

In order to solve the above technical problem, embodiments of the present invention further provide a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the search data synchronization method.

In order to solve the above technical problem, an embodiment of the present invention further provides a computer program product, which includes a computer program and computer instructions, and when the computer program and the computer instructions are executed by a processor, the processor executes the steps of the search data synchronization method.

Compared with the prior art, the application has the following advantages:

the method and the system can limit the current-limiting data pushing of the E-commerce independent station with a large amount of incremental search data, so as to prevent the system from being broken down due to the fact that the data synchronization service cannot process the high-emission incremental search data, and the newly added service data in the platform cannot be pushed to the E-commerce search service; acquiring newly added service data in a service database as incremental search data by responding to an incremental event, and performing data flow statistics on an independent E-commerce station to which the data belong to so as to judge whether the incremental data of the independent E-commerce station exceeds the capacity or not, wherein the independent E-commerce station is required to be used as a current-limiting E-commerce station to perform data current-limiting pushing processing on the incremental search data; therefore, the method and the device can carry out current-limiting pushing on the incremental data of any E-commerce independent station in the platform to the E-commerce search service without carrying out global pushing on the incremental data of all E-commerce independent stations in the platform, prevent influence on data pushing of other E-commerce independent stations with normal incremental data flow, and ensure that buyer users of other E-commerce independent stations with normal flow in the platform can quickly search newly-shelved commodities or newly-generated orders by the E-commerce search service.

And secondly, the data version of the incremental search data is verified to screen the latest version of the data for pushing, the data version verifier is used for pushing the latest data version of the incremental search data in the incremental search data of the same E-commerce independent station and the same data type to the E-commerce search service, the data acquired by the E-commerce search service is guaranteed to be time-efficient data, and the result searched by a buyer user through the E-commerce search service is guaranteed to be the latest commodity or order.

Drawings

The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:

fig. 1 is a schematic diagram of a typical network deployment architecture related to implementing the technical solution of the present application;

FIG. 2 is a schematic flow chart diagram illustrating an exemplary embodiment of a search data synchronization method of the present application;

fig. 3 is a schematic flowchart illustrating a specific implementation of obtaining incremental search data corresponding to a search data incremental event in the present application;

FIG. 4 is a schematic flow chart illustrating an embodiment of the present application for verifying whether incremental data traffic of an independent E-commerce station satisfies a data throttling rule;

fig. 5 is a schematic flow chart illustrating an embodiment of a current limiting pusher for current limiting pushing incremental search data based on a current limiting pushing rule according to the present application;

FIG. 6 is a schematic flow chart diagram illustrating an embodiment of a data version verifier for verifying a data version of incremental search data according to the present application;

FIG. 7 is a schematic flow chart diagram illustrating an embodiment of the present application for subsequent processing of incremental search data for current limited utility independent stations;

FIG. 8 is a functional block diagram of an exemplary embodiment of a search data synchronization apparatus of the present application;

fig. 9 is a block diagram of a basic structure of a computer device according to an embodiment of the present application.

Detailed Description

Reference will now be made in detail to the embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary only for the purpose of explaining the present application and are not to be construed as limiting the present application.

As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. As used herein, the term "and/or" includes all or any element and all combinations of one or more of the associated listed items.

It will be understood by those within the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

As will be appreciated by those skilled in the art, "client," "terminal," and "terminal device" as used herein include both devices that are wireless signal receivers, which are devices having only wireless signal receivers without transmit capability, and devices that are receive and transmit hardware, which have receive and transmit hardware capable of two-way communication over a two-way communication link. Such a device may include: cellular or other communication devices such as personal computers, tablets, etc. having a single line display or a multi-line display or cellular or other communication devices without a multi-line display; PCS (Personal Communications Service), which may combine voice, data processing, facsimile and/or data communication capabilities; a PDA (Personal Digital Assistant), which may include a radio frequency receiver, a pager, internet/intranet access, a web browser, a notepad, a calendar and/or a GPS (Global Positioning System) receiver; a conventional laptop and/or palmtop computer or other device having and/or including a radio frequency receiver. As used herein, a "client," "terminal device" can be portable, transportable, installed in a vehicle (aeronautical, maritime, and/or land-based), or situated and/or configured to operate locally and/or in a distributed fashion at any other location(s) on earth and/or in space. The "client", "terminal Device" used herein may also be a communication terminal, a Internet access terminal, and a music/video playing terminal, and may be, for example, a PDA, an MID (Mobile Internet Device), and/or a Mobile phone with music/video playing function, and may also be a smart television, a set-top box, and other devices.

The hardware referred to by the names "server", "client", "work node", etc. in the present application is essentially an electronic device with the performance of a personal computer, and is a hardware device having necessary components disclosed by the von neumann principles such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device, in which a computer program is stored, and the central processing unit loads a program stored in an external memory into the internal memory to run, executes instructions in the program, and interacts with the input and output devices, thereby completing a specific function.

It should be noted that the concept of "server" as referred to in this application can be extended to the case of a server cluster. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided, and in physical space, the servers can be independent of each other but can be called through an interface, or can be integrated into a physical computer or a set of computer clusters. Those skilled in the art will appreciate this variation and should not be so limited as to restrict the implementation of the network deployment of the present application.

Referring to fig. 1, the hardware basis required for implementing the related art embodiments of the present application may be deployed according to the architecture shown in the figure. The server 80 is deployed in the cloud, and serves as an online server, which may be responsible for further connecting to related data servers and other servers providing related support, so as to form a logically associated server group, so as to provide services for related terminal devices, such as the smart phone 81 and the personal computer 82 shown in the figure, or a third-party server (not shown). Both the smart phone and the personal computer can access the internet in a known network access manner, and establish a data communication link with the cloud server 80 so as to run a terminal application program related to the service provided by the server.

For the server, the application program is usually constructed as a service process, and a corresponding program interface is opened to remotely call the application program running on various terminal devices.

The application program refers to an application program running on a server or a terminal device, the application program realizes the related technical scheme of the application in a programming mode, a program code of the application program can be saved in a nonvolatile storage medium which can be identified by a computer in a form of a computer executable instruction, and is called into a memory by a central processing unit to run, and the related device of the application is constructed through the running of the application program on the computer.

For the server, the application program is usually constructed as a service process, and a corresponding program interface is opened for remote call of the application program running on various terminal devices.

The person skilled in the art will know this: although the various methods of the present application are described based on the same concept so as to be common to each other, they may be independently performed unless otherwise specified. In the same way, for each embodiment disclosed in the present application, it is proposed based on the same inventive concept, and therefore, concepts of the same expression and concepts of which expressions are different but are appropriately changed only for convenience should be equally understood.

Referring to fig. 2, a search data synchronization method according to the present application, in an exemplary embodiment, includes the following steps:

step S11, responding to a search data increment event acting on a service database, acquiring a plurality of increment search data corresponding to the search data increment event, and determining an E-commerce independent station corresponding to the increment search data:

the e-commerce independent station is different from e-commerce shops in a traditional network e-commerce platform, the e-commerce independent station has independent network e-commerce online service, the e-commerce independent station can be regarded as a traditional network e-commerce platform, buyer users of the e-commerce independent station are own user groups, the e-commerce independent station is not like the traditional network e-commerce platform in which the buyer users of each e-commerce shop are own platform users, but the network e-commerce independent station platform can provide corresponding network e-commerce functions for the e-commerce independent station under the platform, for example, the e-commerce search function is provided for the independent station, and each independent station can provide e-commerce search service for searching goods or orders in the independent station for the buyer users of the independent station by using the search function.

The service database is used for storing the service data of the independent E-commerce stations such as commodity data and order data of the independent E-commerce stations in the platform, mapping and storing the service data and a service data identifier representing the independent E-commerce stations and the service data types corresponding to the service data identifier by the service database, so that the server can inquire the service data corresponding to the service data identifier from the service database according to the service data identifier, and further transmit the service data serving as the incremental search data to the E-commerce search service.

The search data increment event is an event triggered by storing new business data into the business database, for the e-commerce independent station platform, after a new commodity is put on any e-commerce independent station in the platform, the server stores the commodity data of the newly put commodity into the business database, the search data increment event acting on the commodity data is triggered, or after one or more buyer users carry out network shopping at any e-commerce independent station, corresponding order data is generated, the order data is stored into the business database by the server, and the search data increment event acting on the order data is triggered.

The server responds to the search data increment event, acquires the data operation log corresponding to the search data increment event, queries the service data corresponding to the service data identifications from the service database as the increment search data according to a plurality of service data identifications contained in the data operation log, and determines the e-commerce independent station corresponding to the data operation log.

The incremental searching data refers to business data which needs to be synchronously pushed to the e-commerce searching service, and the newly added business data in the business database is pushed to the e-commerce searching service, so that a buyer user can inquire a commodity newly put on the shelf of a certain e-commerce independent station or an order generated by shopping in a certain e-commerce platform through the e-commerce searching service.

Step S12, counting the incremental data flow of the E-commerce independent station, checking whether the incremental data flow meets a preset data flow limiting rule, and if not, pushing each incremental search data to a data version checker:

after the server acquires the incremental search data corresponding to the search data incremental event and determines the e-commerce independent station corresponding to the thank you incremental search data, the incremental data traffic of the e-commerce independent station is counted, wherein the incremental data traffic refers to the number of search incremental data which need to be pushed to the e-commerce search service for the e-commerce independent station in unit time, and for the search incremental data of the e-commerce independent station with larger incremental data traffic, the server needs to limit the pushing speed of the search incremental data so as to prevent the stability of the data synchronization service executed by the server from being influenced by concurrent transmission of huge data.

The method comprises the steps that after a server obtains increment search data of any one e-commerce independent station, the increment search data are stored in a data queue to be consumed to form a data stream, the server can count increment data flow of each e-commerce independent station conveniently, specifically, the server counts the increment search data number corresponding to each e-commerce independent station in the data queue to be consumed in a preset counting time window, whether the increment search data number exceeds a preset flow limiting data number or not is judged, if the increment search data number does not exceed the preset flow limiting data number, the server pushes the increment search data corresponding to the current e-commerce independent station in the data queue to be consumed to a data version checker, and if the increment search data number exceeds the flow limiting data number, the server pushes the increment search data corresponding to the current e-commerce independent station in the data queue to be consumed to a push flow limiter in the preset time window.

The data flow limiting rule is preset with corresponding flow limiting data volume, so that the server can determine that the incremental data flow exceeds the flow limiting data volume according to whether the incremental data flow of each electronic commerce independent station exceeds the flow limiting data volume, and determine that the incremental data flow does not exceed the flow limiting data flow.

In one embodiment, the e-commerce independent station determined to have the incremental data traffic meeting the data flow limiting rule is endowed with a flow limiting identifier for characterizing the e-commerce independent station as a flow limiting independent station, and after acquiring the search incremental data of the e-commerce independent station with the flow limiting identifier, the server directly pushes the search incremental data to the flow limiting pusher without counting the incremental data traffic of the e-commerce independent station for flow limiting verification.

The data version checker is used for checking the data version of the search increment data to prevent the search increment data which is newly stored in the business database from being overwritten by the search increment data which is stored earlier, namely, the latest search increment data is screened for the e-commerce search service; the detailed implementation of the data version checker refers to the subsequent steps, which is not repeated herein.

Step S13, when the incremental data flow meets the data flow limit rule, pushing each incremental search data to a flow limit pusher to drive the flow limit pusher to push each incremental search data to the data version checker in a speed-limiting manner based on a speed-limiting pushing rule:

and if the incremental data traffic of the independent E-commerce station to which the currently acquired incremental search data belongs meets the data flow limiting rule, the server pushes the incremental search data to the flow limiting pusher so as to control the pushing speed of the incremental search data.

The method comprises the steps that the current limiting pusher pushes each received increment search data at a speed limit according to a current limiting pushing rule, the current limiting pusher comprises a corresponding data pushing speed to control the pushing speed of the current limiting pusher for pushing each increment search data, the current limiting pusher carries out queue storage and queue pushing on each increment search data according to the current limiting pushing rule, specifically, after a server obtains the latest received increment search data, the increment search data are stored in the most front storage position in a current limiting data queue, sequential increment search data pushing is carried out according to a first-in first-out rule based on the queue, correspondingly, the current limiting pusher pushes the increment search data stored in the most rear storage position in the current limiting data queue to the data version checker, and so on, the current limiting pusher stores each received increment search data in the current limiting data queue and pushes each increment search data stored in the current limiting data queue, and controls the pushing speed of each increment search data in each current limiting data queue to push the check data of each current limiting data according to the pushing speed preset in the current limiting pushing rule.

Step S14, driving the data version checker to check the data version of each increment search data, and pushing the increment search data with the later data version to the e-commerce search service:

the data version checker performs data version check on each increment search data received by the data version checker, after receiving the increment search data, the data version checker acquires a data storage time stamp and data characteristic information of the increment search data, wherein the data storage time stamp refers to storage time of the increment search data stored in the service database, the data characteristic information comprises an electronic commerce independent station identifier and a data attribute identifier, the electronic commerce independent station identifier is used for representing an electronic commerce independent station to which the increment search data belongs, and the data attribute identifier is used for representing a data attribute of the increment search data.

The data version checker acquires the data storage time stamp and the data characteristic information of the incremental search data, stores the incremental search data, the data storage time stamp and the data characteristic information as incremental mapping relation data into a version checking cache space, caches a plurality of incremental mapping relation data in the version checking cache space, so that the data version checker maps the corresponding stored data storage time stamp and the data characteristic information according to each incremental search data in the version checking cache space, performs version checking on the incremental search data, specifically, the data version checker queries a plurality of target incremental search data with the same data characteristic information in the version checking cache space, namely the incremental search data with the same data attribute identification in the same independent station of the electric company, compares the respective data storage time stamps of the target incremental search data, determines that the target incremental search data with the later data storage time stamp is pushed to the electric company search service as the later incremental search data of the data version, and removes the target incremental search data with the later data version from the version checking cache space.

The e-commerce search service refers to a commodity or order search service provided by a network e-commerce independent station platform to each e-commerce independent station under the platform, the search service is generally a common search service, a buyer user at the e-commerce independent station can search commodities in the e-commerce independent station or orders corresponding to online purchases of the commodities in the e-commerce independent station through the e-commerce search service, for example, for the e-commerce search service providing the commodity search, the buyer user side can trigger to generate a search request containing a search text after inputting the corresponding search text through the search function of the independent station where the buyer user side is located, and pushes the search request to the e-commerce search service, the e-commerce search service acquires service data of one or more commodities corresponding to the search text as search results to be pushed to the buyer user side, so that the buyer user side receives the service data and then outputs corresponding commodity objects to a graphical user interface for display, and further allows the user at the buyer side to browse the commodity objects for online shopping.

According to the typical embodiment of the method, the method can limit the E-business independent station with huge increment search data to carry out data current-limiting pushing so as to prevent the system from being crashed due to the fact that the data synchronization service cannot process the increment search data with high emission, and the newly added business data in the platform cannot be pushed to the E-business search service; acquiring newly added service data in a service database as incremental search data by responding to an incremental event, and performing data flow statistics on an independent E-commerce station to which the data belong to so as to judge whether the incremental data of the independent E-commerce station exceeds the quantity or not, wherein the independent E-commerce station is required to be used as a current-limiting E-commerce independent station to perform data current-limiting pushing processing on the incremental search data; therefore, the method can carry out current-limiting pushing on the incremental data of any E-commerce independent station in the platform to the E-commerce search service without carrying out global pushing on the incremental data of all E-commerce independent stations in the platform, so that the data pushing of the E-commerce independent stations with normal incremental data flow is prevented from being influenced, and buyer users of other E-commerce independent stations with normal flow in the platform can quickly search newly-put commodities or newly-generated orders by using the E-commerce search service.

Secondly, the method carries out data version verification on the incremental search data to screen the latest version of the data for pushing, and pushes the latest data version of the incremental search data in the incremental search data of the same E-commerce independent station and the same data type to an E-commerce search service through a data version verifier, so that the data acquired by the E-commerce search service is guaranteed to be time-efficient data, and the result searched by a buyer user through the E-commerce search service is guaranteed to be the latest commodity or order.

The above exemplary embodiments and variations thereof fully disclose the embodiments of the search data synchronization method of the present application, but many variations thereof can be deduced by transforming and augmenting some technical means, and other embodiments are briefly described as follows:

in one embodiment, referring to fig. 3, in the step of acquiring a plurality of incremental search data corresponding to a search data incremental event in response to the search data incremental event acting on the service database, and determining an e-commerce independent station corresponding to the incremental search data, the method includes the following steps:

step S111, responding to the search data increment event, and acquiring a data operation log corresponding to the search data increment event:

the data operation log is a log for recording business data of the business database storage increment, business data identification of each business data of the business database storage increment is stored in the data operation log, and the data operation log has independent station identification corresponding to the independent station of the electronic commerce which represents the data operation log.

Step S112, according to a plurality of service data identifiers included in the data operation log, obtaining incremental search data corresponding to the service data identifiers from the service database:

the service data identification represents the storage position of the service data corresponding to the service data identification in the service database, the server inquires the service data corresponding to the service data identification in the service database, and the service data is used as the incremental search data to perform data synchronization processing.

Step S113, determining an e-commerce independent station corresponding to the data operation log, and pushing each incremental search data to a to-be-consumed data queue:

the data queue to be consumed is used for storing each increment search data acquired by the server from the business database, the server counts the increment data traffic of each independent station of the electronic commerce through the data queue to be consumed, and each increment search data is pushed to the current-limiting pusher or the data version checker through the data queue to be consumed in a data flow mode.

In this embodiment, the data operation log records service data newly stored in the service database, so that the new incremental data is obtained as incremental search data through the data operation log to perform data synchronization, and the incremental search data is pushed in a data stream manner through the data queue to be consumed, so that each piece of incremental search data is pushed in order, and data congestion caused by unordered pushing is prevented.

In an embodiment, referring to fig. 4, the step of counting incremental data traffic of the independent e-commerce station, checking whether the incremental data traffic meets a preset data flow limiting rule or pushing each incremental search data to a flow limiting pusher when the incremental data traffic meets the data flow limiting rule includes the following steps:

step S121, counting the quantity of incremental search data corresponding to the current e-commerce independent station in the data queue to be consumed in a preset counting time window:

the statistical time window is generally set within a time range of 10 to 15 seconds and units of second, or within a time range of 1 to 2 minutes and units of minutes, and those skilled in the art can flexibly design the time range of the statistical time window, which is not repeated herein.

After the server acquires the incremental search data of any independent E-commerce station, the incremental search data are stored in a data queue to be consumed, so that a data stream is formed, and the server can count the number of the incremental search data of each independent E-commerce station conveniently.

Step S122, judging whether the quantity of the increment search data exceeds a preset current-limiting data quantity, if not, pushing the increment search data corresponding to the current independent E-business station in the data queue to be consumed to a data version checker:

after counting the increment search data quantity of any independent E-commerce station, the server judges whether the increment search data quantity exceeds a preset current limiting data quantity, wherein the current limiting data quantity is designed based on the counting time window, and time ranges set for different time units have corresponding current limiting data quantities, for example, when the counting time window is generally set in a time range of 10 seconds and taking seconds as unit time, the current limiting data quantity is generally set in a data quantity range of 500-1000.

For the electric commercial independent stations of which the increment search data quantity does not exceed the current limiting data quantity, the increment search data of the electric commercial independent stations are directly pushed to the data version verifier without being pushed to the current limiting pusher to limit the pushing speed of the data.

Step S123, when the number of the incremental search data exceeds a preset current-limiting data amount, pushing the incremental search data corresponding to the current independent e-commerce station in the to-be-consumed data queue to a current-limiting pusher in a preset current-limiting time window:

the current limiting time window is generally set in a time range of 5-10 minutes or 20-30 minutes, and for the e-commerce independent stations with the increment search data quantity exceeding the current limiting data quantity, the e-commerce independent stations are pushed into the current limiting pusher in the current limiting time window, so that the current limiting pusher limits the increment search data of the e-commerce independent stations which push the data and send the data.

In this embodiment, the data volume of the incremental search data pushed by each e-commerce independent station in the real-time statistics platform is used for pushing the incremental search data of the e-commerce independent station with the excessively high data volume into the speed-limit pusher for speed-limit pushing, so that the data synchronization service is prevented from being broken down due to concurrent data volume pushing.

In one embodiment, referring to fig. 5, the step of driving the current-limit pusher to rate-limit and push each incremental search data to the data version verifier based on the rate-limit pushing rule includes the following steps:

step S131, storing the currently newly acquired incremental search data to the first storage location of the current limit data queue:

the current-limiting data queue is used for storing increment search data pushed to the data version checker, the current-limiting data queue pushes the stored increment search data to the data version checker in sequence according to a first-in first-out rule, the current-limiting pusher determines the receiving time of each received increment search data, and the increment search data with the earliest receiving time is preferentially stored in the data pushing queue.

Step S132, pushing the incremental search data in the last storage position in the current-limiting data queue to a data version verifier:

and the current limiting pusher pushes the increment searching data which is currently positioned at the tail storage position in the current limiting data pushing queue to the data version checker according to a rule of first-in-queue and first-out-queue.

Step S133, by analogy, storing and pushing the incremental search data acting on the current-limiting data queue, and controlling the pushing speed of the incremental search data according to the preset data pushing speed:

and the current-limiting pusher preferentially dequeues and pushes the increment search data which is queued in the current-limiting data queue to the data version checker so as to form a data stream and push the data of each increment search data to the data version checker.

The current limiting pusher pushes the pushing speed of each incremental search data in the current limiting data queue according to the preset data pushing speed, the data pushing speed is generally set to 500 to 1000 data per second, and a person skilled in the art can flexibly design the data pushing speed without repeated description.

In this embodiment, carry out data push speed limit through the increment search data to the electricity merchant independent station of high concurrent propelling movement data and handle to prevent that the data push of high emission from blockking up the data communication link and guarantee that data version calibrator can stably handle increment search data, carry out data push through the mode of queue simultaneously in order to form the data flow, the data receipt order of the data version calibrator of being convenient for.

In an embodiment, referring to fig. 6, the step of driving the data version verifier to verify the data version of each incremental search data and pushing the incremental search data with a later data version to the e-commerce search service includes the following steps:

step S141, determining a data storage time stamp and data characteristic information of newly acquired incremental search data, and correspondingly storing the data storage time stamp, the data characteristic information and the incremental search data into a version check cache space, wherein the data characteristic information comprises an e-commerce independent station identifier and a data attribute identifier:

the data storage timestamp refers to storage time of incremental search data stored in the service database, the data characteristic information includes an e-commerce independent station identifier and a data attribute identifier, the e-commerce independent station identifier is used for representing an e-commerce independent station to which the incremental search data belongs, the data attribute identifier is used for representing a data attribute of the incremental search data, for example, when the incremental search data is order data, the data attribute identifier represents an order number of the incremental search data or other unique identifiers representing the incremental search data as corresponding orders, and when the incremental search data is commodity data, the data attribute identifier represents a commodity number of the incremental search data or other unique identifiers representing the incremental search data as corresponding commodities.

The data version checker acquires the data storage time stamp and the data characteristic information of the incremental search data, and stores the incremental search data, the data storage time stamp and the data characteristic information as incremental mapping relation data into a version checking cache space, wherein a plurality of incremental mapping relation data are cached in the version checking cache space, so that the data version checker performs version checking on the incremental search data based on the data storage time stamp and the data characteristic information which are mapped and stored by the incremental search data in the version checking cache space.

Step S142, querying a plurality of target incremental search data having the same data characteristic information in the version check cache space, and comparing respective data storage timestamps of the target incremental search data:

and the data version checker compares the data storage time stamps of a plurality of incremental search data with the same data characteristic information in the version checking cache space to determine the business data which is newly stored in the business database in the incremental search data with the same independent station of the E-business and the same data attribute.

Step S143, pushing the target incremental search data with the later data storage timestamp to the e-commerce search service, and removing other target incremental search data from the version check cache space:

the target incremental search data with the later data storage timestamp is the business data which is newly stored in the business database, the target incremental search data is the incremental search data with the highest data version in the same type of data, the data version checker pushes the latest incremental search data to the e-commerce search service, and removes other incremental search data with lower data version than the target incremental search data in the version check cache space, so that the storage space is saved, and the data confusion is prevented.

In the embodiment, the data storage time stamps of the incremental search data of the same e-commerce independent station and the same data attribute are compared to determine the latest incremental search data of the data versions in the data to carry out data push, so that the latest data acquired by the e-commerce search service is ensured, and the timeliness of the data is ensured.

In one embodiment, referring to fig. 7, the method includes the following post steps:

step S15, responding to the search data increment event, and acquiring a data operation log corresponding to the search data increment event:

the data operation log is a log for recording business data of the business database storage increment, business data identification of each business data of the business database storage increment is stored in the data operation log, and the data operation log has independent station identification corresponding to the independent station of the electronic commerce which represents the data operation log.

Step S16, according to a plurality of service data identifiers included in the data operation log, obtaining incremental search data corresponding to the service data identifiers from the service database:

the service data identification represents the storage position of the service data corresponding to the service data identification in the service database, the server inquires the service data corresponding to the service data identification in the service database, and the service data are used as the increment search data to perform data synchronization processing.

Step S17, determining an e-commerce independent station corresponding to the data operation log, and pushing each increment search data to the current-limiting pusher when the e-commerce independent station is a current-limiting processing e-commerce independent station:

the current-limiting processing e-commerce independent station is determined to be an e-commerce independent station with incremental data traffic meeting the data current-limiting rule, the e-commerce independent station is endowed with the current-limiting identifier used for representing the e-commerce independent station, and after the server acquires the search incremental data of the e-commerce independent station with the current-limiting identifier, the server directly pushes the search incremental data to the current-limiting pusher without counting the incremental data traffic of the e-commerce independent station to perform current-limiting verification.

In this embodiment, the independent e-commerce station determined that the incremental data traffic satisfies the data traffic limiting rule has a traffic limiting identifier indicating that the independent e-commerce station needs to perform traffic limiting processing, and the incremental search data is directly pushed to the search traffic limiting pusher without performing secondary incremental data traffic statistics on the incremental search data, so as to improve the execution efficiency of the data traffic limiting processing and save unnecessary computational resources.

Further, a search data synchronization apparatus of the present application can be constructed by functionalizing the steps in the methods disclosed in the above embodiments, and according to this idea, please refer to fig. 8, wherein in an exemplary embodiment, the apparatus includes: the incremental data acquisition module 11 is configured to respond to a search data incremental event that acts on a service database, acquire a plurality of incremental search data corresponding to the search data incremental event, and determine an e-commerce independent station corresponding to the incremental search data; the data flow verification module 12 is configured to count incremental data flows of the e-commerce independent station, verify whether the incremental data flows meet preset data flow limiting rules, and if not, push each incremental search data to a data version verifier; the data flow-limiting pushing module 13 is configured to, when the incremental data traffic satisfies the data flow-limiting rule, push each incremental search data to a flow-limiting pusher so as to drive the flow-limiting pusher to speed-limit and push each incremental search data to the data version verifier based on a speed-limiting pushing rule; and the data version checking module 14 is used for driving the data version checker to check the data version of each incremental search data and pushing the incremental search data with the later data version to the e-commerce search service.

In one embodiment, the incremental data obtaining module 11 includes: the operation log obtaining submodule is used for responding to the search data increment event and obtaining a data operation log corresponding to the search data increment event; the service data acquisition submodule is used for acquiring incremental search data corresponding to the service data identifications from the service database according to the plurality of service data identifications contained in the data operation log; and the incremental data storage submodule is used for determining the E-commerce independent station corresponding to the data operation log and pushing each incremental search data to a to-be-consumed data queue.

In one embodiment, the data traffic verification module 12 includes: the data quantity counting submodule is used for counting the quantity of incremental search data corresponding to the current e-commerce independent station in the data queue to be consumed in a preset counting time window; the data quantity checking submodule is used for judging whether the quantity of the incremental search data exceeds a preset current-limiting data quantity or not, and if not, pushing the incremental search data corresponding to the current e-commerce independent station in the data queue to be consumed to the data version checker; and the current limiting data pushing submodule is used for pushing the incremental searching data corresponding to the current e-commerce independent station in the data queue to be consumed into the current limiting pusher in a preset current limiting time window when the quantity of the incremental searching data exceeds a preset current limiting data quantity.

In one embodiment, the data current limiting push module 13 includes: the data storage submodule is used for storing the current latest acquired increment search data to the first storage position of the current limiting data queue; the data pushing submodule is used for pushing the increment searching data at the tail storage position in the current limiting data queue to the data version checker; and the pushing speed control submodule is used for processing the storage and pushing of the incremental search data acting on the current limiting data queue by analogy, and controlling the pushing speed of the incremental search data according to the preset data pushing speed.

In a further embodiment, the data version checking module 14 includes: the mapping data cache submodule is used for determining a data storage time stamp and data characteristic information of newly acquired increment search data, and correspondingly storing the data storage time stamp, the data characteristic information and the increment search data into a version verification cache space, wherein the data characteristic information comprises an e-commerce independent station identifier and a data attribute identifier; the storage time comparison sub-module is used for inquiring a plurality of target increment search data with the same data characteristic information in the version verification cache space and comparing respective data storage time stamps of the target increment search data; and the incremental data pushing submodule is used for pushing the target incremental search data with the later data storage time stamp to the e-commerce search service, and removing other target incremental search data from the version verification cache space.

In order to solve the above technical problem, an embodiment of the present application further provides a computer device, configured to run a computer program implemented according to the search data synchronization method. Referring to fig. 9, fig. 9 is a block diagram of a basic structure of a computer device according to the present embodiment.

As shown in fig. 9, the internal structure of the computer device is schematic. The computer device includes a processor, a non-volatile storage medium, a memory, and a network interface connected by a system bus. The non-volatile storage medium of the computer device stores an operating system, a database, and computer readable instructions, the database can store control information sequences, and the computer readable instructions, when executed by the processor, can cause the processor to implement a search data synchronization method. The processor of the computer device is used for providing calculation and control capability and supporting the operation of the whole computer device. The memory of the computer device may have stored therein computer readable instructions that, when executed by the processor, may cause the processor to perform a method of search data synchronization. The network interface of the computer device is used for connecting and communicating with the terminal. It will be appreciated by those skilled in the art that the configuration shown in fig. 9 is a block diagram of only a portion of the configuration associated with the present application, and is not intended to limit the computing device to which the present application may be applied, and that a particular computing device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.

In this embodiment, the processor is configured to execute specific functions of each module/sub-module in the search data synchronization apparatus of the present application, and the memory stores program codes and various types of data required for executing the modules. The network interface is used for data transmission to and from a user terminal or a server. The memory in this embodiment stores program codes and data necessary for executing all modules/submodules in the search data synchronization device, and the server can call the program codes and data of the server to execute the functions of all the submodules.

The present application also provides a non-volatile storage medium, wherein the search data synchronization method is written as a computer program and stored in the storage medium in the form of computer readable instructions, which when executed by one or more processors, means the execution of the program in a computer, thereby causing the one or more processors to perform the steps of any of the above-described search data synchronization methods.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the computer program is executed. The storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a Random Access Memory (RAM).

In summary, the application can limit data flow-limited pushing of the independent e-commerce station with a huge amount of incremental search data to prevent the system from being crashed due to the fact that the data synchronization service cannot process the high-emission incremental search data, and perform data version verification on the incremental search data to screen the latest version of data for pushing, so as to ensure that the result searched by the buyer user through the e-commerce search service is the latest commodity or order.

It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

Those of skill in the art will appreciate that the various operations, methods, steps in the processes, acts, or solutions discussed in this application can be interchanged, modified, combined, or eliminated. Further, other steps, measures, or schemes in various operations, methods, or flows that have been discussed in this application can be alternated, altered, rearranged, broken down, combined, or deleted. Further, the steps, measures, and schemes in the various operations, methods, and flows disclosed in the present application in the prior art can also be alternated, modified, rearranged, decomposed, combined, or deleted.

The foregoing is only a partial embodiment of the present application, and it should be noted that, for those skilled in the art, several modifications and decorations can be made without departing from the principle of the present application, and these modifications and decorations should also be regarded as the protection scope of the present application.

Claims (10)

1. A search data synchronization method, comprising the steps of:

responding to a search data increment event acting on a service database, acquiring a plurality of increment search data corresponding to the search data increment event, and determining an e-commerce independent station corresponding to the increment search data;

counting the incremental data traffic of the E-commerce independent station, checking whether the incremental data traffic meets a preset data flow limiting rule, and if not, pushing each incremental search data to a data version checker;

when the incremental data traffic meets the data flow limiting rule, pushing each incremental search data into a flow limiting pusher so as to drive the flow limiting pusher to speed-limit and push each incremental search data into the data version checker based on a speed-limit pushing rule;

and driving the data version checker to check the data version of each incremental search data, and pushing the incremental search data with the later data version to the e-commerce search service.

2. The method according to claim 1, wherein the step of obtaining a plurality of incremental search data corresponding to the incremental search data event in response to the incremental search data event acting on the service database, and determining the independent e-commerce station corresponding to the incremental search data comprises the steps of:

responding to the search data increment event, and acquiring a data operation log corresponding to the search data increment event;

acquiring incremental search data corresponding to the service data identifications from the service database according to the plurality of service data identifications contained in the data operation log;

and determining an e-commerce independent station corresponding to the data operation log, and pushing each incremental search data to a to-be-consumed data queue.

3. The method according to claim 2, wherein the step of counting the incremental data traffic of the independent e-commerce station, checking whether the incremental data traffic meets a preset data flow limit rule or pushing each incremental search data to a flow limit pusher when the incremental data traffic meets the data flow limit rule comprises the following steps:

counting the quantity of incremental search data corresponding to the current e-commerce independent station in the data queue to be consumed in a preset counting time window;

judging whether the quantity of the incremental search data exceeds a preset current limiting data quantity or not, and if not, pushing the incremental search data corresponding to the current e-commerce independent station in the to-be-consumed data queue to a data version checker;

and when the quantity of the incremental search data exceeds the preset current limiting data quantity, pushing the incremental search data corresponding to the current E-business independent station in the to-be-consumed data queue to a current limiting pusher in a preset current limiting time window.

4. The method of claim 1, wherein the step of driving the limited-rate presenter to rate-present each of the incremental search data to the data version verifier based on a rate-limiting presentation rule comprises the steps of:

storing the current latest acquired increment search data to the first storage position of the current limiting data queue;

pushing the increment searching data at the tail storage position in the current limiting data queue to a data version checker;

and by parity of reasoning, storing and pushing the incremental search data acting on the current-limiting data queue, and controlling the pushing speed of the incremental search data according to the preset data pushing speed.

5. The method of claim 1, wherein the step of driving the data version verifier to verify the data version of each incremental search data and to push the incremental search data with a later data version to the e-commerce search service comprises the steps of:

determining a data storage time stamp and data characteristic information of newly acquired increment search data, and correspondingly storing the data storage time stamp, the data characteristic information and the increment search data into a version verification cache space, wherein the data characteristic information comprises an e-commerce independent station identifier and a data attribute identifier;

inquiring a plurality of target increment search data with the same data characteristic information in the version verification cache space, and comparing respective data storage time stamps of the target increment search data;

and pushing the target incremental search data with the later data storage time stamp to the E-commerce search service, and removing other target incremental search data from the version check cache space.

6. Method according to claim 1, characterized in that the method comprises the following post-steps:

responding to the search data increment event, and acquiring a data operation log corresponding to the search data increment event;

acquiring incremental search data corresponding to the service data identifications from the service database according to the plurality of service data identifications contained in the data operation log;

and determining an e-commerce independent station corresponding to the data operation log, and pushing each increment search data to the current-limiting pusher when the e-commerce independent station is a current-limiting processing e-commerce independent station.

7. A search data synchronization apparatus, comprising:

the incremental data acquisition module is used for responding to a search data incremental event acting on a service database, acquiring a plurality of incremental search data corresponding to the search data incremental event and determining an e-commerce independent station corresponding to the incremental search data;

the data flow verification module is used for counting the incremental data flow of the E-commerce independent station, verifying whether the incremental data flow meets a preset data flow limiting rule, and if not, pushing each incremental search data to the data version verifier;

the data flow limiting and pushing module is used for pushing each increment search data to a flow limiting pusher when the increment data flow meets the data flow limiting rule so as to drive the flow limiting pusher to push each increment search data to the data version checker in a speed-limiting manner based on a speed-limiting pushing rule;

and the data version checking module is used for driving the data version checker to check the data version of each increment search data and pushing the increment search data with the later data version to the e-commerce search service.

8. An electronic device comprising a central processor and a memory, characterized in that the central processor is configured to invoke execution of a computer program stored in the memory to perform the steps of the method according to any one of claims 1 to 6.

9. Non-volatile storage medium, characterized in that it stores, in the form of computer-readable instructions, a computer program implemented according to the method of any one of claims 1 to 6, which, when invoked by a computer, performs the steps comprised by the method.

10. A computer program product comprising computer program/instructions, characterized in that the computer program/instructions, when executed by a processor, implement the steps of the method of any one of claims 1 to 6.

Images