WO2016091069A1

WO2016091069A1 - Data operation method and device

Info

Publication number: WO2016091069A1
Application number: PCT/CN2015/095544
Authority: WO
Inventors: 朱超; 陈非; 王超
Original assignee: 北京奇虎科技有限公司; 奇智软件（北京）有限公司
Priority date: 2014-12-12
Filing date: 2015-11-25
Publication date: 2016-06-16
Also published as: CN104376127A; CN104376127B

Abstract

Disclosed are a data operation method and device. The method comprises: receiving a read request for data (S101); searching for a recorded write moment at which the data is written into a primary library (S102); and determining a time length from the write moment to a current moment, and sending the read request to the primary library or a secondary library according to the write moment so as to read the data (S103). In the method, after receiving the read request for the data, middleware performs judgement based on the length of time from the write moment to the current moment of the data and a pre-set threshold value as to decide whether to send the read request to the primary library or the secondary library instead of directly sending the read request to the secondary library to read the data. Thus, during reading of the data, even if data synchronisation is not completed in the secondary library, the data can still be read from the primary library directly, the scenario of reading failure caused by non-completion of data synchronisation is avoided, and the success rate of data reading is effectively improved.

Description

Data operation method and device

Technical field

The present invention relates to the field of computer technologies, and in particular, to a data operation method and apparatus.

Background technique

With the development of information technology, users can use the client to access the corresponding database (such as MySQL database, Oracle database, etc.) through the network to obtain the required data. The database is divided into a main library and a slave library, and the data is processed by means of read-write separation: the read request issued by the client is processed by the library, and the write request sent by the client is processed by the main library.

In the prior art, a middleware (a data transfer device) is usually disposed between the database and the client to manage the connection interaction between the database and a large number of clients, and provide data services for each client, that is, the client. The data connection is made between the end and the database through the middleware. After the middleware establishes a connection with the database, the request sent by the user is sent to the main library or the slave library according to the type of the request.

For example, if the client issues a write request, the middleware sends the write request to the main library after receiving the write request; when the client issues a read request, the middleware sends the read request to the slave library.

After the main library executes the write request sent by the client, after the operation of writing the data is completed, the main library sends a log to the slave library, so that the newly written data is synchronized to the slave library, thereby completing the process of synchronously updating the data in the database.

However, it takes a certain time for the main library to synchronize data with the slave library. If the main library synchronizes a certain data to the slave library, the client issues a read request to read the data, then the synchronization process cannot be completed because the synchronization process is not completed. The data is read from the library, which in turn causes the read data to fail. It can be seen that there is a great chance in the prior art that the data cannot be read normally.

Summary of the invention

In view of the above problems, the present invention has been made in order to provide a data operation method and apparatus that overcomes the above problems or at least partially solves the above problems, and can effectively avoid a large probability that data cannot be read normally due to data synchronization.

According to a first aspect of the present invention, a data operation method is provided, comprising: receiving a read request for data;

Finding the write time of the record to write the data to the main library;

The read request is sent to the main library or the slave library to read the data according to the write time.

According to a second aspect of the present invention, a data operation apparatus is provided, including: a receiving module, a lookup module, a time difference module, and a judgment processing module, wherein

The receiving module is configured to receive a read request for data;

The searching module is configured to search for a write moment of writing the data to the main library;

The time module is configured to determine a length of time from the writing time to the current time;

The determining processing module is configured to determine whether the length of time is greater than a preset threshold, and if yes, send the read request to the slave library to read the data; otherwise, send the read request to the main library to read The data.

According to a third aspect of the present invention, a computer program is presented, comprising computer readable code causing the computing device to perform the data manipulation method described above when the computer readable code is run on a computing device .

According to a fourth aspect of the invention, a computer readable medium is proposed, wherein the computer program described above is stored.

According to the data operation method provided by the present invention, after receiving the read request for a certain data, the middleware does not directly send the read request to the slave library to read the data, but searches for the write of the recorded data. At the moment, it is determined whether the length of time from the writing time of the data to the current time is greater than a preset threshold. If the length of time is greater than the threshold, the data may be considered to have been synchronized from the main library to the slave library, then the middleware will read the request. Send to the slave library to read the data; if the length of time is less than the threshold, it means that the data has not been synchronized to the slave library, so the middleware sends the read request to the master library to read directly from the master library. take. In this way, when data is read, even if the data is not synchronized from the library, the data can be read directly from the main library, and the data does not fail because the data is not synchronized. The situation effectively improves the success rate of reading data.

The above description is only an overview of the technical solutions of the present invention, and the above-described and other objects, features and advantages of the present invention can be more clearly understood. , the following specializes in the invention detailed description.

DRAWINGS

Various other advantages and benefits will become apparent to those skilled in the art from a The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting. Throughout the drawings, the same reference numerals are used to refer to the same parts. In the drawing:

1 is a schematic diagram of a data operation process according to an embodiment of the present invention;

2 is a schematic diagram showing a connection architecture of a client, a middleware, and a database in an example according to an embodiment of the present invention;

3 is a schematic diagram of a data operation process in practical application according to an embodiment of the present invention;

4 is a schematic structural diagram of a data operation device according to an embodiment of the present invention;

Figure 5 shows schematically a block diagram of a computing device for performing the method according to the invention;

Fig. 6 schematically shows a storage unit for holding or carrying program code implementing the method according to the invention.

Specific embodiment

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While the embodiments of the present invention have been shown in the drawings, the embodiments Rather, these embodiments are provided so that this disclosure will be more fully understood and the scope of the disclosure will be fully disclosed.

The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

FIG. 1 is a data operation method according to an embodiment of the present invention, which may include the following steps:

S101: Receive a read request for data.

S102: Find a write time of the recorded data.

S103: Send the read request to the main library or the slave library according to the write time to read the data.

As can be seen from the above, in this embodiment, after receiving the read request for a certain data, the middleware does not directly send the read request to the data read from the library, but searches for the recorded data. At the time of writing, the purpose of finding the write time of the data is to determine whether the data has completed the synchronization process from the main library to the slave library according to the write time. In this way, the read request can be sent to the main library or the slave library to read the data according to the write timing of the data. Effectively improve the success rate of reading data.

According to an embodiment of the present invention, the middleware can determine the data corresponding to the read request according to the read request sent by the client, and then send the read request to the corresponding database to read the data. In view of the prior art, when the middleware directly transmits the received read request to the slave library to read the data, there is a case where the read failure occurs because the data is not synchronized. Therefore, in the embodiment of the present invention, when the middleware receives a read request for a certain data, it will find the write time of the data.

That is, in the present invention, based on the write time of the data, it can be determined whether the data is synchronized when the read request is issued. Specifically, the method further includes: determining the write time to the current The length of time is determined by whether the length of time is greater than a preset threshold, and the read request is sent to the main library or the slave library to read the data according to the foregoing judgment result.

If the length of time from the writing time of the data to the current time is greater than a preset threshold, the data may be considered to have been synchronized from the main library to the slave library, then the middleware sends the read request to the slave library to read the data; If the length of time is less than the threshold, it means that the data has not been synchronized to the slave library, so that the middleware sends the read request to the master library to read directly from the master library. In this way, when data is read, even if the data is not synchronized from the library, the data can be read directly from the main library, and the data does not fail because the data is not synchronized. The situation effectively improves the success rate of reading data.

It should be noted here that the data write operation is performed by the main library, that is, after receiving the write request of the data sent by the client, the middleware sends the write request to the main library for data writing. Then, the time at which the data is written is the time at which the main library receives the write request and writes the data. Specifically, in accordance with an embodiment of the present invention, before receiving a read request for data, the method further includes: receiving a write request for the data, and transmitting the write request to a main library for writing, recording the The time at which the data was written.

In this way, all the data written in the main library have their respective write times. Of course, in the present invention, for the recording of the data writing time, the middleware can be notified immediately after the main library completes the writing of the data, and the middleware records the time when the notification is received as the writing time of the data. . Of course, in another mode of the present invention, the main library responsible for writing data can also be used. When the write time corresponding to each data is recorded, the middleware reads the write time of the data recorded by the main library in real time.

According to an embodiment of the present invention, after the middleware finds the writing time of the data, the length of time from the writing time to the current time can be determined. In the embodiment of the present invention, the current time is the time when the middleware receives the read request for the data.

It should also be noted that after the main library writes the data, the data is synchronized to the slave library, specifically, the master library synchronizes the data with the slave library, and the binlog log can be used to synchronize the data, that is, the main library will already be The written data is recorded in the binlog log, and the binlog log is sent to the slave library. After receiving the binlog log from the library, the operation records in the log are replayed, and the data synchronization is completed. Of course, such a synchronous manner does not constitute a limitation of the present invention. Obviously, it takes a certain amount of time to synchronize the data from the library.

Then, for the read request received by the middleware, the data to be read may have been synchronized or may not be synchronized. Therefore, the middleware determines whether the data is synchronized according to the length of time from the current time to the writing time of the data and the preset threshold, and further determines whether to send the read request to the main library or the slave library according to the determination result.

Specifically, the middleware determines whether the length of time from the writing time of the data to the current time is greater than a preset threshold. If the length of time is greater than (or equal to) the preset threshold, it indicates that the slave has completed the synchronization of the data, and the middleware sends a read request for the data to the slave library for reading. If the length of time is less than the preset threshold, it indicates that the slave library has not completed the synchronization of the data. At this time, the middleware sends the read request to the slave library, which causes the read failure. Therefore, in order to successfully read the data, the middle is The piece sends the read request to the main library.

Specifically, the method for setting the preset threshold may be: determining an average duration of the primary library to synchronize data to the slave library, and setting the average duration to the preset threshold. This is because the time required for the main library to synchronize different data to the slave library is different. In order to reflect the time consuming of all data synchronization processes, the average length of time required to synchronize each data is averaged, and the average duration is obtained. The average duration is set to a preset threshold. Through the preset threshold, it is possible to basically determine whether the data has been synchronized.

In a practical application, a small amount of data may take a long time in the synchronization process, and may be greater than a preset threshold. Therefore, as another mode of the embodiment of the present invention, the maximum duration of the synchronization data may be set to the pre-predetermined value. Set the threshold. The above preset threshold can be adjusted according to the needs of the actual application. It is not intended to limit the invention.

Certainly, the foregoing preset threshold is dynamically changed, and the preset threshold will be changed according to the data that has been synchronized. Specifically, the more the number of data that has been synchronized, the more the main library synchronizes the data to the average duration of the slave library. Accurate (that is, the more accurately it reflects the overall time-consuming size).

Specific application examples of the data operation method described in the present invention are as follows:

Figure 2 shows the connection architecture of the client, middleware, main library, and slave library in practical applications. Assuming that the preset threshold is 2 s, data A and data B have been written in the main library, wherein the write time of data A is 10:20:17, and the write time of data B is 10:20:20.

At 10:20:21, the middleware receives the read request I and the read request II sent by the client, wherein the data to be read by the read request I is data A, and the data to be read by the read request II is data B.

The writing time of the data A of the middleware search record is 10:20:17, and the time length of the writing time of the data A to the current time (10:20:21) is 4s. Obviously, the length of time 4s is greater than the preset. The threshold is 2 s, which means that the data A has been synchronized to the slave library, so the middleware sends the read request I to the slave library to read the data A.

The writing time of the data B of the middleware search record is 10:20:20, and the writing time of the data B to the current time (10:20:21) is 1 s. Obviously, the length of time 1s is less than the preset. The threshold is 2s, which means that the data B has not been synchronized yet, so the middleware sends the read request II directly to the main library to read the data B.

As shown in FIG. 3, in combination with the above method and application example, in the present invention, the practical applications of the data operation methods S101 to S103 in FIG. 1 are as follows:

S301: The middleware receives a read request for data.

S302: The middleware searches for the write time of the recorded data.

S303: The middleware determines a length of time from the writing time to the current time.

S304: The middleware determines whether the length of time is greater than a preset threshold. If yes, step S305 is performed; otherwise, step S306 is performed.

S305: The middleware sends the read request to the slave library to read the data.

S306: The middleware sends the read request to the main library to read the data.

The above is the data operation method provided by the embodiment of the present invention. Based on the same idea, the embodiment of the present invention further provides a data operation device, as shown in FIG. 4 .

The data operating device in FIG. 4 is disposed in the middleware and connected to the client and the server. The device includes: a receiving module 401, a searching module 402, a time module 403, and a determining processing module 404, where

The receiving module 401 is configured to receive a read request for data.

The searching module 402 is configured to search for a write moment of writing the data to the main library.

The time module 403 is configured to determine a length of time from the writing time to the current time.

The determining processing module 404 is configured to determine whether the length of time is greater than a preset threshold, and if yes, send the read request to the slave library to read the data; otherwise, send the read request to the main library to read Take this data.

In an embodiment of the present invention, the receiving module 401 is further configured to receive a write request for the data.

In an embodiment of the present invention, the determining processing module 404 is further configured to send the received write request for the data to the main library for writing.

In an embodiment of the present invention, the data operation device further includes: a recording module 405, configured to record a write time when the data is written into the main library.

The preset threshold is set by determining that the main library synchronizes data to an average duration of the slave library, and setting the average duration to the preset threshold; or determining that the primary library synchronizes data to The maximum duration is set to the preset threshold from the maximum duration of the library.

Embodiments of the present invention provide a data operation method and apparatus. After receiving a read request for a certain data, the middleware does not directly send the read request to the slave library to read data, but searches for the recorded The writing time of the data is determined whether the length of time from the writing time of the data to the current time is greater than a preset threshold. If the time length is greater than the threshold, the data may be considered to have been synchronized from the main library to the slave library, then The middleware sends a read request to the slave library to read the data; and if the length of time is less than the threshold, it indicates that the data has not been synchronized to the slave library, so that the middleware sends the read request to the master library to directly Read from the main library. In this way, when data is read, even if the data is not synchronized from the library, the data can be read directly from the main library, and the data does not fail because the data is not synchronized. The situation effectively improves the success rate of reading data.

The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general purpose systems can also be used with the teaching based on the teachings herein. According to the above description, The structure required to construct such a system is obvious. Moreover, the invention is not directed to any particular programming language. It is to be understood that the invention may be embodied in a variety of programming language, and the description of the specific language has been described above in order to disclose the preferred embodiments of the invention.

In the description provided herein, numerous specific details are set forth. However, it is understood that the embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of the description.

Similarly, the various features of the invention are sometimes grouped together into a single embodiment, in the above description of the exemplary embodiments of the invention, Figure, or a description of it. However, the method disclosed is not to be interpreted as reflecting the intention that the claimed invention requires more features than those recited in the claims. Rather, as the following claims reflect, inventive aspects reside in less than all features of the single embodiments disclosed herein. Therefore, the claims following the specific embodiments are hereby explicitly incorporated into the embodiments, and each of the claims as a separate embodiment of the invention.

Those skilled in the art will appreciate that the modules in the devices of the embodiments can be adaptively changed and placed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and further they may be divided into a plurality of sub-modules or sub-units or sub-components. In addition to such features and/or at least some of the processes or units being mutually exclusive, any combination of the features disclosed in the specification, including the accompanying claims, the abstract and the drawings, and any methods so disclosed, or All processes or units of the device are combined. Each feature disclosed in this specification (including the accompanying claims, the abstract and the drawings) may be replaced by alternative features that provide the same, equivalent or similar purpose.

In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features that are included in other embodiments and not in other features, combinations of features of different embodiments are intended to be within the scope of the present invention. Different embodiments are formed and formed. For example, in the following claims, any one of the claimed embodiments can be used in any combination.

The various component embodiments of the present invention may be implemented in hardware, or in a software module running on one or more processors, or in a combination thereof. Those skilled in the art should It is understood that some or all of the functionality of some or all of the components of the data manipulation device in accordance with embodiments of the present invention may be implemented in practice using a microprocessor or digital signal processor (DSP). The invention can also be implemented as a device or device program (e.g., a computer program and a computer program product) for performing some or all of the methods described herein. Such a program implementing the invention may be stored on a computer readable medium or may be in the form of one or more signals. Such signals may be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

For example, Figure 5 schematically illustrates a block diagram of a computing device for performing the method in accordance with the present invention. The computing device conventionally includes a processor 510 and a computer program product or computer readable medium in the form of a memory 520. The memory 520 may be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read Only Memory), an EPROM, a hard disk, or a ROM. Memory 520 has a storage space 530 that stores program code 531 for performing any of the method steps described above. For example, storage space 530 storing program code may include various program code 531 for implementing various steps in the above methods, respectively. The program code can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, compact disks (CDs), memory cards or floppy disks. Such a computer program product is typically a portable or fixed storage unit such as that shown in FIG. The storage unit may have storage segments, storage spaces, and the like that are similarly arranged to memory 520 in the computing device of FIG. The program code can be compressed, for example, in an appropriate form. Typically, the storage unit comprises computer readable code 531 ' for performing the steps of the method according to the invention, ie code that can be read by a processor such as 510, which when executed by the computing device causes the calculation The device performs the various steps in the methods described above.

It is to be noted that the above-described embodiments are illustrative of the invention and are not intended to be limiting, and that the invention may be devised without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as a limitation. The word 'comprising' does not exclude the presence of the elements or steps that are not recited in the claims. The word "a" or "an" The invention can be implemented by means of hardware comprising several distinct elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means can be embodied by the same hardware item. The use of the words first, second, and third does not indicate any order. These words can be interpreted as names.

In addition, it should be noted that the language used in the specification has been selected for the purpose of readability and teaching, and is not intended to be construed or limited. Therefore, many modifications and changes will be apparent to those skilled in the art without departing from the scope of the invention. The disclosure of the present invention is intended to be illustrative, and not restrictive, and the scope of the invention is defined by the appended claims.

The present invention is applicable to computer systems/servers that can operate with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and/or configurations suitable for use with computer systems/servers include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, based on Microprocessor systems, set-top boxes, programmable consumer electronics, networked personal computers, small computer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above, and the like.

The computer system/server can be described in the general context of computer system executable instructions (such as program modules) being executed by a computer system. Generally, program modules may include routines, programs, target programs, components, logic, data structures, and the like that perform particular tasks or implement particular abstract data types. The computer system/server can be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communication network. In a distributed cloud computing environment, program modules may be located on a local or remote computing system storage medium including storage devices.

"an embodiment," or "an embodiment," or "an embodiment," In addition, it is noted that the phrase "in one embodiment" is not necessarily referring to the same embodiment.

Claims

A method of data manipulation, comprising:

Receiving a read request for data;

Finding the write time of the record that writes the data to the main library;

The read request is sent to the main library or the slave library to read the data according to the write time.
The method of claim 1 further comprising:

Determining a length of time from the writing time to the current time;

Determining whether the length of time is greater than a preset threshold;

The sending the read request to the main library or the slave library to read the data according to the writing moment specifically includes:

The read request is sent to the main library or the slave library to read the data according to the above judgment result.
The method of claim 2, wherein

If the length of time is greater than a preset threshold, sending the read request to the slave library to read the data;

Otherwise, the read request is sent to the main library to read the data.
The method of any of claims 1-3, wherein before receiving the read request for the data, the method further comprises:

Receiving a write request for the data;

Sending the write request to the main library for writing;

The write time of the data is recorded.
The method of any of claims 1-4, wherein the preset threshold is set by:

Determining, by the main library, the data to be synchronized to the average duration of the slave library, and setting the average duration to the preset threshold.
The method of any of claims 1-4, wherein the preset threshold is set by:

Determining that the main library synchronizes data to a maximum duration of the slave library, and setting the maximum duration to the preset threshold.
A data operation device, comprising: receiving module, searching a module, a time difference module, and a judgment processing module, wherein

The receiving module is configured to receive a read request for data;

The searching module is configured to search for a write moment of writing the data to the main library;

The time module is configured to determine a length of time from the writing time to the current time;

The determining processing module is configured to determine whether the length of time is greater than a preset threshold, and if yes, send the read request to the slave library to read the data; otherwise, send the read request to the main library to read The data.
The apparatus of claim 7, wherein the receiving module is further for receiving a write request for the data.
The apparatus of claim 8, wherein the determination processing module is further configured to send the received write request for the data to a main library for writing.
The apparatus of claim 9, wherein the apparatus further comprises: a recording module for recording a write time when the data is written to the main library.
The apparatus of any of claims 7-10, wherein the preset threshold is set by:

Determining, by the main library, the data to be synchronized to the average duration of the slave library, and setting the average duration to the preset threshold.
The apparatus of any of claims 7-10, wherein the preset threshold is set by:

Determining that the main library synchronizes data to a maximum duration of the slave library, and setting the maximum duration to the preset threshold.
A computer program comprising computer readable code, when the computer readable code is run on a computing device, causing the computing device to perform the data manipulation method of any of claims 1-6.
A computer readable medium storing the computer program of claim 13.