CN108829735B - Synchronization method, device, server and storage medium for parallel execution plan - Google Patents
Synchronization method, device, server and storage medium for parallel execution plan Download PDFInfo
- Publication number
- CN108829735B CN108829735B CN201810488211.2A CN201810488211A CN108829735B CN 108829735 B CN108829735 B CN 108829735B CN 201810488211 A CN201810488211 A CN 201810488211A CN 108829735 B CN108829735 B CN 108829735B
- Authority
- CN
- China
- Prior art keywords
- data
- sending
- receiving end
- receiving
- parallel execution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a synchronization method, a synchronization device, a server and a storage medium for parallel execution plans. The method comprises the following steps: when a parallel operator node of a parallel execution plan is a sending end and sends data to a receiving end, recording the times of sending the data; when the number of times of sending data reaches the set number of times, sending a detection message to a receiving end; and waiting for the receiving end to return a response message of the detection message, and after receiving the response message, re-executing the data sending and recording the times of sending the data and the operations of sending the detection message and receiving the response message when the times of sending the data reach the set times until the data is sent completely, wherein the response message is sent when the data amount accumulated by the receiving end does not exceed the set amount. The invention can control the flow of the sending data, thereby reducing the memory or disk space occupied by the data in one site or a plurality of sites and improving the overall performance.
Description
Technical Field
The embodiment of the invention relates to a database technology, in particular to a synchronization method, a synchronization device, a server and a storage medium for parallel execution plans.
Background
In a system that supports parallel queries or distributed database, execution plans, referred to as parallel execution plans, may be executed in parallel at various sites. The parallel execution plan is based on a common execution plan, and parallel operator nodes with communication functions are added, wherein the parallel operator nodes comprise collection (gather), distribution (distribute), divergence (scatter) and the like.
The parallel execution plan is sent to each site for parallel execution, and the parallel operators in the parallel execution plan can control the movement and synchronization of data among the sites, and finally the data are gathered to the main site for query and uploaded to the client.
Due to the fact that data volume of each site is different, differences of site system resources or the characteristics of a parallel execution plan, the phenomenon that data transmission among the sites is asynchronous may exist in the execution process of the parallel execution plan, so that a large amount of inquired data are accumulated on one or more sites, more memory or disk space is occupied, and overall performance is reduced.
Disclosure of Invention
In view of this, embodiments of the present invention provide a synchronization method, an apparatus, a server, and a storage medium for parallel execution plans, so as to reduce memory or disk space occupied by data in one site or multiple sites and improve overall performance.
In a first aspect, an embodiment of the present invention provides a synchronization method for parallel execution plans, where the method includes:
when a parallel operator node of a parallel execution plan is a sending end and sends data to a receiving end, recording the times of sending the data;
when the number of times of sending data reaches a set number of times, sending a detection message to the receiving end;
and waiting for the response message of the detection message returned by the receiving end, and after receiving the response message, re-executing the data sending and recording the times of sending the data and the operations of sending the detection message and receiving the response message when the times of sending the data reach the set times until the data is sent completely, wherein the response message is sent when the data amount accumulated by the receiving end does not exceed the set amount.
In a second aspect, an embodiment of the present invention further provides a synchronization method for parallel execution plans, where the method includes:
when a parallel operator node of a parallel execution plan is a receiving end, receiving a detection message sent by a sending end, wherein the detection message is sent by the sending end when the number of times of sending data reaches a set number of times;
determining the data amount accumulated by the receiving end;
and if the data amount accumulated by the receiving end exceeds the set number, sending a response message to the sending end when the receiving end processes the data and the accumulated data amount does not exceed the set number.
In a third aspect, an embodiment of the present invention further provides a synchronization apparatus for parallel execution of plans, where the apparatus includes:
the data sending frequency recording module is used for recording the frequency of sending data when the parallel operator nodes of the parallel execution plan are the sending ends and send the data to the receiving ends;
the detection message sending module is used for sending a detection message to the receiving end when the number of times of sending data reaches a set number of times;
and the control module is used for waiting for the response message of the detection message returned by the receiving end, and after receiving the response message, re-executing the data sending and recording the times of sending the data and the operations of sending the detection message and receiving the response message when the times of sending the data reach the set times until the data is sent completely, wherein the response message is sent when the data amount accumulated by the receiving end does not exceed the set amount.
In a fourth aspect, an embodiment of the present invention further provides a synchronization apparatus for parallel execution of plans, where the apparatus includes:
the system comprises a detection message receiving module, a data sending module and a data sending module, wherein the detection message receiving module is used for receiving a detection message sent by a sending end when a parallel operator node of a parallel execution plan is a receiving end, and the detection message is sent by the sending end when the number of times of sending data reaches a set number of times;
the accumulated data volume determining module is used for determining the data volume accumulated by the receiving end;
and the response message sending module is used for sending a response message to the sending end when the receiving end processes data and the accumulated data amount does not exceed the set number if the data amount accumulated by the receiving end exceeds the set number.
In a fifth aspect, an embodiment of the present invention further provides a database server, where the server includes:
one or more processors;
a storage device for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors implement the method for synchronizing parallel execution plans according to any embodiment of the present invention.
In a sixth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the synchronization method for parallel execution plans according to any embodiment of the present invention.
According to the technical scheme of the embodiment of the invention, when the parallel operator node of the parallel execution plan is a sending end and sends data to a receiving end, the number of times of sending the data is recorded, a detection message is sent to the receiving end when the number of times of sending the data reaches the set number of times, a response message of the detection message returned by the receiving end is waited, and the data is sent to the receiving end after the response message is received, so that the flow of the sent data can be controlled, the phenomenon of accumulation on some nodes caused by unlimited sending of the data is avoided, the memory or disk space occupied by the data in one site or a plurality of sites can be reduced, and the overall performance is improved.
Drawings
FIG. 1 is a flowchart of a method for synchronizing parallel execution plans according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating a parallel execution plan in a synchronization method for parallel execution plans, according to an embodiment of the present invention;
FIG. 3 is a flowchart of a method for synchronizing parallel execution plans according to a second embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a synchronization apparatus for parallel execution of plans according to a third embodiment of the present invention;
FIG. 5 is a schematic structural diagram of a synchronization apparatus for parallel execution of plans according to a fourth embodiment of the present invention;
fig. 6 is a schematic structural diagram of a database server according to a fifth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some but not all of the relevant aspects of the present invention are shown in the drawings.
Example one
Fig. 1 is a flowchart of a synchronization method for parallel execution plans according to an embodiment of the present invention, where this embodiment is applicable to a case of synchronizing data of each site in a parallel execution plan, and the method may be executed by a synchronization apparatus for parallel execution plans, where the apparatus may be implemented by software and/or hardware, and may be generally integrated in a database server. The method specifically comprises the following steps:
and step 110, when the parallel operator node of the parallel execution plan is a sending end and sends data to a receiving end, recording the times of sending the data.
The parallel execution plan refers to an execution plan that can be executed in parallel at each site in a database system supporting parallel queries or distributed database systems. For parallel query, an execution plan is executed in parallel on each thread, and each thread can be called a site; for a distributed database system, execution plans are executed in parallel in databases, one database being referred to as a site. The parallel execution plan is to add a parallel operator (also called mobility operator) node with a communication function on the basis of a common execution plan, wherein the parallel operator includes collection (GATHER), Distribution (DISTRIBUTE), divergence (scanner), and the like, and one parallel execution plan may include one or more parallel operator nodes distributed on different levels of an execution plan tree. One site can communicate with the same parallel operator nodes in other sites in the parallel execution plan through the parallel operator nodes in the parallel execution plan, namely the parallel execution plan is sent to each site to be executed in parallel, and data can be controlled to move and synchronize among the sites through the parallel operator nodes in the parallel execution plan and finally summarized to the main site for query to be uploaded to a client.
When a plurality of sites execute parallel execution plans in parallel, the execution plans of all the sites are identical, and the sites communicate and synchronize by using parallel operator nodes. The communication actions performed by the same parallel operator at different sites may differ. For example, master site a and slave site B execute the parallel execution plan shown in fig. 2, where GATHER is a parallel operator node. The GATHER operator node of the master site a is used to collect data of the GATHER operator node of the slave site B, and the GATHER operator node of the slave site B sends the data to the GATHER operator node of the master site a, where the GATHER operator node of the slave site B is called a sending end and the GATHER operator node of the master site a is called a receiving end. For other parallel operator nodes, such as SCATTER, the SCATTER operator node of the master site is the sender, the SCATTER operator node of the slave site is the receiver, and some parallel operator nodes may send and receive data simultaneously, both the sender and the receiver. The receiving end and the sending end can be in the same database server or different database servers.
The station judges whether the station is a sending end or a receiving end according to a parallel operator in the parallel execution plan, and when the station is the sending end, the initial value of the number of times of sending data is set to be 0, and the sending end sends data to the receiving end once, so that the counting is increased once, and the data volume sent to the receiving end is counted, namely the total number of times of sending data is the data volume sent to the receiving end.
And step 120, when the number of times of sending the data reaches the set number of times, sending a detection message to the receiving end.
The set number is a value within a range of 1-10 ten thousand, which is only an example and is not limited, and specifically, a user can select a proper set number according to needs. If the data size interaction is large, the set times can be set to be smaller; in the case of small data amount interaction, the set number of times can be set to be slightly larger, so that accumulation of more data by one station or a plurality of stations in the parallel execution plan can be avoided.
When the number of times of sending the data reaches the set number of times, sending a detection message to the receiving end, and stopping sending the data to the receiving end, thereby detecting whether the receiving end accumulates more data.
When a parallel operator node of a parallel execution plan is a sending end and sends data to a receiving end, recording the times of sending the data, and the method comprises the following steps:
when a parallel operator node of a parallel execution plan is a sending end, setting a variable, and assigning an initial value of the variable to be 0, wherein the variable is used for recording the times of sending data to a receiving end by the sending end;
when the sending end sends data to the receiving end for one time, the value of the variable is increased by one;
when sending the detection message to the receiving end, the method further comprises:
and clearing the value of the variable.
When the parallel operator node of the parallel execution plan is a sending end, a variable is set for recording the times of sending data to a receiving end by the sending end, and when the sending end sends data to the receiving end once, the value of the variable is increased by one, so that the times of sending the data are counted conveniently.
When the detection message is sent to the receiving end, the data sending to the receiving end is stopped, the data sending in the current round is determined to be finished, the variable value is cleared, the data sending is restarted subsequently, and the data sending times are recorded conveniently.
Wherein the response message is sent by the receiving end when the amount of accumulated data does not exceed a set amount. The set number is a value within a range of 1-10 ten thousand, which is only an example and is not limited, and the specific value can be the same as the set number.
If more data are accumulated on the receiving end, the receiving end does not immediately respond to the detection message, and the receiving end responds when the accumulated data meet the requirement. The sending end sends the detection message, does not send data to the receiving end any more, needs to wait for the receiving end to return the response message of the detection message, and determines that the data of the receiving end is not accumulated when receiving the response message, and can continue to send the data to the receiving end, thereby realizing the current limiting and ensuring the data synchronization of each station.
And after receiving the response message, determining whether the number of times of data transmission reaches the set number of times, and only determining whether the number of times of data retransmission reaches the set number of times after receiving the response message.
And step 140, judging whether the data is sent completely, if not, executing the step 110, and if so, ending the step.
According to the technical scheme of the embodiment, when the parallel operator nodes of the parallel execution plan are the sending end and send data to the receiving end, the times of sending the data are recorded, the detection message is sent to the receiving end when the times of sending the data reach the set times, the response message of the detection message returned by the receiving end is waited, and the data are sent to the receiving end again after the response message is received, so that the flow of the sent data can be controlled, the phenomenon of accumulation on some nodes caused by unlimited sending of the data is avoided, the memory or disk space occupied by the data in one site or a plurality of sites can be reduced, and the overall performance is improved.
Example two
Fig. 3 is a flowchart of a synchronization method for parallel execution plans according to a second embodiment of the present invention, where this embodiment is applicable to a case of synchronizing data of each site in a parallel execution plan, and the method may be executed by a synchronization apparatus for parallel execution plans, where the apparatus may be implemented by software and/or hardware, and may be generally integrated in a database server. The method specifically comprises the following steps:
The parallel execution plan refers to an execution plan that can be executed in parallel at each site in a database system supporting parallel queries or distributed database systems. For parallel query, an execution plan is executed in parallel on each thread, and each thread can be called a site; for a distributed database system, execution plans are executed in parallel in databases, one database being referred to as a site. The parallel execution plan is to add a parallel operator (also called mobility operator) node with a communication function on the basis of a common execution plan, wherein the parallel operator includes collection (GATHER), Distribution (DISTRIBUTE), divergence (scanner), and the like, and one parallel execution plan may include one or more parallel operator nodes distributed on different levels of an execution plan tree. One site can communicate with the same parallel operator nodes in other sites in the parallel execution plan through the parallel operator nodes in the parallel execution plan, namely the parallel execution plan is sent to each site to be executed in parallel, and data can be controlled to move and synchronize among the sites through the parallel operator nodes in the parallel execution plan and finally summarized to the main site for query to be uploaded to a client.
When a plurality of sites execute parallel execution plans in parallel, the execution plans of all the sites are identical, and the sites communicate and synchronize by using parallel operator nodes. The communication actions performed by the same parallel operator at different sites may differ. For example, master site a and slave site B execute the parallel execution plan shown in fig. 2, where GATHER is a parallel operator node. The GATHER operator node of the master site a is used to collect data of the GATHER operator node of the slave site B, and the GATHER operator node of the slave site B sends the data to the GATHER operator node of the master site a, where the GATHER operator node of the slave site B is called a sending end and the GATHER operator node of the master site a is called a receiving end. For other parallel operator nodes, such as SCATTER, the SCATTER operator node of the master site is the sender, the SCATTER operator node of the slave site is the receiver, and some parallel operator nodes may send and receive data simultaneously, both the sender and the receiver. The receiving end and the sending end can be in the same database server or different database servers.
The station judges whether the station is a sending end or a receiving end according to the parallel operator in the parallel execution plan, and can receive the detection message sent by the sending end when the station is the receiving end. The sending end records the times of sending data when sending the data to the receiving end, and sends a detection message to the receiving end when the times of sending the data reach the set times.
The set number is a value within a range of 1-10 ten thousand, which is only an example and is not limited, and specifically, a user can select a proper set number according to needs. If the data size interaction is large, the set times can be set to be smaller; in the case of small data amount interaction, the set number of times can be set to be slightly larger, so that accumulation of more data by one station or a plurality of stations in the parallel execution plan can be avoided.
After receiving the detection message, judging whether the parallel operator node of the parallel execution plan at the receiving end is locally accumulated with data, namely determining the accumulated data quantity.
The set number is a value within a range of 1-10 ten thousand, which is only an example and is not limited, and the specific value may be the same as the set number.
If the local accumulated quantity of the parallel operator nodes serving as the parallel execution plan of the receiving end exceeds the set quantity, a detection mark is set on the parallel operator nodes and used for identifying that the detection message is received, at the moment, the data can be processed in batches, after a batch of data is processed, the residual accumulated data quantity is checked, if the residual accumulated data quantity is larger than the set quantity, after a batch of data is processed, the residual accumulated data quantity is checked again, and if the residual accumulated data quantity is smaller than or equal to the set quantity, a response message is sent to the sending end, and the detection mark on the parallel operator nodes is cleared. And the subsequent sending end can send the data to the receiving end again after receiving the response message.
When the receiving end processes data, the specific amount of data to be processed in a batch is determined by the attribute of the database.
According to the technical scheme, the detection message sent by the sending end is received when the parallel operator nodes of the parallel execution plan are the receiving end, the data quantity accumulated by the receiving end is determined, if the accumulated data quantity exceeds the set quantity, the response message is sent to the sending end when the processed data is received and the accumulated data quantity does not exceed the set quantity, so that the sending end can send the data to the receiving end again, the flow of the sent data can be controlled, the accumulation phenomenon on certain nodes caused by unlimited sending of the data is avoided, the memory or disk space occupied by the data in one site or a plurality of sites can be reduced, and the overall performance is improved.
On the basis of the technical scheme, the method can also optionally comprise the following steps:
and when the data quantity accumulated by the receiving end does not exceed the set quantity, directly sending a response message to the sending end.
And when the receiving end receives the detection message and determines that the accumulated data amount does not exceed the set number, the receiving end directly sends a response message to the sending end, so that the sending end can immediately start sending the next round of data, and the data processing time is saved.
EXAMPLE III
Fig. 4 is a schematic structural diagram of a synchronization apparatus for parallel execution plans according to a third embodiment of the present invention, where this embodiment is applicable to a case of synchronizing data of each station in a parallel execution plan, and the apparatus may execute the synchronization method for parallel execution plans according to the first embodiment of the present invention, and the apparatus may be implemented by software and/or hardware, and may be generally integrated in a database server. As shown in fig. 4, the synchronization apparatus for parallel execution planning according to the present embodiment includes: a data transmission times recording module 310, a detection message transmitting module 320 and a control module 330.
The data sending frequency recording module 310 is configured to record the frequency of sending data when a parallel operator node of a parallel execution plan is a sending end and sends data to a receiving end;
a detection message sending module 320, configured to send a detection message to the receiving end when the number of times of sending data reaches a set number of times;
the control module 330 is configured to wait for the receiving end to return a response message of the detection message, and after receiving the response message, re-execute the above-mentioned data sending and recording the number of times of sending data and the operations of sending the detection message and receiving the response message when the number of times of sending data reaches a set number of times until the data sending is completed, where the response message is sent by the receiving end when the amount of data accumulated by the receiving end does not exceed the set number.
Optionally, the data sending number recording module includes:
the variable setting unit is used for setting a variable and assigning an initial value of the variable to be 0 when a parallel operator node of a parallel execution plan is a sending end, wherein the variable is used for recording the times of sending data to a receiving end by the sending end;
a variable increasing unit, configured to increase a value of the variable by one when the transmitting end transmits data to the receiving end once;
the device further comprises:
and the variable zero clearing module is used for clearing the value of the variable when the detection message is sent to the receiving end.
The synchronization device for parallel execution plans can execute the synchronization method for parallel execution plans provided by the first embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method. For details of the technique not described in detail in this embodiment, reference may be made to a synchronization method for parallel execution plans provided in an embodiment of the present invention.
Example four
Fig. 5 is a schematic structural diagram of a synchronization apparatus for parallel execution plans according to a fourth embodiment of the present invention, where this embodiment is applicable to a case of synchronizing data of each site in a parallel execution plan, and the apparatus may execute the synchronization method for parallel execution plans according to a second embodiment of the present invention, and the apparatus may be implemented by software and/or hardware, and may generally be integrated in a database server. As shown in fig. 5, the synchronization apparatus for parallel execution planning according to the present embodiment includes: a detection message receiving module 410, an accumulated data amount determining module 420, and a response message transmitting module 430.
The detection message receiving module 410 is configured to receive a detection message sent by a sending end when a parallel operator node of a parallel execution plan is a receiving end, where the detection message is sent by the sending end when the number of times of sending data reaches a set number of times;
a data accumulation amount determining module 420, configured to determine an amount of data accumulated at the receiving end;
a response message sending module 430, configured to send a response message to the sender when the receiving end processes data and the amount of accumulated data does not exceed a set amount if the amount of data accumulated by the receiving end exceeds the set amount.
Optionally, the apparatus further comprises:
and the response message direct sending module is used for directly sending the response message to the sending end when the data quantity accumulated by the receiving end does not exceed the set quantity.
The synchronization device for parallel execution plans can execute the synchronization method for parallel execution plans provided by the second embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method. For details of the technique not described in detail in this embodiment, reference may be made to the synchronization method for parallel execution plans provided in the second embodiment of the present invention.
EXAMPLE five
Fig. 6 is a schematic structural diagram of a database server according to a fifth embodiment of the present invention, as shown in fig. 6, the database server includes a processor 510, a memory 520, an input device 530, and an output device 540; the number of the processors 510 in the server may be one or more, and one processor 510 is taken as an example in fig. 6; the processor 510, the memory 520, the input device 530 and the output device 540 in the server may be connected by a bus or other means, and the bus connection is exemplified in fig. 6.
The memory 520 may be used as a computer-readable storage medium for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the synchronization method of the parallel execution plan in the embodiment of the present invention (for example, the data transmission number recording module 310, the detection message transmitting module 320, and the control module 330 in the synchronization apparatus of the parallel execution plan, or the detection message receiving module 410, the stacked data amount determining module 420, and the response message transmitting module 430 in the synchronization apparatus of the parallel execution plan). The processor 510 executes various functional applications of the server and data processing by executing software programs, instructions, and modules stored in the memory 520, that is, implements the synchronization method of the parallel execution plan described above.
The memory 520 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the server, and the like. Further, the memory 520 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, memory 520 may further include memory located remotely from processor 510, which may be connected to a server over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 530 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the server. The output device 540 may include a display device such as a display screen.
EXAMPLE six
An embodiment of the present invention further provides a storage medium containing computer-executable instructions, which when executed by a computer processor, perform a synchronization method for parallel execution planning, the method including:
when a parallel operator node of a parallel execution plan is a sending end and sends data to a receiving end, recording the times of sending the data;
when the number of times of sending data reaches a set number of times, sending a detection message to the receiving end;
and waiting for the response message of the detection message returned by the receiving end, and after receiving the response message, re-executing the data sending and recording the times of sending the data and the operation of sending the detection message when the times of sending the data reach the set times, wherein the response message is sent by the receiving end when the accumulated data amount does not exceed the set amount.
Alternatively, the computer-executable instructions, when executed by a computer processor, are for performing a method of synchronizing a parallel execution plan, the method comprising:
when a parallel operator node of a parallel execution plan is a receiving end, receiving a detection message sent by a sending end, wherein the detection message is sent by the sending end when the number of times of sending data reaches a set number of times;
determining the data amount accumulated by the receiving end;
and if the data amount accumulated by the receiving end exceeds the set number, sending a response message to the sending end when the receiving end processes the data and the accumulated data amount does not exceed the set number.
Of course, the storage medium containing the computer-executable instructions provided by the embodiments of the present invention is not limited to the method operations described above, and may also perform related operations in the synchronization method for parallel execution plans provided by any embodiments of the present invention.
From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly, can also be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.
It should be noted that, in the embodiment of the synchronization apparatus for parallel execution of plans, the included units and modules are only divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A method for synchronizing parallel execution plans, the method comprising:
when a parallel operator node of a parallel execution plan is a sending end and sends data to a receiving end, recording the times of sending the data;
when the number of times of sending data reaches a set number of times, sending a detection message to the receiving end;
waiting for the receiving end to return a response message of the detection message, and after receiving the response message, re-executing the data sending and recording the times of sending the data and the operations of sending the detection message and receiving the response message when the times of sending the data reach the set times until the data sending is finished, wherein the response message is sent when the data amount accumulated by the receiving end does not exceed the set number;
wherein, the set times is determined by the data size: the larger the data volume interaction is, the smaller the set times is; the smaller the data amount interaction is, the larger the setting number is.
2. The method of claim 1, wherein when a parallel operator node of a parallel execution plan is a sender and sends data to a receiver, recording the number of times the data is sent comprises:
when a parallel operator node of a parallel execution plan is a sending end, setting a variable, and assigning an initial value of the variable to be 0, wherein the variable is used for recording the times of sending data to a receiving end by the sending end;
when the sending end sends data to the receiving end for one time, the value of the variable is increased by one;
when sending the detection message to the receiving end, the method further comprises:
and clearing the value of the variable.
3. A method for synchronizing parallel execution plans, the method comprising:
when a parallel operator node of a parallel execution plan is a receiving end, receiving a detection message sent by a sending end, wherein the detection message is sent by the sending end when the number of times of sending data reaches a set number of times;
determining the data amount accumulated by the receiving end;
if the data amount accumulated by the receiving end exceeds the set number, sending a response message to the sending end when the receiving end processes the data and the accumulated data amount does not exceed the set number;
wherein, the set times is determined by the data size: the larger the data volume interaction is, the smaller the set times is; the smaller the data amount interaction is, the larger the setting number is.
4. The method of claim 3, further comprising:
and when the data quantity accumulated by the receiving end does not exceed the set quantity, directly sending a response message to the sending end.
5. A synchronization apparatus for parallel execution of plans, the apparatus comprising:
the data sending frequency recording module is used for recording the frequency of sending data when the parallel operator nodes of the parallel execution plan are the sending ends and send the data to the receiving ends;
the detection message sending module is used for sending a detection message to the receiving end when the number of times of sending data reaches a set number of times;
the control module is used for waiting for the response message of the detection message returned by the receiving end, and after receiving the response message, re-executing the data sending and recording the times of sending the data and the operations of sending the detection message and receiving the response message when the times of sending the data reach the set times until the data sending is finished, wherein the response message is sent when the data amount accumulated by the receiving end does not exceed the set amount;
wherein, the set times is determined by the data size: the larger the data volume interaction is, the fewer the set times are; the smaller the data amount interaction is, the larger the setting number is.
6. The apparatus of claim 5, wherein the data transmission number recording module comprises:
the variable setting unit is used for setting a variable and assigning an initial value of the variable to be 0 when a parallel operator node of a parallel execution plan is a sending end, wherein the variable is used for recording the times of sending data to a receiving end by the sending end;
a variable increasing unit, configured to increase a value of the variable by one when the transmitting end transmits data to the receiving end once;
the device further comprises:
and the variable zero clearing module is used for clearing the value of the variable when the detection message is sent to the receiving end.
7. A synchronization apparatus for parallel execution of plans, the apparatus comprising:
the system comprises a detection message receiving module, a data sending module and a data sending module, wherein the detection message receiving module is used for receiving a detection message sent by a sending end when a parallel operator node of a parallel execution plan is a receiving end, and the detection message is sent by the sending end when the number of times of sending data reaches a set number of times;
the accumulated data volume determining module is used for determining the data volume accumulated by the receiving end;
a response message sending module, configured to send a response message to the sending end when the receiving end processes data and the amount of accumulated data does not exceed a set amount if the amount of data accumulated by the receiving end exceeds the set amount;
wherein, the set times is determined by the data size: the larger the data volume interaction is, the fewer the set times are; the smaller the data amount interaction is, the larger the setting number is.
8. The apparatus of claim 7, further comprising:
and the response message direct sending module is used for directly sending the response message to the sending end when the data quantity accumulated by the receiving end does not exceed the set quantity.
9. A database server, the server comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of synchronization of parallel execution plans of any one of claims 1-2 or 3-4.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out a method for synchronizing parallel execution plans as claimed in any one of claims 1-2 or 3-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810488211.2A CN108829735B (en) | 2018-05-21 | 2018-05-21 | Synchronization method, device, server and storage medium for parallel execution plan |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810488211.2A CN108829735B (en) | 2018-05-21 | 2018-05-21 | Synchronization method, device, server and storage medium for parallel execution plan |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108829735A CN108829735A (en) | 2018-11-16 |
| CN108829735B true CN108829735B (en) | 2021-06-29 |
Family
ID=64148242
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810488211.2A Active CN108829735B (en) | 2018-05-21 | 2018-05-21 | Synchronization method, device, server and storage medium for parallel execution plan |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108829735B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110569257B (en) * | 2019-09-16 | 2022-04-01 | 上海达梦数据库有限公司 | Data processing method, corresponding device, equipment and storage medium |
| CN111753028B (en) * | 2020-07-02 | 2023-08-25 | 上海达梦数据库有限公司 | Data transmission method, device, equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1941954A (en) * | 2005-09-29 | 2007-04-04 | 中兴通讯股份有限公司 | Method for controlling HSDPA flow based on feedback mode |
| CN102571277A (en) * | 2011-12-29 | 2012-07-11 | 中兴通讯股份有限公司 | Method and device for sending serial number detection message |
| CN104639298A (en) * | 2013-11-08 | 2015-05-20 | 腾讯科技（深圳）有限公司 | Data transmission method, device and system |
| CN106302201A (en) * | 2015-05-14 | 2017-01-04 | 华为技术有限公司 | Flow control methods, equipment and system |
| CN107656937A (en) * | 2016-07-26 | 2018-02-02 | 北京京东尚科信息技术有限公司 | Method and apparatus for realizing read-write data consistency |
| CN107688660A (en) * | 2017-09-08 | 2018-02-13 | 上海达梦数据库有限公司 | The execution method and device of parallel executive plan |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102480500A (en) * | 2010-11-26 | 2012-05-30 | 黑龙江亿林网络技术服务有限公司 | Distributed harmful information management system and method |
| CN102081678B (en) * | 2011-03-14 | 2013-07-03 | 华中科技大学 | Method for searching optimal execution plan in database query |
| CN102323946B (en) * | 2011-09-05 | 2013-03-27 | 天津神舟通用数据技术有限公司 | Implementation method for operator reuse in parallel database |
| CN102780593B (en) * | 2012-07-25 | 2015-08-26 | 中兴通讯股份有限公司 | Based on the method for BFD protocol detection link, device and network processing unit |
| US10019478B2 (en) * | 2013-09-05 | 2018-07-10 | Futurewei Technologies, Inc. | Mechanism for optimizing parallel execution of queries on symmetric resources |
| CN108052635A (en) * | 2017-12-20 | 2018-05-18 | 江苏瑞中数据股份有限公司 | A kind of heterogeneous data source unifies conjunctive query method |
-
2018
- 2018-05-21 CN CN201810488211.2A patent/CN108829735B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1941954A (en) * | 2005-09-29 | 2007-04-04 | 中兴通讯股份有限公司 | Method for controlling HSDPA flow based on feedback mode |
| CN102571277A (en) * | 2011-12-29 | 2012-07-11 | 中兴通讯股份有限公司 | Method and device for sending serial number detection message |
| CN104639298A (en) * | 2013-11-08 | 2015-05-20 | 腾讯科技（深圳）有限公司 | Data transmission method, device and system |
| CN106302201A (en) * | 2015-05-14 | 2017-01-04 | 华为技术有限公司 | Flow control methods, equipment and system |
| CN107656937A (en) * | 2016-07-26 | 2018-02-02 | 北京京东尚科信息技术有限公司 | Method and apparatus for realizing read-write data consistency |
| CN107688660A (en) * | 2017-09-08 | 2018-02-13 | 上海达梦数据库有限公司 | The execution method and device of parallel executive plan |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108829735A (en) | 2018-11-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110427299B (en) | Log processing method, related device and system for micro-service system application | |
| CN110995513B (en) | Data sending and receiving method in Internet of things system, internet of things equipment and platform | |
| CN108121511B (en) | Data processing method, device and equipment in distributed edge storage system | |
| CN107301178B (en) | Data query processing method, device and system | |
| CN110413650B (en) | Method, device, equipment and storage medium for processing service data | |
| CN109558065B (en) | Data deleting method and distributed storage system | |
| CN111258978B (en) | Data storage method | |
| US9680719B2 (en) | Communication system, client terminal, and server | |
| CN109167840B (en) | Task pushing method, node autonomous server and edge cache server | |
| CN103312544A (en) | Method, equipment and system for controlling terminals during log file reporting | |
| CA2896865A1 (en) | Method and system for using a recursive event listener on a node in hierarchical data structure | |
| CN111813573B (en) | Communication method of management platform and robot software and related equipment thereof | |
| CN103310087A (en) | Service data statistic analysis method and device | |
| CN104301161A (en) | Computing method, computing device and communication system for business quality index | |
| CN108140035B (en) | Database replication method and device for distributed system | |
| CN108829735B (en) | Synchronization method, device, server and storage medium for parallel execution plan | |
| CN108154024B (en) | Data retrieval method and device and electronic equipment | |
| CN111694793A (en) | Log storage method and device and log query method and device | |
| CN110955460A (en) | Service process starting method and device, electronic equipment and storage medium | |
| CN113596746B (en) | Cluster message processing method and device, electronic equipment and medium | |
| CN104468674A (en) | Data migration method and device | |
| JP7133037B2 (en) | Message processing method, device and system | |
| CN108805741B (en) | Fusion method, device and system of power quality data | |
| CN113901076A (en) | Data synchronization method, device, server and storage medium | |
| CN112541038A (en) | Time series data management method, system, computing device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |