CN104850618B - A kind of system and method that ordered data is provided - Google Patents
A kind of system and method that ordered data is provided Download PDFInfo
- Publication number
- CN104850618B CN104850618B CN201510250580.4A CN201510250580A CN104850618B CN 104850618 B CN104850618 B CN 104850618B CN 201510250580 A CN201510250580 A CN 201510250580A CN 104850618 B CN104850618 B CN 104850618B
- Authority
- CN
- China
- Prior art keywords
- connection
- data
- core buffer
- record
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Fuzzy Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention relates to a kind of system and method for providing ordered data, which includes：Client, for generating the inquiry request with ordering requirements；Multiple data sources, each of the multiple data source generate the data record of sequence according to corresponding inquiry request and are transferred to corresponding connection；Coupling part, for managing and being assigned to the connection of each data source；Sorting mechanism, for providing the data record of global sequence to client by the data record of partial ordering of the corresponding connection processing from each data source.Sorting mechanism controls the use of memory using the boundary marker with the corresponding buffering area of each connection, and by connecting the global sequence for doing heapsort to complete data to each repeatedly.
Description
Technical field
System and method the present invention relates to ordered data is obtained from multi-data source.
Background technology
With the continuous development of internet, in sharp increase, conventional individual database is being handled the data volume on internet
Apparent bottleneck is faced during large-scale data, the realization side of distributed data base is all set about studying by major Internet company
Case.Include two classes in the implementation of distributed data base, one kind is the solution of client, introduces a new client
End carries out data burst processing, and another kind of is to introduce database middleware, complete by the middleware to the burst processing of data
Into application program only needs to access the database middleware, and entire access process is almost with primary database is accessed
The same.
It is comparatively relatively more for the solution of the middleware of MySQL between in the database in the solution of part,
The problem of needing one solved extremely important when MySQL middlewares are realized is exactly from multiple MySQL case-based systems numbers
According to when how these data are carried out summarizing sequence.
Some database middlewares increased income such as Cobar does not support the data sorting for being related to multiple MySQL examples.Also one
A little middlewares support sequence, but are all to get all data being related to from each MySQL examples when realization,
Then in the memory of machine where these data being placed on middleware or fall on disk, then it be ranked up again,
Client is issued again after drained.
Machine itself limited memory where middleware, is not just handled when data volume is big, if necessary to which data are gone to
It is then realized on disk and understands more complicated and performance meeting degradation, and if necessary to which data to be specially aggregated into other machine
On, then it can increase network I/O, can equally reduce performance.
The content of the invention
System and method the object of the present invention is to provide ordered data is effectively obtained from multi-data source.
According to an aspect of the present invention, a kind of system for providing ordered data is provided, including：Client, for generating
Inquiry request with ordering requirements；Multiple data sources, each of the multiple data source inquire about son request production according to corresponding
The data record of raw sequence is simultaneously transferred to corresponding connection；Coupling part, for managing and being assigned to the connection of each data source；Row
Order mechanism, for performing following steps：The inquiry request with ordering requirements is received from client, and parses the inquiry request
It is asked with the subquery generated corresponding to each data source；The corresponding connection of each data source is obtained, is established corresponding with each connection slow
Rush area and empty and labeled as less than, and the request of each subquery of generation by corresponding linkup transmit to corresponding data
Source, wherein the data source returns to orderly data record by accordingly connecting；According to predetermined rule come poll respectively connection with
Determine which is connected with data and can be read, wherein, when determining that one is connected with that data can be read and the corresponding memory of the connection delays
Rush area less than, then read in the connection all readable data records and store into the corresponding core buffer of the connection, when
The buffer tag to have expired when data volume in the buffering area is more than predetermined threshold, and works as and determine to need through the connection
The data of reading are all read to be terminated the linkage flag to read when finishing；Perform the heapsort of all connections；When heap top
When connecting corresponding core buffer non-empty, first record of taking-up is sent to client from the core buffer, also, again
First record taking-up in the corresponding core buffer of connection on heap top is simultaneously sent to client by the heapsort of multiple all connections
End, until the corresponding core buffer of the connection on heap top for sky, wherein taking out first from the corresponding core buffer in heap top
A record be sent to after client to judge the core buffer whether less than, when less than when, cancel the core buffer
Full scale is remembered；Terminate when the corresponding core buffer of connection on heap top is labeled reading for empty and heap top the connection, then mark
Remember that the connection processing finishes, otherwise continue the poll；When all connections have all been disposed, then end mark is sent out
Client is given, otherwise continues the poll.Wherein, the connection is according to following regular value：If the connection is marked as
It is disposed, it is infinitely great to the ascending sort connection value, it is infinitely small to the value of the descending sort connection；If the connection is not
It is marked as being disposed and the corresponding buffering area of the connection is sky, it is infinitely small to the value of the ascending sort connection, to descending
Sort the connection value it is infinitely great；If not above-mentioned two situations, the value of the connection is in the corresponding buffering area of the connection
The sort field value of first record.
According to another aspect of the present invention, a kind of method that ordered data is provided is provided, including step：It is received from client
Inquiry request with ordering requirements, and the subquery for parsing the inquiry request to generate corresponding to each data source is asked；It obtains
It takes the corresponding connection of each data source, establish and the corresponding buffering area of each connection and empties and be labeled as less than and generation
Each subquery request gives corresponding data source by corresponding linkup transmit, wherein the data source is returned in order by accordingly connecting
Data record；Carrying out poll according to predetermined rule, respectively connection can be read with determining which is connected with data, wherein, when definite one
It is connected with that data can be read and the corresponding core buffer of the connection is less than then reading all readable data notes in the connection
It records and stores in the corresponding core buffer of the connection, this is buffered when the data volume in the buffering area is more than predetermined threshold
Area is reading the linkage flag labeled as expire, and when determining that the data read by the connection is needed, which all to read, to be finished
Take end；Perform the heapsort of all connections；When the corresponding core buffer non-empty of the connection on heap top, from the core buffer
It is middle to take out first record and is sent to client, also, repeat the heapsort of all connections and the connection on heap top it is corresponding in
Deposit first in buffering area record take out be sent to client, until the corresponding core buffer of the connection on heap top for sky,
In to judge the core buffer taking out first record from the corresponding core buffer in heap top and be sent to after client
Whether less than, when less than when, cancel the core buffer full scale note；When the corresponding core buffer of the connection on heap top is sky
And the connection on heap top is labeled reading and terminates, then the connection processing is marked to finish, and otherwise continues the poll；When all
Connection be all disposed, then end mark is sent to client, otherwise continues the poll.Wherein, the connection
According to following regular value：If the connection is marked as being disposed, to ascending sort, the connection value is infinitely great, to descending
Sort the connection value it is infinitely small；If the connection is not labeled as being disposed and the corresponding buffering area of the connection is sky,
It is infinitely small to the value of the ascending sort connection, it is infinitely great to the value of the descending sort connection；If not above two feelings
Condition, the value of the connection are the sort field values of first record in the corresponding buffering area of the connection.
Description of the drawings
Describe the embodiment of the present invention in detail below with reference to the accompanying drawings, wherein：
The system based on multi-data source offer ordered data that Fig. 1 is the present invention forms and the schematic diagram of data sorting；
Fig. 2 is the flow chart of the method that ordered data is provided based on multi-data source of the present invention.
Specific embodiment
According to an embodiment of the invention, a kind of system that ordered data is provided is provided, including data source components, connecting portion
Point, sorting mechanism part and client.Each data source provides the data of local order by coupling part to sorting mechanism, through row
After order mechanism processing, the data of global orderly are finally provided in client.
Each data source of data source components is, for example, MySQL examples.
Coupling part is used to managing and being assigned to the connection of each data source, each data source is accessed with to realize sorting mechanism
Pipeline so that sorting mechanism can obtain data from each data source.
The data that sorting mechanism is obtained from each data source are that order obtains according to the rules, i.e., the number that each data source provides
According to itself being local order, such as all it is incremental order.Sorting mechanism needs to carry out the data from different data sources complete
Office's sequence.
Sorting mechanism is, for example, database middleware.The database middleware is to there is a TCP connection between a MySQL.
Data on MySQL examples are constantly sent to middleware by the connection.
Respectively there are one buffering areas for the transmitting terminal and receiving terminal of TCP connection.MySQL examples are sending data to TCP connection
When, in the buffering area for the transmitting terminal that TCP connection can be placed data into first, it is subsequently sent to opposite receiving terminal, receiving terminal
Can first data be stored in the buffering area of receiving terminal of the TCP connection later by receiving data, these operations are all by operating
What system was completed in itself.
Middleware receives the data on MySQL examples by TCP connection, is exactly the reception in middleware reading TCP connection
Content in the buffering area at end.
When the buffering area of receiving terminal is less than the data on MySQL examples can just continue through TCP connection and send over, no
Then the transmission of the data on MySQL examples will be blocked.
According to an embodiment of the invention, the schematic diagram of entire sequence processing is as shown in Figure 1, also there are four portions on corresponding figure
Point：
MySQL example sets 101；
TCP connection 102 between middleware and MySQL examples；
Middleware sorting mechanism 103；And
Client 104.
According to an embodiment of the invention, a kind of system composition, wherein data source components MySQL example sets is shown in FIG. 1
MySQL examples there are six 101.Sequence middleware 103 there is one to be connected with each MySQL examples, share six connect A, B, C,
D, E and F.
By way of example it is assumed that there are some data, these data on each MySQL examples to transmit discharge by coupling part
Sequence middleware.The connection data to be transmitted of A have 0,6 and 12, and the connection data to be transmitted of B have 1,7 and 13, and connection C to be transmitted
Data have 8 and 20, and the connection data to be transmitted of D have 9,15 and 21, and the connection data to be transmitted of E have 4 and 10, and connection F will be transmitted
Data have 5,11 and 17.The data to be transmitted all are ranked in every connection, it is assumed that are sequentially from small to large.
The method according to the present invention for providing ordered data includes step as described below.
Sorting mechanism receives the inquiry request with ordering requirements from client, and parses the inquiry request to generate pair
It should be asked in the subquery of each data source.
Then, sorting mechanism obtains the connection corresponding to above-mentioned each data source, establish with the corresponding buffering area of each connection simultaneously
Empty and labeled as less than, and the request of each subquery of generation by corresponding linkup transmit to corresponding data source.It is described
Each data source responds corresponding subquery and retrieves corresponding data record in its database, and passes through its corresponding connection and return
Return the data record to sort on request.
Sorting mechanism 103 carrys out poll according to predetermined rule and respectively connects to determine which is connected with data and can be read.Work as row
Order mechanism determines a certain to be connected with that data can be read and the corresponding core buffer of the connection is less than then reading institute in the connection
There is readable data record and store into the corresponding core buffer of the connection.When the data volume in the buffering area is more than predetermined
It is to have expired the buffer tag during threshold value.Then, sorting mechanism determines to need whether the data read by the connection have been located
Reason finishes, and the linkage flag is terminated to read when the data that need to read by the connection all read and finish.
Then, sorting mechanism 103 performs the heapsort of all connections.The ranking value of connection is determined according to following rule：Such as
The fruit connection is marked as being disposed, infinitely great to the ascending sort connection value, to the value of the descending sort connection without
It limits small；If the connection is not labeled as being disposed and the corresponding buffering area of the connection is sky, to the ascending sort connection
Value is infinitely small, the value infinity to the descending sort connection；If not above-mentioned two situations, the value of the connection is the company
Connect the sort field value of first record in corresponding buffering area.
After a heapsort, determine whether the corresponding buffering area of connection positioned at heap top is empty.When the connection on heap top
During corresponding core buffer non-empty, first record of taking-up is sent to client 104 from the core buffer, then repeats
It the heapsort of all connections and first record in the corresponding core buffer of connection on heap top is taken out is sent to client
The processing at end, until the corresponding core buffer of the connection on heap top for sky, wherein being taken from the corresponding core buffer in heap top
Go out first record be sent to after client to judge the core buffer whether less than, when less than when, cancel the memory and delay
Rush the note of full scale in area.
Terminate when the corresponding core buffer of connection on heap top is labeled reading for empty and heap top the connection, then mark
This is connected as being disposed, and otherwise continues the poll, to attempt to read the data in some connection.
When all connections have all been disposed, then end mark is sent to client represents that sequencer procedure terminates,
Otherwise the poll is continued, to attempt to read the data in some connection.
The detailed process of the method for present invention offer ordered data is described with reference to Fig. 2.
The middleware that sorts receives the inquiry request with ordering requirements from client, is, for example, SQL request.
The inquiry request is received in step S2001 sequence middlewares and parses the inquiry request.If the inquiry request relates to
And multiple MySQL examples, then it splits to the inquiry request, the subquery generated corresponding to each MySQL examples is asked.
If inquiry request only relates to single MySQL examples, decomposition query request is not required, middleware only needs to receive
The data that the single MySQL examples send over.
Step S2003 sequence middleware obtain the connection in relation to each MySQL examples, then step S2005 establish and
These connect corresponding buffering areas for storing data record, and these buffer empties and labeled as less than.
It is connected in step S2007 sequence middlewares by these and sub- SQL request of generation is sent to corresponding MySQL realities
Example.
Each MySQL examples receive corresponding sub- SQL request, retrieve data according to sub- SQL request, and accordingly data are done
Then ordered data is returned to sequence middleware by sequence by corresponding connection.
According to predetermined rule carrying out poll in step S2009 sequence middlewares, which is connected with data and can read, if
It is readable not to be connected with data, then continues waiting for, and whether to be connected with data readable for poll again.It for example, can be according to setting
Time interval come periodically to it is each connection do poll, with judge connect whether there are data can be read.
How this specific implementation that judges dependent on connection is performed.For example, in the connection realized in TCP layer, TCP
Respectively there are one buffering areas for the transmitting terminal and receiving terminal of connection.Database such as MySQL examples to TCP connection send data when
It waits, in the buffering area for the transmitting terminal that TCP connection can be placed data into first, is subsequently sent to opposite receiving terminal, receives termination
Can first data be stored in the buffering area of receiving terminal of the TCP connection later by receiving data, these operations are all by accordingly grasping
Make system completion.
If judging can therefrom read data in the presence of connection in step S2009, the connection is judged in step S2011
Whether corresponding core buffer has expired.
Whether buffering area here has been full of whether the data volume referred in buffering area has been over predetermined threshold value.It is reading
When taking the data in a connection, it should which the data that can be read during this is connected all read and finish.Therefore a connection is being read
Data during, even if the corresponding buffering area of the connection has been expired, also the digital independent that can be read is finished and is stored in
In the buffering area (such as buffering area actually useful to preserve the space of " spilling part ").
When step S2011 judge buffering area less than the middleware that then sorts is all in step S2012 reads the connection can
Whether the data record of reading is simultaneously stored into the corresponding core buffer of the connection, then judge buffering area in step S2013
It is full, the buffer tag it is to have expired in step S2014 when expire constantly.
The present invention controls the use of memory by this mode of " full " mark of the setting buffers to connection, realizes
The balance of efficiency and availability.
Next, whether the data read by the connection are needed according to the judgement of the data of reading in step S2015
It all reads and finishes, i.e., whether the data that the corresponding data source of the connection (MySQL examples) should provide all have been read
It finishes.The judgement can be realized according to techniques known in the art.It is finished if read, in step S2017 this
Linkage flag terminates to read.
Next, heapsort is done to all connections in step S2019 or all connections are carried out at a heap
Reason.
Heapization processing is the key that improving performance in the present invention, because if simply using the thinking of traditional conflation algorithm
To handle entire sequencer procedure multiple connected Data duplications can be caused to compare.On the thin of " heapization processing " of the invention
Section, later special description.
After carrying out heap processing to all connections, the corresponding memory of connection in heap top is judged in step S2021
Whether buffering area is empty.
If sky, then the data that explanation also has connection do not send over, and rejudge the data of which connection again
It is readable.
If there is data record in the corresponding core buffer of the connection on heap top, first data record in buffering area
It is exactly not yet to export record minimum in the data record to client (this refers to the situation of ascending order, if descending sort, then
It is the largest record), first record is taken out from the core buffer of the connection on heap top in step S2029 and is sent to client
End." take out record " mentioned here refers to after the record is read from core buffer, the record from core buffer
Middle deletion, in this way, second original record of the core buffer of the connection on heap top becomes first record, it is meant that should
" value " of connection is changed.Then judge whether the core buffer of the connection is full again in step S2030, when
It is discontented, then cancel the note of full scale of the connection buffering area in step S2031.Then, then go to step S2019 all connections are done
Whether heapization processing, the core buffer then proceeded to depending on the connection of heap top have data to decide whether to continue cycling through processing still again
It goes to receive data.
When judging that the corresponding buffering area of connection positioned at heap top is sky in step S2021, then go to step S2022 and judge the company
Whether labeled read is connect to terminate.Terminate if do not read, go to step S2009, otherwise mark the connection in step S2023
It is disposed.
Then, judge whether that all connections have all been disposed in step S2025.
It has all been disposed if all of connection, has then gone to step S2027 and end mark is sent to client.Example
Such as, end mark, which corresponds to, refers to EOF bags in MySQL agreements.
It is connected if also had handle, then turns S2009 to rejudge the data which is connected readable.
According to the present invention, the limit of the buffering area of connection ensure that middleware required memory when handling ordered
It is limited and controllable, and heap part ensure that number of comparisons is that the minimum time complexity for ensuring entirely to sort is O
(nlgn)。
Heapization processing
Conflation algorithm is a kind of known sort algorithm, is had for two or more orderly sequences to be merged into one
The sequence of sequence.Assuming that clooating sequence is from small to large, it is every when n (n is greater than 2 integer) ordered sequences are related to
All alignments once, are got minimum element by secondary needs, then by the minimum element from corresponding orderly sequence
It is removed in row.Assuming that these ordered sequences are to be denoted as 1...k...n respectively, the wherein first element of sequence k is minimum.It is inciting somebody to action
After the first element of sequence k removes, first element of all ordered sequences compares once again, that is, compare n (n-1)/
2.But remaining n-1 ordered sequence compared each other at this time, it is only necessary to by the new first element and n- of sequence k
The first element of 1 ordered sequence is compared, i.e., compares n-1 times again.Actually since remaining n-1 ordered sequence
First element compared, if comparative result before has all had record, only need the new head of sequence k
A element is compared with the minimum element in the first element of remaining n-1 ordered sequence, so optimally
It only needs to compare 1 element that can get next minimum again, it is likely that the new header element of sequence k may be than other
Sequence first element it is all big, then still need again to compare once with remaining n-1 ordered sequence.But if n- before
The comparative result of 1 sequence is all recorded and recorded in the form of heap, then only needs to compare lgn times i.e.
It can.
Robert Freud (Robert W.Floyd) and WILLIAMS-DARLING Ton (J.Williams) were sent out jointly in 1964
Understand famous Heap algorithm (Heap Sort).Heapsort generates a kind of orderly pile structure, is complete binary tree.Heap
It is divided into most raft (or big root heap, great Ding Dui) and most rickle (or rootlet heap, little Ding Dui).In most raft, each node
Value all be not more than its father node value.In most rickle, the value of non-leaf nodes is less than the value of its child nodes.
It is exactly to do heapsort to each connection that heapization processing is done to connection.Heap (the y-bend generated in the heapization processing of the present invention
Tree) in node elements be connection rather than connection in record.
For the in the case of of sorting from small to large, using most rickle, i.e. the element on heap top is minimum, and according to following rule really
Surely the value connected：
If the corresponding buffering area of the connection is empty (not having data), i.e., data are transmitted across from MySQL examples not yet
Come, the value of the connection, which takes, infinitely small (is regarded as the minimum value of system, i.e., the system there can not possibly be numeric ratio this minimum value more
It is small)；
Finished if the data to be transmitted of the connection are all processed, the value of the connection take it is infinitely great (it is believed that
It is the maximum of system, i.e., the system can not possibly have numeric ratio this maximum bigger)；
If not above-mentioned two situations, the value of the connection is the sequence of first record in the corresponding buffering area of the connection
Field value.
In the case of most rickle, if there is connecting, there are no data to come, then the empty connection is bound to be pulled to
Heap top, if there is the data of connection have all been disposed, then the connection can be pulled to the bottommost of heap.It removes
Beyond both of these case, other when first item to be recorded as minimum connection must be located at the heap top of heap.
The heap of the present invention is exactly the first item record for making to meet as the connection of element in heap parent node connection
Value is less than the value of the first item record in child nodes connection.
If it is required that according to sorting from big to small, the present invention can be used most raft, and when relatively adjustment ratio
Compared with function.
Embodiment shown in specification and drawings is only used for explanation and illustration, the scope being not intended to limit the present invention, the present invention
It is defined by the claims.
Claims (8)
1. a kind of method that ordered data is provided, including step：
The inquiry request with ordering requirements is received from client, and parses the inquiry request and corresponds to each data source to generate
Subquery request；
It obtains the corresponding connection of each data source, establish and the corresponding buffering area of each connection and empties and be labeled as less than and handle
Each subquery generated is asked through corresponding linkup transmit to corresponding data source, wherein the data source is returned by accordingly connecting
Return orderly data record；
According to predetermined rule come poll respectively connection to determine which is connected with data and can be read, wherein, when determining that one is connected with
Data can be read and the corresponding core buffer of the connection is less than then reading all readable data records in the connection and deposit
Store up in the corresponding core buffer of the connection, when the data volume in the buffering area be more than predetermined threshold when the buffer tag
To have expired, and the linkage flag is tied to read when determining to need the data read by the connection all to read to finish
Beam；
Perform the heapsort of all connections；
When the corresponding core buffer non-empty of the connection on heap top, first record of taking-up is sent to visitor from the core buffer
Family end, also, repeat the heapsort of all connections and first record in the corresponding core buffer of connection on heap top is taken
Go out to be sent to client, until the corresponding core buffer of the connection on heap top is sky, wherein from the corresponding memory buffer in heap top
Taken out in area first record be sent to after client to judge the core buffer whether less than, when less than when, cancelling should
The note of full scale of core buffer；
Terminate when the corresponding core buffer of connection on heap top is labeled reading for empty and heap top the connection, then mark the company
It connects and is disposed, otherwise continue the poll；
When all connections have all been disposed, then end mark are sent to client, otherwise continue the poll,
Wherein, the connection is according to following regular value：
It is infinitely great to the ascending sort connection value if the connection is marked as being disposed, to the descending sort connection
Value is infinitely small；
If the connection is not labeled as being disposed and the corresponding buffering area of the connection is sky, to ascending sort, the connection takes
It is worth infinitely small, the value infinity to the descending sort connection；
If not above-mentioned two situations, the value of the connection is the sort field of first record in the corresponding buffering area of the connection
Value.
2. according to the method described in claim 1, the data source is MySQL examples.
3. according to the method described in claim 1, wherein described carry out poll respectively to connect to be according to setting according to predetermined rule
Time interval come periodically to it is each connection do poll.
4. according to the method described in claim 1, the connection is the connection realized in TCP layer.
5. a kind of system that ordered data is provided, including：
Client, for generating the inquiry request with ordering requirements；
Multiple data sources, each data record and biography that sequence is generated according to corresponding inquiry request of the multiple data source
It is delivered to corresponding connection；
Coupling part, for managing and being assigned to the connection of each data source；
Sorting mechanism, for performing following steps：
The inquiry request with ordering requirements is received from client, and parses the inquiry request and corresponds to each data source to generate
Subquery request；
It obtains the corresponding connection of each data source, establish and the corresponding buffering area of each connection and empties and be labeled as less than and handle
Each subquery generated is asked through corresponding linkup transmit to corresponding data source, wherein the data source is returned by accordingly connecting
Return orderly data record；
According to predetermined rule come poll respectively connection to determine which is connected with data and can be read, wherein, when determining that one is connected with
Data can be read and the corresponding core buffer of the connection is less than then reading all readable data records in the connection and deposit
Store up in the corresponding core buffer of the connection, when the data volume in the buffering area be more than predetermined threshold when the buffer tag
To have expired, and the linkage flag is tied to read when determining to need the data read by the connection all to read to finish
Beam；
Perform the heapsort of all connections；
When the corresponding core buffer non-empty of the connection on heap top, first record of taking-up is sent to visitor from the core buffer
Family end, also, repeat the heapsort of all connections and first record in the corresponding core buffer of connection on heap top is taken
Go out to be sent to client, until the corresponding core buffer of the connection on heap top is sky, wherein from the corresponding memory buffer in heap top
Taken out in area first record be sent to after client to judge the core buffer whether less than, when less than when, cancelling should
The note of full scale of core buffer；
Terminate when the corresponding core buffer of connection on heap top is labeled reading for empty and heap top the connection, then mark the company
It connects and is disposed, otherwise continue the poll；
When all connections have all been disposed, then end mark are sent to client, otherwise continue the poll,
Wherein, the connection is according to following regular value：
It is infinitely great to the ascending sort connection value if the connection is marked as being disposed, to the descending sort connection
Value is infinitely small；
If the connection is not labeled as being disposed and the corresponding buffering area of the connection is sky, to ascending sort, the connection takes
It is worth infinitely small, the value infinity to the descending sort connection；
If not above-mentioned two situations, the value of the connection is the sort field of first record in the corresponding buffering area of the connection
Value.
6. system according to claim 5, the data source is MySQL examples.
7. system according to claim 5, wherein described carry out poll respectively to connect to be according to setting according to predetermined rule
Time interval come periodically to it is each connection do poll.
8. system according to claim 5, the connection is the connection realized in TCP layer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510250580.4A CN104850618B (en) | 2015-05-18 | 2015-05-18 | A kind of system and method that ordered data is provided |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510250580.4A CN104850618B (en) | 2015-05-18 | 2015-05-18 | A kind of system and method that ordered data is provided |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104850618A CN104850618A (en) | 2015-08-19 |
| CN104850618B true CN104850618B (en) | 2018-06-01 |
Family
ID=53850262
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510250580.4A Active CN104850618B (en) | 2015-05-18 | 2015-05-18 | A kind of system and method that ordered data is provided |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104850618B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101965522B1 (en) | 2014-10-20 | 2019-04-03 | 아브 이니티오 테크놀로지 엘엘시 | Specifying and applying rules to data |
| CN108460127B (en) * | 2018-02-28 | 2022-01-28 | 泰康保险集团股份有限公司 | Method, device and equipment for acquiring ordered data |
| CN109857366B (en) * | 2019-02-20 | 2021-06-22 | 武汉轻工大学 | Insertion ordering method, system, equipment and storage medium based on external memory |
| CN110377642B (en) * | 2019-07-24 | 2020-06-02 | 杭州太尼科技有限公司 | Device for rapidly acquiring ordered sequence data |
| CN111914125A (en) * | 2020-07-16 | 2020-11-10 | 北京爱芯科技有限公司 | Data processing method and device, storage medium and electronic equipment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102968496A (en) * | 2012-12-04 | 2013-03-13 | 天津神舟通用数据技术有限公司 | Parallel sequencing method based on task derivation and double buffering mechanism |
| CN103116655A (en) * | 2013-03-06 | 2013-05-22 | 亿赞普(北京)科技有限公司 | Clustered data query method, client side and system |
| CN103399944A (en) * | 2013-08-14 | 2013-11-20 | 曙光信息产业(北京)有限公司 | Implementation method and implementation device for data duplication elimination query |
| CN104111936A (en) * | 2013-04-18 | 2014-10-22 | 阿里巴巴集团控股有限公司 | Method and system for querying data |
| CN104363277A (en) * | 2014-11-13 | 2015-02-18 | 上海交通大学 | Allocation management system and management method for bandwidth resources in cloud game distributed system |
| CN104601732A (en) * | 2015-02-12 | 2015-05-06 | 北京金和软件股份有限公司 | Method for merging multichannel data quickly |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6446128B1 (en) * | 1997-12-01 | 2002-09-03 | Netselector, Inc. | Site access via intervening control layer |
-
2015
- 2015-05-18 CN CN201510250580.4A patent/CN104850618B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102968496A (en) * | 2012-12-04 | 2013-03-13 | 天津神舟通用数据技术有限公司 | Parallel sequencing method based on task derivation and double buffering mechanism |
| CN103116655A (en) * | 2013-03-06 | 2013-05-22 | 亿赞普(北京)科技有限公司 | Clustered data query method, client side and system |
| CN104111936A (en) * | 2013-04-18 | 2014-10-22 | 阿里巴巴集团控股有限公司 | Method and system for querying data |
| CN103399944A (en) * | 2013-08-14 | 2013-11-20 | 曙光信息产业(北京)有限公司 | Implementation method and implementation device for data duplication elimination query |
| CN104363277A (en) * | 2014-11-13 | 2015-02-18 | 上海交通大学 | Allocation management system and management method for bandwidth resources in cloud game distributed system |
| CN104601732A (en) * | 2015-02-12 | 2015-05-06 | 北京金和软件股份有限公司 | Method for merging multichannel data quickly |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104850618A (en) | 2015-08-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104850618B (en) | A kind of system and method that ordered data is provided | |
| CN1233801A (en) | Message broker apparatus, method and computer program product | |
| US6978458B1 (en) | Distributing data items to corresponding buckets for use in parallel operations | |
| CN101547064B (en) | Interleaving and rate matching and de-interleaving and rate de-matching methods | |
| CN108924187B (en) | Task processing method and device based on machine learning and terminal equipment | |
| TW200833010A (en) | System and method for optimizing changes of data sets | |
| CN109766389B (en) | Block chain light client verification query method based on bitmap index | |
| CN101162471B (en) | Method and device for insert sorting | |
| CN105264488B (en) | For using array to merging the method and system of ordered list | |
| US20160337442A1 (en) | Scheduled network communication for efficient re-partitioning of data | |
| CN107567621A (en) | For performing the method, system and computer program product of numeric search | |
| CN105740264A (en) | Distributed XML database sorting method and apparatus | |
| CN107451091A (en) | High speed information interactive system based on FPGA CPU mixed architectures | |
| CN108200112A (en) | Distributed storage method and system | |
| CN1829200A (en) | Systems and methods for implementing counters in a network processor | |
| EP0814582A2 (en) | Data routing | |
| US20020065793A1 (en) | Sorting system and method executed by plural computers for sorting and distributing data to selected output nodes | |
| CN103309940A (en) | Method for sorting out-of-order data streams | |
| CN110428140B (en) | Multi-equipment process scheduling method for process sequence selection | |
| US20110258205A1 (en) | Bit string data sorting apparatus, sorting method, and program | |
| US20150350039A1 (en) | Deep packet inspection (dpi) of network packets for keywords of a vocabulary | |
| CN117176638A (en) | Routing path determining method and related components | |
| CN109491807A (en) | Data exchange method, device and system | |
| CN100499564C (en) | Packet processing engine | |
| US20160055112A1 (en) | Return available ppi credits command |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| EXSB | Decision made by sipo to initiate substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20191128 Address after: 100176 room 222, 2f, building C, No. 18, Kechuang 11th Street, Beijing Economic and Technological Development Zone, Daxing District, Beijing Patentee after: Beijing Jingdong three hundred and sixty degree e-commerce Co., Ltd. Address before: 100080 Beijing city Haidian District xingshikou Road No. 65 west Shan Creative Park District 11C four floor East West 1-4 layer 1-4 layer Co-patentee before: Beijing Jingdong Century Commerce Co., Ltd. Patentee before: Beijing Jingdong Shangke Information Technology Co., Ltd. |