CN108900434A - Data collection distribution method and device - Google Patents
Data collection distribution method and device Download PDFInfo
- Publication number
- CN108900434A CN108900434A CN201810793438.8A CN201810793438A CN108900434A CN 108900434 A CN108900434 A CN 108900434A CN 201810793438 A CN201810793438 A CN 201810793438A CN 108900434 A CN108900434 A CN 108900434A
- Authority
- CN
- China
- Prior art keywords
- data source
- priority
- queue
- data
- divided
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
- H04L47/62—Queue scheduling characterised by scheduling criteria
- H04L47/625—Queue scheduling characterised by scheduling criteria for service slots or service orders
- H04L47/6275—Queue scheduling characterised by scheduling criteria for service slots or service orders based on priority
Abstract
The embodiment of the present invention provides a kind of data collection distribution method and device, belongs to data processing field.This method is by obtaining multiple data sources, then the priority of the multiple data source is obtained, the multiple data source is divided into multiple queues by the priority again based on the multiple data source, each queue is divided into multiple subqueues by the different data classifications based on the multiple data source, it is then based on the priority of each queue and the priority of each subqueue, the multiple data source is sent, therefore, data source is divided into different multiple queues and multiple subqueues in this programme, data source is sent according to the priority of multiple queues and multiple subqueues, which thereby enhance being effectively treated for data source, it avoids because data volume is excessive timely and effectively to be handled data source the problem of.
Description
Technical field
The present invention relates to data processing fields, in particular to a kind of data collection distribution method and device.
Background technique
With cloud computing, the development of big data technology, the source multiplicity of data, data format is not quite similar, with big number
Potential useful information is obtained from mass data according to technology and has obtained the extensive concern of various aspects.In the data system of globalization
The transfer of daily record data is existed simultaneously in system and is summarized, is distributed.
There is not an effective mode to the collection of data and distribution in the prior art, typically passes through lifting means
Performance accelerates the transmission of data, but in the huge situation of data volume, still not can guarantee data can effectively be transmitted, because
How this, effectively collect, distribute the data of separate sources, to guarantee that data are timely and effectively handled, increasingly by weight
Depending on.
Summary of the invention
In view of this, the embodiment of the present invention is designed to provide a kind of data collection distribution method and device, to improve
Above-mentioned technical problem.
In a first aspect, the embodiment of the invention provides a kind of data collection distribution method, the method includes：
Obtain multiple data sources；
Obtain the priority of the multiple data source；
The multiple data source is divided into multiple queues by the priority based on the multiple data source；
Each queue is divided into multiple subqueues by the different data classifications based on the multiple data source；
The priority of priority and each subqueue based on each queue, sends the multiple data source.
Further, the priority of the multiple data source is obtained, including：
Judge whether carry priority tag in the multiple data source；
If so, obtaining the priority of the multiple data source from the multiple data source；
If it is not, being then not carry the data source of the priority tag according to the corresponding priority of preset rules distribution.
Further, the multiple data source is divided into multiple queues by the priority based on the multiple data source, including：
Data source with equal priority is divided into the same queue, to obtain multiple queues.
Further, each queue is divided into multiple subqueues by the different data classifications based on the multiple data source,
Including：
Data source in the same queue is divided into multiple subqueues according to different data types；Or
Data source in the same queue is divided into multiple subqueues according to different Data subjects；Or
Data source in the same queue is divided according to different data source IP for multiple subqueues.
Further, based on the different data classifications of the multiple data source by each queue be divided into multiple subqueues it
Afterwards, the priority of priority and each subqueue based on each queue, it is described before being sent to the multiple data source
Method further includes：
Corresponding cpu resource and memory source are distributed for each subqueue；
Cpu resource and memory source based on each subqueue determine the priority of each subqueue.
Second aspect, the embodiment of the invention provides a kind of data collection diostribution device, described device includes：
Data acquisition module, for obtaining multiple data sources；
Priority obtains module, for obtaining the priority of the multiple data source；
The multiple data source is divided into multiple teams for the priority based on the multiple data source by queue division module
Column；
Each queue is divided into more by subqueue division module for the different data classifications based on the multiple data source
A subqueue；
Data transmission blocks, for the priority of priority and each subqueue based on each queue, to the multiple
Data source is sent.
Further, whether the priority obtains module, excellent specifically for judging to carry in the multiple data source
First grade label；If so, obtaining the priority of the multiple data source from the multiple data source；If it is not, being then not carry
There is the data source of the priority tag to distribute corresponding priority according to preset rules.
Further, the queue division module, for the data source with equal priority to be divided into the same queue,
To obtain multiple queues.
Further, the subqueue division module, for by the data source in the same queue according to different
Data type is divided into multiple subqueues；Or
Data source in the same queue is divided into multiple subqueues according to different Data subjects；Or
Data source in the same queue is divided according to different data source IP for multiple subqueues.
Further, described device further includes：
Resource distribution module, for distributing corresponding cpu resource and memory source for each subqueue；Based on every height team
The cpu resource and memory source of column determine the priority of each subqueue.
The third aspect, the embodiment of the present invention provide a kind of electronic equipment, including processor and memory, the memory
It is stored with computer-readable instruction fetch, when the computer-readable instruction fetch is executed by the processor, operation such as above-mentioned the
On the one hand the step in the method provided.
Fourth aspect, the embodiment of the present invention provide a kind of readable storage medium storing program for executing, are stored thereon with computer program, the meter
The step in the method that first aspect offer is as above provided is run when calculation machine program is executed by processor.
The beneficial effect of the embodiment of the present invention is：
The embodiment of the present invention provides a kind of data collection distribution method and device, this method by obtaining multiple data sources,
Then the priority of the multiple data source is obtained, then the priority based on the multiple data source divides the multiple data source
For multiple queues, each queue is divided into multiple subqueues by the different data classifications based on the multiple data source, then base
In the priority of each queue and the priority of each subqueue, the multiple data source is sent, therefore, in this programme
Data source is divided into different multiple queues and multiple subqueues, according to the priority of multiple queues and multiple subqueues to data
Source is sent, and being effectively treated for data source is which thereby enhanced, avoid due to data volume is excessive data source can not be carried out and
When the problem of effectively handling.
Other features and advantages of the present invention will be illustrated in subsequent specification, also, partly be become from specification
It is clear that by implementing understanding of the embodiment of the present invention.The objectives and other advantages of the invention can be by written theory
Specifically noted structure is achieved and obtained in bright book, claims and attached drawing.
Detailed description of the invention
In order to illustrate the technical solution of the embodiments of the present invention more clearly, below will be to needed in the embodiment attached
Figure is briefly described, it should be understood that the following drawings illustrates only certain embodiments of the present invention, therefore is not construed as pair
The restriction of range for those of ordinary skill in the art without creative efforts, can also be according to this
A little attached drawings obtain other relevant attached drawings.
Fig. 1 shows a kind of structural block diagram that can be applied to the electronic equipment in the embodiment of the present application；
Fig. 2 is a kind of flow chart of data collection distribution method provided in an embodiment of the present invention；
Fig. 3 is a kind of structural block diagram of data collection diostribution device provided in an embodiment of the present invention.
Specific embodiment
Below in conjunction with attached drawing in the embodiment of the present invention, technical solution in the embodiment of the present invention carries out clear, complete
Ground description, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.Usually exist
The component of the embodiment of the present invention described and illustrated in attached drawing can be arranged and be designed with a variety of different configurations herein.Cause
This, is not intended to limit claimed invention to the detailed description of the embodiment of the present invention provided in the accompanying drawings below
Range, but it is merely representative of selected embodiment of the invention.Based on the embodiment of the present invention, those skilled in the art are not doing
Every other embodiment obtained under the premise of creative work out, shall fall within the protection scope of the present invention.
It should be noted that：Similar label and letter indicate similar terms in following attached drawing, therefore, once a certain Xiang Yi
It is defined in a attached drawing, does not then need that it is further defined and explained in subsequent attached drawing.Meanwhile of the invention
In description, term " first ", " second " etc. are only used for distinguishing description, are not understood to indicate or imply relative importance.
Fig. 1 is please referred to, Fig. 1 shows a kind of structural block diagram of electronic equipment 100 that can be applied in the embodiment of the present application.
Electronic equipment 100 may include that data collection diostribution device, memory 101, storage control 102, processor 103, peripheral hardware connect
Mouth 104, input-output unit 105, audio unit 106, display unit 107.
The memory 101, storage control 102, processor 103, Peripheral Interface 104, input-output unit 105, sound
Frequency unit 106, each element of display unit 107 are directly or indirectly electrically connected between each other, to realize the transmission or friendship of data
Mutually.It is electrically connected for example, these elements can be realized between each other by one or more communication bus or signal wire.The data
Collect diostribution device include at least one can be stored in the form of software or firmware (firmware) in the memory 101 or
The software function module being solidificated in the operating system (operating system, OS) of the data collection diostribution device.Institute
It states processor 103 to be used to execute the executable module stored in memory 101, such as the data collection diostribution device includes
Software function module or computer program.
Wherein, memory 101 may be, but not limited to, random access memory (Random Access Memory,
RAM), read-only memory (Read Only Memory, ROM), programmable read only memory (Programmable Read-Only
Memory, PROM), erasable read-only memory (Erasable Programmable Read-Only Memory, EPROM),
Electricallyerasable ROM (EEROM) (Electric Erasable Programmable Read-Only Memory, EEPROM) etc..
Wherein, memory 101 is for storing program, and the processor 103 executes described program after receiving and executing instruction, aforementioned
Method performed by the server that the stream process that any embodiment of the embodiment of the present invention discloses defines can be applied to processor 103
In, or realized by processor 103.
Processor 103 can be a kind of IC chip, the processing capacity with signal.Above-mentioned processor 103 can
To be general processor, including central processing unit (Central Processing Unit, abbreviation CPU), network processing unit
(Network Processor, abbreviation NP) etc.；Can also be digital signal processor (DSP), specific integrated circuit (ASIC),
Ready-made programmable gate array (FPGA) either other programmable logic device, discrete gate or transistor logic, discrete hard
Part component.It may be implemented or execute disclosed each method, step and the logic diagram in the embodiment of the present invention.General processor
It can be microprocessor or the processor 103 be also possible to any conventional processor etc..
Various input/output devices are couple processor 103 and memory 101 by the Peripheral Interface 104.Some
In embodiment, Peripheral Interface 104, processor 103 and storage control 102 can be realized in one single chip.Other one
In a little examples, they can be realized by independent chip respectively.
Input-output unit 105 realizes user and the server (or local terminal) for being supplied to user input data
Interaction.The input-output unit 105 may be, but not limited to, mouse and keyboard etc..
Audio unit 106 provides a user audio interface, may include one or more microphones, one or more raises
Sound device and voicefrequency circuit.
Display unit 107 provides an interactive interface (such as user's operation circle between the electronic equipment 100 and user
Face) or for display image data give user reference.In the present embodiment, the display unit 107 can be liquid crystal display
Or touch control display.It can be the capacitance type touch control screen or resistance of support single-point and multi-point touch operation if touch control display
Formula touch screen etc..Single-point and multi-point touch operation is supported to refer to that touch control display can sense on the touch control display one
Or at multiple positions simultaneously generate touch control operation, and the touch control operation that this is sensed transfer to processor 103 carry out calculate and
Processing.
Various input/output devices are couple processor 103 and memory 101 by the Peripheral Interface 104.Some
In embodiment, Peripheral Interface 104, processor 103 and storage control 102 can be realized in one single chip.Other one
In a little examples, they can be realized by independent chip respectively.
Input-output unit 105 is used to be supplied to the interaction that user input data realizes user and processing terminal.It is described defeated
Entering output unit 105 may be, but not limited to, mouse and keyboard etc..
It is appreciated that structure shown in FIG. 1 is only to illustrate, the electronic equipment 100 may also include more than shown in Fig. 1
Perhaps less component or with the configuration different from shown in Fig. 1.Each component shown in Fig. 1 can use hardware, software
Or combinations thereof realize.
The embodiment of the present invention also provides a kind of readable storage medium storing program for executing, is stored thereon with computer program, the computer journey
The step in data collection distribution method as described below is run when sequence is executed by processor.
Referring to figure 2., Fig. 2 is a kind of flow chart of data collection distribution method provided in an embodiment of the present invention, the side
Method includes the following steps：
Step S110：Obtain multiple data sources.
Step S120：Obtain the priority of the multiple data source.
The data source of acquisition can be different types of data source or same type of data source, wherein according to data
The data format in source determines the type of data.
In order to realize the successive transmission to multiple data sources, after receiving multiple data sources, the multiple data are judged
Whether priority tag is carried in source, if so, the priority of the multiple data source is obtained from the multiple data source,
If it is not, being then not carry the data source of the priority tag according to the corresponding priority of preset rules distribution.
Wherein, preset rules are：According to data source IP, the theme of data, data type (such as json format, generally
Text formatting) etc. multiple dimensions, for example, if distributing priority according to source IP, data that corresponding a certain source IP is 10.1.1.2
It is 1 that priority is distributed in source, and the data source distribution priority for being 10.0.0.3 for source IP is 2；If dividing according to the theme of data
It is such as 10 for the data source distribution priority that the theme of data is life, the theme for data is work with priority
It is 8 that data source, which distributes priority,；If distributing priority according to the type of data, as the type for data is document format
It is 7 that data source, which distributes priority, and the type for data is that the data source distribution priority of picture format is 8.
It should be noted that can generally carry above-mentioned data source IP, the theme of data in data source, in the type of data
At least one, the corresponding priority of each classification is defined in advance in advance, if simultaneously including at least in a data source
Two kinds of data dimensions such as include the type of data source IP and data, then when distributing priority for the data source, by two dimensions
Priority, which is averaged, distributes priority for data source, and e.g., the corresponding priority of data source IP of the data source is 8, data
The corresponding priority of type be 7, then finally be the data source distribute priority be 7.5.
Wherein, for priority number since 0, numerical value is smaller, and priority is higher.
Step S130：The multiple data source is divided into multiple queues by the priority based on the multiple data source.
In order to realize the distribution to multiple data sources, multiple data sources can be divided into multiple queues, then according to queue
Priority be distributed, as an implementation, the data source with equal priority can be divided into the same queue,
To obtain multiple queues.
Certainly, as other embodiment, the priority of multiple data sources according to priority can also just be arranged
Sequence, for example, the data source before priority is 10 is divided into first queue, by the data source point in priority between 10-20
For second queue, the data source after being 20 in priority is divided into third queue.
It certainly, as an implementation, can also be by the number for belonging to same Data subject before priority is 20
It is divided into a queue according to source, the data source for belonging to the same Data subject after being 20 in priority is divided into a queue,
Thus it can be divided into multiple queues based on different Data subjects.
Step S140：Each queue is divided into multiple subqueues by the different data classifications based on the multiple data source.
In order to further be segmented to data source, multiple queues can also be divided into multiple subqueues, i.e., it will be described same
Data source in one queue is divided into multiple subqueues according to different data types；Or by the number in the same queue
It is divided into multiple subqueues according to different Data subjects according to source；Or by the data source in the same queue according to different
Data source IP points are multiple subqueues.
For example, if the data source in each queue is the same priority, for the data to this same priority
Source is sequentially sent, then these data sources can be further subdivided into multiple subqueues, example based on different data types
Data source if data type is document is divided into a subqueue, and data type is divided into another for the data source of picture format
A subqueue.Alternatively, can also be by these data sources based on the different subdivided multiple subqueues of Data subject, for example, data master
The data source of entitled life theme is divided into a subqueue, and Data subject is that the data source of work topic is divided into another height
Queue.Alternatively, these data sources can be also based on to source IP is divided into multiple subqueues, such as by the data of certain a kind of source IP address
Source is divided into a subqueue, and the data source of another kind of source IP address is divided into another subqueue.It thus can be by above-mentioned
Multiple queues are divided into multiple subqueues by method again, i.e., multiple data sources are divided into the queue of multilayer.
Step S150：The priority of priority and each subqueue based on each queue, to the multiple data source into
Row is sent.
It is preferential such as the data source in each queue since multiple queues are divided according to the priority of data source
Grade it is identical, then for the priority of each queue assignment be it is equal with the priority of data source in the queue, for example, in some queue
The priority of data source be 5, then the priority of the queue is also 5, so, data can be carried out according to the priority of queue
Transmission, i.e., first send the data source in the high queue of priority.
It should be noted that data source can be sent to data storage device or data processing equipment, to data source
Stored or carried out other processing.
Due to being also divided into multiple subqueues in queue, the priority of multiple subqueues can be allocated based on data classification,
The data source in queue for being such as 5 by priority has been divided into three subqueues based on the type of data, and each data type is corresponding
Priority it is different, thus can be based on the priority of three subqueues so three subqueues can correspond to different priority
Height successively sends the data source in queue.For example, data source is divided into four queues, wherein four queues is preferential
Grade is respectively 1,2,4,8, then preferentially sends the data source in the queue that priority is 1, and the queue that priority is 1 has been divided into three again
A subqueue, the priority of three of them subqueue are respectively 2,5,7, then first send priority be 2 in subqueue in data
Then source retransmits the data source in the subqueue that priority is 5, then retransmit the data in the subqueue that priority is 7
Source.
In addition, in order to carry out isolation transmission to each subqueue, can be distributed for each subqueue corresponding cpu resource and
Memory source can determine the priority of each subqueue based on the cpu resource and memory source of each subqueue.
As soon as the priority of subqueue is higher, the resource accounting for its distribution is higher, and when transmission, each subqueue is used
Different processes accomplishes the security isolation between process, so that the data between each subqueue, which send family, not to be influenced.
In addition, can also monitor handling capacity the moment in the present embodiment, the reception of unit time sends byte number, unit time
Reception send record strip number, number of failures, cpu resource service condition, memory service condition etc., to can be obtained by monitoring
Above-mentioned monitoring information is obtained, in the case where monitoring information is undesirable, can pass through and modify the correlation that data are sent directly in webpage
Parameter options modify above-mentioned monitoring information, realize the control of data distribution.In addition, administrator can also give different use
Family configures different permissions, and the user under different rights is enabled to monitor the distribution with all data sources under the user account
State.
Referring to figure 3., Fig. 3 is a kind of structural block diagram of data collection diostribution device 200 provided in an embodiment of the present invention, institute
Stating device includes：
Data acquisition module 210, for obtaining multiple data sources；
Priority obtains module 220, for obtaining the priority of the multiple data source；
The multiple data source is divided into more by queue division module 230 for the priority based on the multiple data source
A queue；
Subqueue division module 240 divides each queue for the different data classifications based on the multiple data source
For multiple subqueues；
Data transmission blocks 250, for the priority of priority and each subqueue based on each queue, to described more
A data source is sent.
Further, the priority obtains module 220, specifically for judging whether carry in the multiple data source
Priority tag；If so, obtaining the priority of the multiple data source from the multiple data source；If it is not, being then not take
Data source with the priority tag distributes corresponding priority according to preset rules.
Further, the queue division module 230, for the data source with equal priority to be divided into the same team
Column, to obtain multiple queues.
Further, the subqueue division module 240, for by the data source in the same queue according to difference
Data type be divided into multiple subqueues；Or
Data source in the same queue is divided into multiple subqueues according to different Data subjects；Or
Data source in the same queue is divided according to different data source IP for multiple subqueues.
Further, described device further includes：
Resource distribution module, for distributing corresponding cpu resource and memory source for each subqueue；Based on every height team
The cpu resource and memory source of column determine the priority of each subqueue.
It is apparent to those skilled in the art that for convenience and simplicity of description, the device of foregoing description
Specific work process, no longer can excessively be repeated herein with reference to the corresponding process in preceding method.
In conclusion the embodiment of the present invention provides a kind of data collection distribution method and device, this method is more by obtaining
Then a data source obtains the priority of the multiple data source, then the priority based on the multiple data source will be described more
A data source is divided into multiple queues, and each queue is divided into multiple sub- teams based on the different data classifications of the multiple data source
Column, are then based on the priority of each queue and the priority of each subqueue, send to the multiple data source, because
Data source, is divided into different multiple queues and multiple subqueues by this in this programme, according to multiple queues and multiple subqueues
Priority sends data source, which thereby enhances being effectively treated for data source, avoid because data volume is excessive can not be right
The problem of data source is timely and effectively handled.
In several embodiments provided herein, it should be understood that disclosed device and method can also pass through
Other modes are realized.The apparatus embodiments described above are merely exemplary, for example, flow chart and block diagram in attached drawing
Show the device of multiple embodiments according to the present invention, the architectural framework in the cards of method and computer program product,
Function and operation.In this regard, each box in flowchart or block diagram can represent the one of a module, section or code
Part, a part of the module, section or code, which includes that one or more is for implementing the specified logical function, to be held
Row instruction.It should also be noted that function marked in the box can also be to be different from some implementations as replacement
The sequence marked in attached drawing occurs.For example, two continuous boxes can actually be basically executed in parallel, they are sometimes
It can execute in the opposite order, this depends on the function involved.It is also noted that every in block diagram and or flow chart
The combination of box in a box and block diagram and or flow chart can use the dedicated base for executing defined function or movement
It realizes, or can realize using a combination of dedicated hardware and computer instructions in the system of hardware.
In addition, each functional module in each embodiment of the present invention can integrate one independent portion of formation together
Point, it is also possible to modules individualism, an independent part can also be integrated to form with two or more modules.
It, can be with if the function is realized and when sold or used as an independent product in the form of software function module
It is stored in a computer readable storage medium.Based on this understanding, technical solution of the present invention is substantially in other words
The part of the part that contributes to existing technology or the technical solution can be embodied in the form of software products, the meter
Calculation machine software product is stored in a storage medium, including some instructions are used so that a computer equipment (can be a
People's computer, server or network equipment etc.) it performs all or part of the steps of the method described in the various embodiments of the present invention.
And storage medium above-mentioned includes：USB flash disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), arbitrary access are deposited
The various media that can store program code such as reservoir (RAM, Random Access Memory), magnetic or disk.
The foregoing is only a preferred embodiment of the present invention, is not intended to restrict the invention, for the skill of this field
For art personnel, the invention may be variously modified and varied.All within the spirits and principles of the present invention, made any to repair
Change, equivalent replacement, improvement etc., should all be included in the protection scope of the present invention.It should be noted that：Similar label and letter exist
Similar terms are indicated in following attached drawing, therefore, once being defined in a certain Xiang Yi attached drawing, are then not required in subsequent attached drawing
It is further defined and explained.
The above description is merely a specific embodiment, but scope of protection of the present invention is not limited thereto, any
Those familiar with the art in the technical scope disclosed by the present invention, can easily think of the change or the replacement, and should all contain
Lid is within protection scope of the present invention.Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
It should be noted that, in this document, relational terms such as first and second and the like are used merely to a reality
Body or operation are distinguished with another entity or operation, are deposited without necessarily requiring or implying between these entities or operation
In any actual relationship or order or sequence.Moreover, the terms "include", "comprise" or its any other variant are intended to
Non-exclusive inclusion, so that the process, method, article or equipment including a series of elements is not only wanted including those
Element, but also including other elements that are not explicitly listed, or further include for this process, method, article or equipment
Intrinsic element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that
There is also other identical elements in process, method, article or equipment including the element.
Claims (10)
1. a kind of data collection distribution method, which is characterized in that the method includes：
Obtain multiple data sources；
Obtain the priority of the multiple data source；
The multiple data source is divided into multiple queues by the priority based on the multiple data source；
Each queue is divided into multiple subqueues by the different data classifications based on the multiple data source；
The priority of priority and each subqueue based on each queue, sends the multiple data source.
2. the method according to claim 1, wherein obtain the priority of the multiple data source, including：
Judge whether carry priority tag in the multiple data source；
If so, obtaining the priority of the multiple data source from the multiple data source；
If it is not, being then not carry the data source of the priority tag according to the corresponding priority of preset rules distribution.
3. the method according to claim 1, wherein the priority based on the multiple data source will be the multiple
Data source is divided into multiple queues, including：
Data source with equal priority is divided into the same queue, to obtain multiple queues.
4. according to the method described in claim 3, it is characterized in that, the different data classifications based on the multiple data source will
Each queue is divided into multiple subqueues, including：
Data source in the same queue is divided into multiple subqueues according to different data types；Or
Data source in the same queue is divided into multiple subqueues according to different Data subjects；Or
Data source in the same queue is divided according to different data source IP for multiple subqueues.
5. the method according to claim 1, wherein the different data classifications based on the multiple data source will
Each queue is divided into after multiple subqueues, the priority of priority and each subqueue based on each queue, to described more
Before a data source is sent, the method also includes：
Corresponding cpu resource and memory source are distributed for each subqueue；
Cpu resource and memory source based on each subqueue determine the priority of each subqueue.
6. a kind of data collection diostribution device, which is characterized in that described device includes：
Data acquisition module, for obtaining multiple data sources；
Priority obtains module, for obtaining the priority of the multiple data source；
The multiple data source is divided into multiple queues for the priority based on the multiple data source by queue division module；
Each queue is divided into multiple sons for the different data classifications based on the multiple data source by subqueue division module
Queue；
Data transmission blocks, for the priority of priority and each subqueue based on each queue, to the multiple data
Source is sent.
7. device according to claim 6, which is characterized in that the priority obtains module, is specifically used for described in judgement
Whether priority tag is carried in multiple data sources；If so, obtaining the multiple data source from the multiple data source
Priority；If it is not, being then not carry the data source of the priority tag according to the corresponding priority of preset rules distribution.
8. device according to claim 6, which is characterized in that the queue division module, for that will have similarity priority
The data source of grade is divided into the same queue, to obtain multiple queues.
9. device according to claim 8, which is characterized in that the subqueue division module, being used for will be described same
Data source in queue is divided into multiple subqueues according to different data types；Or
Data source in the same queue is divided into multiple subqueues according to different Data subjects；Or
Data source in the same queue is divided according to different data source IP for multiple subqueues.
10. device according to claim 6, which is characterized in that described device further includes：
Resource distribution module, for distributing corresponding cpu resource and memory source for each subqueue；Based on each subqueue
Cpu resource and memory source determine the priority of each subqueue.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810793438.8A CN108900434A (en) | 2018-07-18 | 2018-07-18 | Data collection distribution method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810793438.8A CN108900434A (en) | 2018-07-18 | 2018-07-18 | Data collection distribution method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108900434A true CN108900434A (en) | 2018-11-27 |
Family
ID=64350836
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810793438.8A Pending CN108900434A (en) | 2018-07-18 | 2018-07-18 | Data collection distribution method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108900434A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110675648A (en) * | 2019-08-20 | 2020-01-10 | 中国平安财产保险股份有限公司 | Method, system and server for data source acquisition and data deduplication acquisition of parking lot |
| CN110795505A (en) * | 2019-10-23 | 2020-02-14 | 北京仿真中心 | Cross-domain synchronous distribution system and method for user data |
| CN110990444A (en) * | 2019-11-27 | 2020-04-10 | 中诚信征信有限公司 | Data query method and device |
| CN111064799A (en) * | 2019-12-25 | 2020-04-24 | 北京首信科技股份有限公司 | Data pushing method, device, equipment and storage medium applied to Internet of things |
| CN112989887A (en) * | 2019-12-16 | 2021-06-18 | 深圳云天励飞技术有限公司 | File merging method and device and electronic equipment |
| CN113342410A (en) * | 2021-05-25 | 2021-09-03 | 山石网科通信技术股份有限公司 | Feature library data processing method and device |
| CN113472896A (en) * | 2020-09-24 | 2021-10-01 | 中国电建集团海外投资有限公司 | Data sending method based on public cloud |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101262651A (en) * | 2008-04-18 | 2008-09-10 | 中国科学院计算技术研究所 | Base station downlink data dispatching method and system in broadband wireless MAN |
| CN101594307B (en) * | 2009-06-30 | 2012-12-05 | 中兴通讯股份有限公司 | Multi-queue based scheduling method and system |
| CN102811159A (en) * | 2011-06-03 | 2012-12-05 | 中兴通讯股份有限公司 | Method and device for scheduling upstream service |
| CN106559355A (en) * | 2016-12-12 | 2017-04-05 | 天津逐帜科技发展有限公司 | IP Telecommunication Network edge gateway equipment resource management method based on fair algorithm |
| CN106789739A (en) * | 2017-02-24 | 2017-05-31 | 上海斐讯数据通信技术有限公司 | A kind of multistage multiple row can configure schedule management method and system |
-
2018
- 2018-07-18 CN CN201810793438.8A patent/CN108900434A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101262651A (en) * | 2008-04-18 | 2008-09-10 | 中国科学院计算技术研究所 | Base station downlink data dispatching method and system in broadband wireless MAN |
| CN101594307B (en) * | 2009-06-30 | 2012-12-05 | 中兴通讯股份有限公司 | Multi-queue based scheduling method and system |
| CN102811159A (en) * | 2011-06-03 | 2012-12-05 | 中兴通讯股份有限公司 | Method and device for scheduling upstream service |
| CN106559355A (en) * | 2016-12-12 | 2017-04-05 | 天津逐帜科技发展有限公司 | IP Telecommunication Network edge gateway equipment resource management method based on fair algorithm |
| CN106789739A (en) * | 2017-02-24 | 2017-05-31 | 上海斐讯数据通信技术有限公司 | A kind of multistage multiple row can configure schedule management method and system |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110675648A (en) * | 2019-08-20 | 2020-01-10 | 中国平安财产保险股份有限公司 | Method, system and server for data source acquisition and data deduplication acquisition of parking lot |
| CN110795505A (en) * | 2019-10-23 | 2020-02-14 | 北京仿真中心 | Cross-domain synchronous distribution system and method for user data |
| CN110990444A (en) * | 2019-11-27 | 2020-04-10 | 中诚信征信有限公司 | Data query method and device |
| CN112989887A (en) * | 2019-12-16 | 2021-06-18 | 深圳云天励飞技术有限公司 | File merging method and device and electronic equipment |
| CN111064799A (en) * | 2019-12-25 | 2020-04-24 | 北京首信科技股份有限公司 | Data pushing method, device, equipment and storage medium applied to Internet of things |
| CN113472896A (en) * | 2020-09-24 | 2021-10-01 | 中国电建集团海外投资有限公司 | Data sending method based on public cloud |
| CN113472896B (en) * | 2020-09-24 | 2023-10-27 | 中国电建集团海外投资有限公司 | Public cloud-based data transmission method |
| CN113342410A (en) * | 2021-05-25 | 2021-09-03 | 山石网科通信技术股份有限公司 | Feature library data processing method and device |
| CN113342410B (en) * | 2021-05-25 | 2022-09-06 | 山石网科通信技术股份有限公司 | Feature library data processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108900434A (en) | Data collection distribution method and device | |
| US11068964B2 (en) | Method and device for processing service request | |
| CN108776934B (en) | Distributed data calculation method and device, computer equipment and readable storage medium | |
| CN107729137B (en) | Server, block chain signature verification decryption method and storage medium | |
| CN109324977A (en) | Detection method, device and the electronic equipment of application program Caton | |
| CN105787077A (en) | Data synchronizing method and device | |
| CN108111554B (en) | Control method and device for access queue | |
| CN108462760A (en) | Electronic device, cluster access domain name automatic generation method and storage medium | |
| US8413159B2 (en) | Classifying workload for a service | |
| US10439957B1 (en) | Tenant-based management system and method for distributed computing environments | |
| CN111694646A (en) | Resource scheduling method and device, electronic equipment and computer readable storage medium | |
| CN111352800A (en) | Big data cluster monitoring method and related equipment | |
| US9854031B2 (en) | Cloud service agent based on service level agreement(SLA) | |
| CN108829469A (en) | A kind of application program page methods of exhibiting and device | |
| CN104038501B (en) | Display terminal cluster management system and display terminal cluster management method | |
| CN108563668A (en) | The method, apparatus and terminal device of the Hybrid page jump H5 pages | |
| CN112799824A (en) | Current limiting method, device, equipment and storage medium | |
| CN111142989B (en) | Object management method, device and storage medium | |
| CN109697166B (en) | Test data preparation method and related device | |
| US20180210804A1 (en) | System and method for dynamic scaling of concurrent processing threads | |
| US20180196928A1 (en) | Network visualization for access controls | |
| EP4142258A1 (en) | Edge computing network, data transmission method and apparatus, device and storage medium | |
| CN110489612A (en) | A kind of the multi objective monitoring method and device of information | |
| CN112583660B (en) | Main domain and standby domain test comparison method, device and system of recommendation platform | |
| CN110782211A (en) | Data processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20181127 |