CN110147356A - Data transmission method and device - Google Patents
Data transmission method and device Download PDFInfo
- Publication number
- CN110147356A CN110147356A CN201910399729.3A CN201910399729A CN110147356A CN 110147356 A CN110147356 A CN 110147356A CN 201910399729 A CN201910399729 A CN 201910399729A CN 110147356 A CN110147356 A CN 110147356A
- Authority
- CN
- China
- Prior art keywords
- data
- data transmission
- downstream
- storage service
- different
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/181—Append-only file systems, e.g. using logs or journals to store data providing write once read many [WORM] semantics
Abstract
The invention proposes a kind of data transmission method and devices, wherein and different acquisition data to be parsed into the data packet of Uniform data format this method comprises: acquire being customized of different data sources to obtain different acquisition data；The data packet write-in data pipe of the Uniform data format is cached；The data packet cached in the data pipe is read, and downstream data storage service is written into the data packet of reading, to realize that data are transmitted；During data transmission, the flow rate for reading data is limited according to the preset flow peak value of the downstream data storage service.The present invention is localized buffering to data by data pipe as a result, then by flow control assembly dynamic regulation data transfer throughput, makes entire transmission process safety and stability, to effectively improve data transmission rate.
Description
Technical field
The present invention relates to technical field of data transmission, in particular to a kind of data transmission method, a kind of data transmission device
And a kind of computer readable storage medium and computer equipment.
Background technique
During service operation, it will usually generate the daily record data of magnanimity, for convenience it is subsequent to the daily record data into
Row is checked and is counted, it is necessary to be carried out unified acquisition and transmission to the daily record data, and be stored to unified daily record data and store
In service, however, due to data volume frequent fluctuation and can not be estimated in data collection and transmission, once data volume compared with
When big, it may be difficult to which guarantee data security efficient transmission.
Summary of the invention
The present invention is directed to solve one of the technical problem in above-mentioned technology at least to a certain extent.For this purpose, of the invention
One purpose is to propose a kind of data transmission method, is localized buffering to data by data pipe, then pass through flow
Control assembly dynamic regulation data transfer throughput makes entire transmission process safety and stability, to effectively improve data transmission rate.
Second object of the present invention is to propose a kind of computer readable storage medium.
Third object of the present invention is to propose a kind of computer equipment.
Fourth object of the present invention is to propose a kind of data transmission device.
In order to achieve the above objectives, first aspect present invention embodiment proposes a kind of data transmission method, and this method includes
Following steps: acquiring to obtain different acquisition data being customized of different data sources, and by different acquisition data solutions
Analyse into the data packet of Uniform data format；The data packet write-in data pipe of the Uniform data format is cached；It reads
The data packet cached in the data pipe, and downstream data storage service is written into the data packet of reading, to realize that data pass
It is defeated；During data transmission, according to the preset flow peak value of the downstream data storage service to the flow for reading data
Rate is limited.
Data transmission method according to an embodiment of the present invention first acquires to obtain not being customized of different data sources
With acquisition data, and different acquisition data are parsed into the data packet of Uniform data format；Then by Uniform data format
Data packet write-in data pipe cached；Then the data packet that caches in data pipe is read, and by the data packet of reading
Downstream data storage service is written, to realize that data are transmitted；And it during data transmission, is stored and is taken according to downstream data
The preset flow peak value of business limits the flow rate for reading data.The present invention is by by Uniform data format as a result,
Data packet write-in data pipe is cached, then by flow control assembly dynamic regulation data transfer throughput, makes entirely to transmit
Process safety is stablized, to effectively improve data transmission rate.
In addition, the data transmission method proposed according to that above embodiment of the present invention can also have following additional technology special
Sign:
Optionally, to being customized of different data sources acquisition before, also by web administration platform to data transfer task into
Row configuration, and pass through Zookeeper for configuration information real-time synchronization to different data task nodes.
Optionally, in the operational process of the data transfer task, the data task node will be also currently running
Mission bit stream is sent to the Zookeeper, so that the web administration platform obtains the task by the Zookeeper
Information.
Optionally, during data are transmitted, Xiang Suoshu downstream data storage service sends inquiry instruction, to obtain
The service detection information that downstream data storage service is fed back according to the inquiry instruction is stated, and according to the service detection information pair
The health status of the downstream data storage service is determined, and obtains corresponding preset flow peak according to judgement result
Value.
In order to achieve the above objectives, second aspect of the present invention embodiment proposes a kind of computer readable storage medium, thereon
It is stored with data distributing program, such as above-mentioned data transmission method is realized when which is executed by processor.
Computer readable storage medium according to an embodiment of the present invention, by storing data transfer program, such data are passed
Such as above-mentioned data transmission method is realized when defeated program is executed by processor, and makes entire transmission process safety and stability, thus effectively
Improve data transfer rate.
In order to achieve the above objectives, third aspect present invention embodiment proposes a kind of computer equipment, including memory, place
The computer program managing device and storage on a memory and can running on a processor, the processor execute the computer journey
When sequence, such as above-mentioned data transmission method is realized.
Computer equipment according to an embodiment of the present invention, by memory stores data transfer program, such data transmission
Such as above-mentioned data transmission method is realized when program is executed by processor, and makes entire transmission process safety and stability, to effectively mention
High data transmission rate.
In order to achieve the above objectives, fourth aspect present invention embodiment proposes a kind of data transmission device, comprising: acquisition mould
Block for acquiring to being customized of different data sources to obtain different acquisition data, and different acquisition data is parsed
At the data packet of Uniform data format；Cache module, for by the data packet of the Uniform data format be written data pipe into
Row caching；Data transmission control module is write for reading the data packet cached in the data pipe, and by the data packet of reading
Enter downstream data storage service, to realize that data are transmitted, and according to the downstream data storage during data transmission
The preset flow peak value of service limits the flow rate for reading data.
The data transmission device provided according to embodiments of the present invention, by acquisition module to being customized of different data sources
Different acquisition data are parsed into the data packet of Uniform data format to obtain different acquisition data by acquisition；Pass through again
Cache module caches the data packet write-in data pipe of Uniform data format；Then it is read by data transmission control module
The data packet cached in data pipe is taken, and downstream data storage service is written into the data packet of reading, to realize that data are transmitted,
And during data transmission according to the preset flow peak value of downstream data storage service to the flow rate for reading data
It is limited.The present invention is cached by the way that data pipe is written in the data packet of Uniform data format as a result, then passes through flow
Control assembly dynamic regulation data transfer throughput makes entire transmission process safety and stability, to effectively improve data transmission rate.
In addition, the data transmission device proposed according to that above embodiment of the present invention can also have following additional technology special
Sign:
Further, data transmission device further includes configuration module, for determining in the acquisition module different data sources
Before inhibition and generation acquisition, data transfer task is configured by web administration platform, and passes through Zookeeper for configuration information
Real-time synchronization is to different data task nodes.
Optionally, in the operational process of the data transfer task, the data task node will be also currently running
Mission bit stream is sent to the Zookeeper, so that the web administration platform obtains the task by the Zookeeper
Information.
Optionally, the data transmission control module is also used to, and during data transmission, Xiang Suoshu downstream data is deposited
Storage service sends inquiry instruction, is believed with obtaining the downstream data storage service according to the service detection that the inquiry instruction is fed back
Breath, and determines according to health status of the service detection information to the downstream data storage service, and according to sentencing
Determine result and obtains corresponding preset flow peak value.
Detailed description of the invention
Fig. 1 is the flow chart according to the data transmission method of the embodiment of the present invention；
Fig. 2 is the flow chart according to the data transmission method of one embodiment of the invention；
Fig. 3 is the distributed structure/architecture figure according to the data transmission method of one embodiment of the invention；
Fig. 4 is the data transmission system architecture diagram according to the data transmission method of one embodiment of the invention；
Fig. 5 is the block diagram according to the data transmission device of the embodiment of the present invention；
Fig. 6 is the block diagram according to the data transmission device of one embodiment of the invention.
Specific embodiment
The embodiment of the present invention is described below in detail, examples of the embodiments are shown in the accompanying drawings, wherein from beginning to end
Same or similar label indicates same or similar element or element with the same or similar functions.Below with reference to attached
The embodiment of figure description is exemplary, it is intended to is used to explain the present invention, and is not considered as limiting the invention.
In data collection and transmission, since data source is different, (daily record data is likely to be present in journal file, number
According to library, queue service etc.), data protocol is different, causes data transmission procedure to be difficult to be managed collectively, although at present can be with
It is handled by the open source component such as Logstash/Flume, still, required configuration is complicated, higher use cost is needed,
When run multiple Logstash/Flume service when, it is necessary to manpower intervention is managed, can equally bring very high O&M at
This；In addition, due to data volume frequent fluctuation and can not estimate, when the amount of data is large, prior art can not continue to protect
Hinder data safety, high efficiency of transmission, the i.e. prior art lack mass data caching and control ability, may cause in flow peak period
Loss of data or due to upstream and downstream service data read-write pressure to cause collapse etc. failures.
For this purpose, data transmission method proposed by the present invention, acquires being customized of different data sources different to obtain
Data are acquired, and different acquisition data are parsed into the data packet of Uniform data format；Then by the number of Uniform data format
It is cached according to packet write-in data pipe；Then the data packet cached in data pipe is read, and the data packet of reading is written
Downstream data storage service, to realize that data are transmitted；And during data transmission, according to downstream data storage service
Preset flow peak value limits the flow rate for reading data.The present invention is by by the data of Uniform data format as a result,
Packet write-in data pipe is cached, then by flow control assembly dynamic regulation data transfer throughput, makes entire transmission process
Safety and stability, to effectively improve data transmission rate.
In order to better understand the above technical scheme, the exemplary reality that the present invention will be described in more detail below with reference to accompanying drawings
Apply example.Although showing exemplary embodiment of the present invention in attached drawing, it being understood, however, that may be realized in various forms this hair
It is bright and should not be limited by the embodiments set forth herein.It is to be able to thoroughly understand this on the contrary, providing these embodiments
Invention, and the scope of the present invention can be fully disclosed to those skilled in the art.
In order to better understand the above technical scheme, in conjunction with appended figures and specific embodiments to upper
Technical solution is stated to be described in detail.
Fig. 1 is the flow chart according to the data transmission method of the embodiment of the present invention.As shown in Figure 1 and Figure 4, the present invention is implemented
Example data transmission method the following steps are included:
Step 101, being customized of different data sources is acquired to obtain different acquisition data, and by different acquisition data
It is parsed into the data packet of Uniform data format.
That is, corresponding data acquisition components are customized for different data sources in data transmission operational process, with
Data acquisition is carried out to different data sources, and different acquisition data are parsed into the data packet of Uniform data format.
Step 102, the data packet write-in data pipe of Uniform data format is cached.
It should be noted that after the completion of data acquisition, in order to which the data traffic peak for ensureing instantaneous will not be to downstream data
Storage service causes excessive write-in pressure, delays so first writing data into data pipe (Data Channel) and carrying out data
It deposits.
As one embodiment, data pipe (Data Channel) includes the queue of memory grade and local disk queue, number
According to the queue of memory grade is first written, local disk is write data into again after detecting EMS memory occupation and reaching certain number and is lined up, wherein
Certain number is user's self-defining.
Step 103, the data packet cached in data pipe is read, and the write-in downstream data storage of the data packet of reading is taken
Business, to realize that data are transmitted.
Step 104, during data are transmitted, according to the preset flow peak value of downstream data storage service to reading number
According to flow rate limited.
That is, during data transmission, according to the preset flow peak of the downstream data storage service got
The flow of data is read in value, dynamic adjustment from data pipe (Data Channel), thus to the flow rate for reading data
It is limited.
Preferably, during data are transmitted, downstream data storage service sends inquiry instruction, to obtain downstream number
According to the service detection information that storage service is fed back according to inquiry instruction, and according to service detection information to downstream data storage service
Health status determined, and according to determining that result obtains corresponding preset flow peak value.
As one embodiment, during data transmission, downstream data storage service defines three kinds of health status
(Green, Yellow, Red), while corresponding preset flow peak value is defined for three kinds of health status, it is mainly reflected in every
Second transmission is how many data；System according to preset detection cycle, look by periodically downstream data storage service transmission as a result,
Instruction is ask, to obtain the service detection information that downstream data storage service is fed back according to inquiry instruction, and is believed according to service detection
Breath determines the health status of downstream data storage service, to obtain corresponding preset flow peak according to judging result
Value.
It should be noted that service detection information include detection service whether can touch reach, service node CPU memory is using feelings
Condition and data-interface response condition；Three kinds of health status are respectively corresponded according to the corresponding situation of above-mentioned service detection information
(Green, Yellow, Red), wherein reached when detection service can be touched, service node CPU memory using less, data-interface respond it is fast,
Then judge the health status of downstream data storage service for Green；It is reached when detection service can be touched, service node CPU memory uses
The response of medium, data-interface is medium, then judges the health status of downstream data storage service for Yellow；When detection service can be touched
It reaches, more, the data-interface low-response of service node CPU memory use, then judges that the health status of downstream data storage service is
Red；Specific service node cpu memory using how much with correspond to which kind of health status, data-interface response how soon and how slowly with
Which kind of corresponding health status can preparatory self-defining.
Data transmission method according to an embodiment of the present invention first acquires to obtain not being customized of different data sources
With acquisition data, and different acquisition data are parsed into the data packet of Uniform data format；Then by Uniform data format
Data packet write-in data pipe cached；Then the data packet that caches in data pipe is read, and by the data packet of reading
Downstream data storage service is written, to realize that data are transmitted；And it during data transmission, is stored and is taken according to downstream data
The preset flow peak value of business limits the flow rate for reading data.The present invention is by by Uniform data format as a result,
Data packet write-in data pipe is cached, then by flow control assembly dynamic regulation data transfer throughput, makes entirely to transmit
Process safety is stablized, to effectively improve data transmission rate.
Fig. 2 is the flow chart according to the data transmission method of one embodiment of the invention.As Figure 2-3, which passes
The difference of transmission method and above-described embodiment is further comprising the steps of:
Step 105, data transfer task is configured by web administration platform, and by Zookeeper that configuration information is real
When be synchronized to different data task nodes.
Step 106, in the operational process of data transfer task, mission bit stream that data task node will be also currently running
It is sent to the Zookeeper, so that web administration platform obtains mission bit stream by Zookeeper.
It should be noted that step 105 and step 106 are executed before to the acquisition of being customized of different data sources
's；By step 105 and step 106 to reach under distributed structure/architecture, the convenient management that unitizes of data transfer task.
The data transmission method that the embodiment of the present invention proposes is by increasing Web Visualization Management Platform, Configuration Online data
Transformation task, so that high degree reduces use cost；In addition, by distributed structure/architecture, automation, unified management data transmission
Service, to reduce O&M cost.
In addition, the embodiment of the present invention also proposed a kind of computer readable storage medium, it is stored thereon with data transmission journey
Sequence realizes such as above-mentioned data transmission method when the data distributing program is executed by processor.
Computer readable storage medium according to an embodiment of the present invention, by storing data transfer program, such data are passed
Such as above-mentioned data transmission method is realized when defeated program is executed by processor, and makes entire transmission process safety and stability, thus effectively
Improve data transfer rate.
In addition, the embodiment of the present invention also proposed a kind of computer equipment, including memory, processor and it is stored in storage
On device and the computer program that can run on a processor, when the processor executes the computer program, realize as above-mentioned
Data transmission method.
Computer equipment according to an embodiment of the present invention, by memory stores data transfer program, such data transmission
Such as above-mentioned data transmission method is realized when program is executed by processor, and makes entire transmission process safety and stability, to effectively mention
High data transmission rate.
Fig. 5 is the block diagram according to the data transmission device of the embodiment of the present invention；As shown in figure 5, the data are transmitted
Device includes acquisition module 501, cache module 502 and data transmission control module 503；
Wherein, acquisition module 501, for being acquired to being customized of different data sources to obtain different acquisition data, and will
Different acquisition data are parsed into the data packet of Uniform data format；Cache module 502, for by the data of Uniform data format
Packet write-in data pipe is cached；Data transmission control module 503, for reading the data packet cached in data pipe, and
Downstream data storage service is written into the data packet of reading, to realize that data are transmitted, and during data transmission according to
The preset flow peak value of downstream data storage service limits the flow rate for reading data.
As one embodiment, data transmission control module 503 is also used to, and during data transmission, is downstream counted
Inquiry instruction is sent according to storage service, to obtain the service detection information that downstream data storage service is fed back according to inquiry instruction,
And determined according to health status of the service detection information to downstream data storage service, and phase is obtained according to judgement result
The preset flow peak value answered.
It should be noted that the explanation previously for the embodiment of data transmission method is equally applicable to the present embodiment
Data transmission device, details are not described herein again.
The data transmission device provided according to embodiments of the present invention, by acquisition module to being customized of different data sources
Different acquisition data are parsed into the data packet of Uniform data format to obtain different acquisition data by acquisition；Pass through again
Cache module caches the data packet write-in data pipe of Uniform data format；Then it is read by data transmission control module
The data packet cached in data pipe is taken, and downstream data storage service is written into the data packet of reading, to realize that data are transmitted,
And during data transmission according to the preset flow peak value of downstream data storage service to the flow rate for reading data
It is limited.The present invention is cached by the way that data pipe is written in the data packet of Uniform data format as a result, then passes through flow
Control assembly dynamic regulation data transfer throughput makes entire transmission process safety and stability, to effectively improve data transmission rate.
Fig. 5 is the block diagram according to the data transmission device of one embodiment of the invention；As shown in figure 5, the data
The difference of transmitting device and above-described embodiment is to further include configuration module 504, is used in acquisition module 501 to different data sources
Before being customized acquisition, data transfer task is configured by web administration platform, and will be matched by Zookeeper
Confidence ceases real-time synchronization to different data task nodes.
Further, in the operational process of data transfer task, data task node also believes being currently running for task
Breath is sent to Zookeeper, so that web administration platform obtains mission bit stream by Zookeeper.
It should be noted that the explanation previously for the embodiment of data transmission method is equally applicable to the present embodiment
Data transmission device, details are not described herein again.
The data transmission device that the embodiment of the present invention proposes is by increasing Web Visualization Management Platform, Configuration Online data
Transformation task, so that high degree reduces use cost；In addition, by distributed structure/architecture, automation, unified management data transmission
Service, to reduce O&M cost.
It should be understood by those skilled in the art that, the embodiment of the present invention can provide as method, system or computer program
Product.Therefore, complete hardware embodiment, complete software embodiment or reality combining software and hardware aspects can be used in the present invention
Apply the form of example.Moreover, it wherein includes the computer of computer usable program code that the present invention, which can be used in one or more,
The computer program implemented in usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) produces
The form of product.
The present invention be referring to according to the method for the embodiment of the present invention, the process of equipment (system) and computer program product
Figure and/or block diagram describe.It should be understood that every one stream in flowchart and/or the block diagram can be realized by computer program instructions
The combination of process and/or box in journey and/or box and flowchart and/or the block diagram.It can provide these computer programs
Instruct the processor of general purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices to produce
A raw machine, so that being generated by the instruction that computer or the processor of other programmable data processing devices execute for real
The device for the function of being specified in present one or more flows of the flowchart and/or one or more blocks of the block diagram.
These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates,
Enable the manufacture of device, the command device realize in one box of one or more flows of the flowchart and/or block diagram or
The function of being specified in multiple boxes.
These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that counting
Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, thus in computer or
The instruction executed on other programmable devices is provided for realizing in one or more flows of the flowchart and/or block diagram one
The step of function of being specified in a box or multiple boxes.
It should be noted that in the claims, any reference symbol between parentheses should not be configured to power
The limitation that benefit requires.Word "comprising" does not exclude the presence of component or step not listed in the claims.Before component
Word "a" or "an" does not exclude the presence of multiple such components.The present invention can be by means of including several different components
It hardware and is realized by means of properly programmed computer.In the unit claims listing several devices, these are filled
Several in setting, which can be, to be embodied by the same item of hardware.The use of word first, second, and third is not
Indicate any sequence.These words can be construed to title.
Although preferred embodiments of the present invention have been described, it is created once a person skilled in the art knows basic
Property concept, then additional changes and modifications may be made to these embodiments.So it includes excellent that the following claims are intended to be interpreted as
It selects embodiment and falls into all change and modification of the scope of the invention.
Obviously, various changes and modifications can be made to the invention without departing from essence of the invention by those skilled in the art
Mind and range.In this way, if these modifications and changes of the present invention belongs to the range of the claims in the present invention and its equivalent technologies
Within, then the present invention is also intended to include these modifications and variations.
In the description of the present invention, it is to be understood that, term " first ", " second " are used for description purposes only, and cannot
It is interpreted as indication or suggestion relative importance or implicitly indicates the quantity of indicated technical characteristic.Define as a result, " the
One ", the feature of " second " can explicitly or implicitly include one or more of the features.In the description of the present invention,
The meaning of " plurality " is two or more, unless otherwise specifically defined.
In the present invention unless specifically defined or limited otherwise, term " installation ", " connected ", " connection ", " fixation " etc.
Term shall be understood in a broad sense, for example, it may be being fixedly connected, may be a detachable connection, or integral；It can be mechanical connect
It connects, is also possible to be electrically connected；It can be directly connected, can also can be in two elements indirectly connected through an intermediary
The interaction relationship of the connection in portion or two elements.It for the ordinary skill in the art, can be according to specific feelings
Condition understands the concrete meaning of above-mentioned term in the present invention.
In the present invention unless specifically defined or limited otherwise, fisrt feature in the second feature " on " or " down " can be with
It is that the first and second features directly contact or the first and second features pass through intermediary mediate contact.Moreover, fisrt feature exists
Second feature " on ", " top " and " above " but fisrt feature be directly above or diagonally above the second feature, or be merely representative of
First feature horizontal height is higher than second feature.Fisrt feature can be under the second feature " below ", " below " and " below "
One feature is directly under or diagonally below the second feature, or is merely representative of first feature horizontal height less than second feature.
In the description of this specification, reference term " one embodiment ", " some embodiments ", " example ", " specifically show
The description of example " or " some examples " etc. means specific features, structure, material or spy described in conjunction with this embodiment or example
Point is included at least one embodiment or example of the invention.In the present specification, schematic expression of the above terms are not
It is interpreted as that identical embodiment or example must be directed to.Moreover, particular features, structures, materials, or characteristics described
It may be combined in any suitable manner in any one or more of the embodiments or examples.In addition, without conflicting with each other, this
The technical staff in field can be by the spy of different embodiments or examples described in this specification and different embodiments or examples
Sign is combined.
Although the embodiments of the present invention has been shown and described above, it is to be understood that above-described embodiment is example
Property, it is not considered as limiting the invention, those skilled in the art within the scope of the invention can be to above-mentioned
Embodiment is changed, modifies, replacement and variant.
Claims (10)
1. a kind of data transmission method, which comprises the following steps:
Being customized of different data sources is acquired to obtain different acquisition data, and different acquisition data are parsed into system
The data packet of one data format；
The data packet write-in data pipe of the Uniform data format is cached；
The data packet cached in the data pipe is read, and downstream data storage service is written into the data packet of reading, with reality
Existing data transmission；
During data transmission, according to the preset flow peak value of the downstream data storage service to the flow for reading data
Rate is limited.
2. data transmission method as described in claim 1, which is characterized in that different data sources being customized acquisition
Before, also data transfer task is configured by web administration platform, and pass through Zookeeper for configuration information real-time synchronization
To different data task nodes.
3. data transmission method as claimed in claim 2, which is characterized in that in the operational process of the data transfer task
In, the mission bit stream being currently running also is sent to the Zookeeper by the data task node, so as to the web administration
Platform obtains the mission bit stream by the Zookeeper.
4. data transmission method as claimed in any one of claims 1-3, which is characterized in that during data transmission,
Inquiry instruction is sent to the downstream data storage service, to obtain the downstream data storage service according to the inquiry instruction
The service detection information of feedback, and carried out according to health status of the service detection information to the downstream data storage service
Determine, and corresponding preset flow peak value is obtained according to judgement result.
5. a kind of computer readable storage medium, which is characterized in that be stored thereon with data distributing program, the data distributing program
Such as data transmission method of any of claims 1-4 is realized when being executed by processor.
6. a kind of computer equipment including memory, processor and stores the meter that can be run on a memory and on a processor
Calculation machine program, which is characterized in that when the processor executes the computer program, realize such as any one of claim 1-4
The data transmission method.
7. a kind of data transmission device characterized by comprising
Acquisition module for acquiring being customized of different data sources to obtain different acquisition data, and different is adopted
Collection data are parsed into the data packet of Uniform data format；
Cache module, for caching the data packet write-in data pipe of the Uniform data format；
Data transmission control module is written for reading the data packet cached in the data pipe, and by the data packet of reading
Downstream data storage service to realize that data are transmitted, and is stored according to the downstream data during data transmission and is taken
The preset flow peak value of business limits the flow rate for reading data.
8. data transmission device as claimed in claim 7, which is characterized in that further include configuration module, in the acquisition
Before module acquires being customized of different data sources, data transfer task is configured by web administration platform, and leads to
Zookeeper is crossed by configuration information real-time synchronization to different data task nodes.
9. data transmission device as claimed in claim 8, which is characterized in that in the operational process of the data transfer task
In, the mission bit stream being currently running also is sent to the Zookeeper by the data task node, so as to the web administration
Platform obtains the mission bit stream by the Zookeeper.
10. data transmission device as claimed in any one of claims 7-9, which is characterized in that the Data Transmission Controlling mould
Block is also used to, and during data transmission, Xiang Suoshu downstream data storage service sends inquiry instruction, to obtain the downstream
The service detection information that data storage service is fed back according to the inquiry instruction, and according to the service detection information under described
The health status of trip data storage service is determined, and obtains corresponding preset flow peak value according to judgement result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910399729.3A CN110147356A (en) | 2019-05-14 | 2019-05-14 | Data transmission method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910399729.3A CN110147356A (en) | 2019-05-14 | 2019-05-14 | Data transmission method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110147356A true CN110147356A (en) | 2019-08-20 |
Family
ID=67595382
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910399729.3A Pending CN110147356A (en) | 2019-05-14 | 2019-05-14 | Data transmission method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110147356A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111522642A (en) * | 2020-04-21 | 2020-08-11 | 观为监测技术无锡股份有限公司 | Method for acquiring state monitoring data of wind turbine generator, query method and electronic equipment |
| CN111694828A (en) * | 2020-06-08 | 2020-09-22 | 山东伏羲智库互联网研究院 | Data processing method, device, system and storage medium |
| CN112988702A (en) * | 2019-12-12 | 2021-06-18 | 天翼电子商务有限公司 | Heterogeneous data source real-time data transmission method and system, storage medium and terminal |
| CN113676417A (en) * | 2021-10-25 | 2021-11-19 | 北京安华金和科技有限公司 | Flow control method and system |
| CN114650565A (en) * | 2022-05-23 | 2022-06-21 | 深圳市天翊瑞霖智能科技有限公司 | Intelligent flow control management system for intelligent gateway based on data analysis |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3223467A1 (en) * | 2014-12-08 | 2017-09-27 | Huawei Technologies Co., Ltd. | Data transmission method and device |
| CN108153828A (en) * | 2017-12-12 | 2018-06-12 | 顺丰科技有限公司 | A kind of persistence method of real time data, device and equipment, storage medium |
| CN108846076A (en) * | 2018-06-08 | 2018-11-20 | 山大地纬软件股份有限公司 | The massive multi-source ETL process method and system of supporting interface adaptation |
-
2019
- 2019-05-14 CN CN201910399729.3A patent/CN110147356A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3223467A1 (en) * | 2014-12-08 | 2017-09-27 | Huawei Technologies Co., Ltd. | Data transmission method and device |
| CN108153828A (en) * | 2017-12-12 | 2018-06-12 | 顺丰科技有限公司 | A kind of persistence method of real time data, device and equipment, storage medium |
| CN108846076A (en) * | 2018-06-08 | 2018-11-20 | 山大地纬软件股份有限公司 | The massive multi-source ETL process method and system of supporting interface adaptation |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112988702A (en) * | 2019-12-12 | 2021-06-18 | 天翼电子商务有限公司 | Heterogeneous data source real-time data transmission method and system, storage medium and terminal |
| CN111522642A (en) * | 2020-04-21 | 2020-08-11 | 观为监测技术无锡股份有限公司 | Method for acquiring state monitoring data of wind turbine generator, query method and electronic equipment |
| CN111522642B (en) * | 2020-04-21 | 2024-02-13 | 观为监测技术无锡股份有限公司 | Method for acquiring and inquiring state monitoring data of wind turbine generator, and electronic equipment |
| CN111694828A (en) * | 2020-06-08 | 2020-09-22 | 山东伏羲智库互联网研究院 | Data processing method, device, system and storage medium |
| CN113676417A (en) * | 2021-10-25 | 2021-11-19 | 北京安华金和科技有限公司 | Flow control method and system |
| CN113676417B (en) * | 2021-10-25 | 2022-02-15 | 北京安华金和科技有限公司 | Flow control method and system |
| CN114650565A (en) * | 2022-05-23 | 2022-06-21 | 深圳市天翊瑞霖智能科技有限公司 | Intelligent flow control management system for intelligent gateway based on data analysis |
| CN114650565B (en) * | 2022-05-23 | 2022-08-16 | 深圳市天翊瑞霖智能科技有限公司 | Intelligent flow control management system for intelligent gateway based on data analysis |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110147356A (en) | Data transmission method and device | |
| US9501555B2 (en) | Real-time data management for a power grid | |
| CN105005274B (en) | Big data in management process control system | |
| CN110971614A (en) | Internet of things adaptation method and system, computer equipment and storage medium | |
| JP6549351B2 (en) | Collection of data within process control system and distribution to big data machines | |
| CN110609512A (en) | Internet of things platform and Internet of things equipment monitoring method | |
| US7849156B2 (en) | Method, apparatus and computer program product for discovering and prioritizing patterns of component usage in a virtual application container for enhancing prefetching | |
| GB2574906A (en) | Pipeline data processing | |
| CN108092813A (en) | Data center's total management system server hardware Governance framework and implementation method | |
| CN109739877A (en) | Database Systems and data managing method | |
| CN105827678B (en) | Communication means and node under a kind of framework based on High Availabitity | |
| CN109739816A (en) | The date storage method and device of comprehensive monitoring system based on rail traffic | |
| CN108833522A (en) | A kind of believable system and method for determining node | |
| CN105574769A (en) | Method and apparatus for establishing social network dynamic relation graph | |
| CN109347647A (en) | Group message distribution method and device | |
| CN110008131A (en) | A kind of method and device for the region AB experiment management realized based on algorithm | |
| CN109460345A (en) | The calculation method and system of real time data | |
| CN109783463A (en) | File synchronisation method, device and computer readable storage medium | |
| CN107087232A (en) | The real-time status detection method and system of user | |
| CN109829094A (en) | Distributed reptile system | |
| CN109299196A (en) | Data managing method, network and data acquisition storage application method and its system | |
| CN109062500A (en) | A kind of metadata management server, data-storage system and date storage method | |
| CN109271438A (en) | A kind of data bank access method and its system | |
| CN109743249A (en) | Forming method, integrated gateway and the integrated system of the integrated gateway of passenger information system | |
| CN109117295A (en) | A kind of overtime monitoring method of transaction and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 361000 G02, 3rd floor, No.10, Punan 1st Road, Dongpu Road, Siming District, Xiamen City, Fujian Province Applicant after: Xiamen draft Co., Ltd Address before: 361000 G02, 3rd floor, No.10, Punan 1st Road, Dongpu Road, Siming District, Xiamen City, Fujian Province Applicant before: XIAMEN HUANLEGUANG TECHNOLOGY Co.,Ltd. |