CN107147654A - Group data acquisition system and data acquisition method - Google Patents
Group data acquisition system and data acquisition method Download PDFInfo
- Publication number
- CN107147654A CN107147654A CN201710374654.4A CN201710374654A CN107147654A CN 107147654 A CN107147654 A CN 107147654A CN 201710374654 A CN201710374654 A CN 201710374654A CN 107147654 A CN107147654 A CN 107147654A
- Authority
- CN
- China
- Prior art keywords
- data
- unit data
- extracted
- storage area
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 16
- 238000003860 storage Methods 0.000 claims abstract description 116
- 230000005540 biological transmission Effects 0.000 claims abstract description 36
- 238000013523 data management Methods 0.000 claims abstract description 29
- 238000004458 analytical method Methods 0.000 claims abstract description 15
- 238000009826 distribution Methods 0.000 claims description 4
- 239000000284 extract Substances 0.000 claims description 4
- 239000004744 fabric Substances 0.000 claims description 2
- 238000012544 monitoring process Methods 0.000 abstract description 2
- 238000007405 data analysis Methods 0.000 abstract 1
- 238000004590 computer program Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 238000012545 processing Methods 0.000 description 5
- 238000005057 refrigeration Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 238000003032 molecular docking Methods 0.000 description 3
- 238000010438 heat treatment Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 241000208340 Araliaceae Species 0.000 description 1
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 description 1
- 235000003140 Panax quinquefolius Nutrition 0.000 description 1
- VQLYBLABXAHUDN-UHFFFAOYSA-N bis(4-fluorophenyl)-methyl-(1,2,4-triazol-1-ylmethyl)silane;methyl n-(1h-benzimidazol-2-yl)carbamate Chemical compound C1=CC=C2NC(NC(=O)OC)=NC2=C1.C=1C=C(F)C=CC=1[Si](C=1C=CC(F)=CC=1)(C)CN1C=NC=N1 VQLYBLABXAHUDN-UHFFFAOYSA-N 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 235000008434 ginseng Nutrition 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
- H04L69/164—Adaptation or special uses of UDP protocol
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0805—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
Landscapes
- Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Databases & Information Systems (AREA)
- Environmental & Geological Engineering (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Computer Security & Cryptography (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of data acquisition, in particular to a unit data acquisition system and a data acquisition method, aiming at solving the problem that the existing unit data acquisition system can not support UDP protocol, the system comprises: the data transmission module can be used for monitoring a preset UDP port, the monitored unit data are uploaded to the distributed storage system from the designated storage area, the unit data extracted from the distributed storage system are analyzed through the data management module by the at least one data analysis module, and the analysis result is stored in the designated database.
Description
Technical field
The present invention relates to data acquisition technology field, more particularly to a kind of unit data collecting system and data acquisition side
Method.
Background technology
To ensure that unit (e.g., air-conditioner set) is run steadily in the long term in the case of safe and reliable, wireless telecommunications can be used
Operation of the mode to air-conditioner set performs real-time monitoring, to grasp every service data of air-conditioner set in time.
Under prior art, it will usually following two wireless communication modes, every service data of air-conditioner set is uploaded to
Management system：
Mode one：
Every service data of air-conditioner set is uploaded to by management system by 2G networks, however, aforesaid way not only can
Substantial amounts of charging flow is produced, moreover, the transmission rate of 2G networks is also unsatisfactory.
Mode two：
Every service data of air-conditioner set is uploaded to by management system by WiFi module, however, aforesaid way is usual
It is to be realized for transmission control protocol (Transmission Control Protocol, TCP), total well known, Transmission Control Protocol
It is connection-oriented, i.e. communicating pair need to be reliably connected, however, pin before formally data are sent by three session establishments
It is more complicated during aforesaid way later maintenance to the situation of multi type air conditioner group.
In view of this, the present invention realizes WiFi module using UDP (User Data Protocol, UDP)
Radio communication, however, for udp protocol, existing unit data collecting system is not applied to simultaneously, therefore, need to redesign one
Unit data collecting system is planted, to meet the transmission requirement of udp protocol.
The content of the invention
The embodiment of the present invention provides a kind of unit data collecting system and collecting method, to solve existing unit
The problem of data collecting system can not support udp protocol.
Concrete technical scheme provided in an embodiment of the present invention is as follows：
A kind of unit data collecting system, including：
Data transmission module, for the UDP udp port of preset monitored, and by some units listened to
Data are stored in designated storage area, and determine that the amount of storage of the designated storage area is met when imposing a condition, from described
Corresponding unit data are extracted in designated storage area, and the unit data extracted are uploaded to specified distributed storage system
System；
Data management module, for receive data resolution module transmission data requesting instructions when, from it is described specify point
Corresponding unit data are extracted in cloth storage system, and the unit data extracted are sent to corresponding data parsing mould
Block；
At least one data resolution module, for receiving the corresponding unit data that the data management module is sent, with
And corresponding unit data are parsed, and analysis result is preserved to specified database.
Optionally, shown designated storage area is local file system, wherein, the local file system includes at least one
Individual storage file.
Optionally, some unit data listened to are stored in before designated storage area, the data transmission module
For：
Based on preset rules, Effective judgement is carried out to each the unit data listened to respectively；
Each unit data with validity are filtered out, and determine each comfortable local file system of each unit data
Corresponding storage file in system；
It is described that some unit data listened to are stored in designated storage area, including：
Each unit data with validity are stored in each self-corresponding storage file.
Optionally, when determining that the amount of storage satisfaction of the designated storage area imposes a condition, from the designated storage area
It is middle to extract corresponding unit data, and when the unit data extracted are uploaded into specified distributed memory system, the data
Transport module is used for：
Each storage file included respectively for local file system, performs following operate：
When determining that amount of storage reaches given threshold, all or part of unit data of storage are extracted；
The all or part of unit data extracted are uploaded to the specified distributed memory system.
Optionally, corresponding unit data are parsed, and when analysis result is preserved to specified database, institute
Stating data resolution module is used for：
Respectively for each the unit data received, following operate is performed：
Based on default udp protocol, each field that unit packet contains is parsed；
It is determined that each self-corresponding actual parameter of each field；
Each actual parameter of determination is preserved to the specified database.
Optionally, the data management module is additionally operable to：Manage and safeguard at least one described data resolution module.
A kind of unit collecting method, including：
The UDP udp port of preset monitored, and some unit data listened to are stored in specified deposit
Storage area domain；
When determining that the amount of storage satisfaction of the designated storage area imposes a condition, phase is extracted from the designated storage area
The unit data answered, and the unit data extracted are stored in specified distributed memory system；
Data requesting instructions based on generation, extract corresponding unit data from the specified distributed memory system,
And the unit data extracted are parsed, and analysis result is preserved to specified database.
Optionally, shown designated storage area is local file system, wherein, the local file system includes at least one
Individual storage file.
Optionally, some unit data listened to are stored in before designated storage area, in addition to：
Based on preset rules, Effective judgement is carried out to each the unit data listened to respectively；
Each unit data with validity are filtered out, and determine each comfortable local file system of each unit data
Corresponding storage file in system；
It is described that some unit data listened to are stored in designated storage area, including：
Each unit data with validity are stored in each self-corresponding storage file.
Optionally, when determining that the amount of storage satisfaction of the designated storage area imposes a condition, from the designated storage area
It is middle to extract corresponding unit data, and the unit data extracted are stored in specified distributed memory system, including：
Each storage file included respectively for local file system, performs following operate：
When determining that amount of storage reaches given threshold, all or part of unit data of storage are extracted；
The all or part of unit data extracted are stored in the specified distributed memory system.
Optionally, the unit data extracted are parsed, and analysis result is preserved to specified database, including：
Respectively for each the unit data received, following operate is performed：
Based on default udp protocol, each field that unit packet contains is parsed；
It is determined that each self-corresponding actual parameter of each field；
Each actual parameter of determination is preserved to the specified database.
In the embodiment of the present invention, unit data collecting system includes：Data transmission module, data management module and at least one
Individual data resolution module, wherein, data transmission module can be used for the udp port of preset monitored, and by some units listened to
Data upload to HDFS from designated storage area, then by least one data resolution module, by data management module to from
The unit data extracted in HDFS are parsed, and analysis result is preserved to specified database, so, the unit after improvement
Data collecting system can not only support udp protocol, moreover, by multi-level storage machine system, can be by all unit data listened to
Correctly included, and then improve data acquisition rate and data-handling efficiency, so as to improve Consumer's Experience.
Brief description of the drawings
Fig. 1 be the embodiment of the present invention in, unit data collecting system structural representation；
Fig. 2 be the embodiment of the present invention in, unit collecting method flow chart.
Embodiment
In order to solve in the problem of existing unit data collecting system can not support udp protocol, the embodiment of the present invention, weight
A kind of unit data collecting system and collecting method are newly devised, the system includes：Data transmission module, data management mould
Block and at least one data resolution module, wherein, data transmission module is used for the UDP UDP ends of preset monitored
Mouthful, and some unit data listened to are stored in designated storage area, and determine that the amount of storage of designated storage area expires
When foot imposes a condition, corresponding unit data are extracted from designated storage area, and the unit data extracted are uploaded to finger
Determine distributed memory system；Data management module, for receive data resolution module transmission data requesting instructions when, from finger
Determine to extract corresponding unit data in distributed memory system, and the unit data extracted are sent to the parsing of corresponding data
Module；At least one data resolution module, the corresponding unit data for receiving data management module transmission, and to corresponding
Unit data parsed, and analysis result is preserved to specified database.
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Site preparation is described, it is clear that described embodiment is only a part of embodiment of the invention, is not whole embodiments.It is based on
Embodiment in the present invention, it is every other that those of ordinary skill in the art are obtained under the premise of creative work is not made
Embodiment, belongs to the scope of protection of the invention.
The solution of the present invention will be described in detail by specific embodiment below, certainly, the present invention be not limited to
Lower embodiment.
In the embodiment of the present invention, all units (being illustrated below by taking air-conditioner set as an example) are equipped with accordingly
WiFi module, the air-conditioner set data that each air-conditioner set is produced all can be by the WiFi module that is associated with itself to outgoing
Send, wherein, between air-conditioner set and WiFi module, row data communication can be entered by serial ports, is not limited herein.
As shown in fig.1, in the embodiment of the present invention, devising a kind of unit data collecting system, at least passed including data
Defeated module, data management module and at least one data resolution module, wherein,
Data transmission module, is protected for receiving some air-conditioner set data, and by some air-conditioner set data received
There is designated storage area, and determine that the amount of storage of designated storage area is met when imposing a condition, from designated storage area
Corresponding air-conditioner set data are extracted, and the air-conditioner set data extracted are uploaded to specified distributed memory system；
Data management module, for receive data resolution module transmission data requesting instructions when, from specified distribution
Corresponding air-conditioner set data are extracted in storage system, and the air-conditioner set data extracted are sent to the parsing of corresponding data
Module；
At least one data resolution module, the corresponding air-conditioner set data for receiving data management module transmission, with
And corresponding air-conditioner set data are parsed, and analysis result is preserved to specified database.Specifically, refering to Fig. 2 institutes
Show, the preferred embodiment of the present invention is described in detail with reference to accompanying drawing, air-conditioner set data are adopted in the embodiment of the present invention
The idiographic flow of collection is as follows：
Step 200：Data transmission module receives some air-conditioner set data, and by some air-conditioner set data received
It is stored in designated storage area.
Specifically, in the embodiment of the present invention, the server specified in advance for each WiFi module, configuration data transport module
Domain name and UDP (User Datagram Protocol, UDP) port, when WiFi module receive correspondence sky
Adjust after the air-conditioner set data that unit is uploaded, WiFi module can be sent out air-conditioner set data to be passed in the form of UDP message bag
It is sent on configured good udp port.
The present embodiment, why using UDP communications, is to be towards connectionless because of udp protocol, i.e. communicating pair need not
Foundation is reliably connected, as long as transmitting terminal determines the udp port of receiving terminal, so that it may send data to receiving terminal.
Further, data transmission module can monitor the udp port for the server specified, once listening to udp port has
The air-conditioner set data listened to can be performed preliminary screening, the air-conditioner set data that screening is obtained are stored in specified by data
Storage region, preferably, the preferred designated storage area of the present embodiment is local file system, wherein, local file system bag
Containing at least one storage file, above-mentioned storage file is to refer to a kind of storage region.
Specifically, data transmission module can be based on preset rules, each the air-conditioner set data received are entered respectively
Row Effective judgement, filters out each air-conditioner set data with validity, and determine each air conditioner with validity
Group each self-corresponding storage file of data, wherein, Effective judgement can include：Whether air-conditioner set data are partial data,
Whether air-conditioner set data are interference data, and whether air-conditioner set data are wrong data, etc..
For example, data transmission module can be detected using CRC (Cyclic Redundancy Check, CRC),
Whether the air-conditioner set data received are wrong data.
When right in each air-conditioner set data of data transmission module determination with validity, each comfortable local file system
After the storage file answered, each above-mentioned air-conditioner set data can be stored in each self-corresponding storage file.
Step 210：When data transmission module determines that the amount of storage satisfaction of designated storage area imposes a condition, from specified storage
Corresponding air-conditioner set data are extracted in region, and the air-conditioner set data extracted are uploaded to specified distributed storage system
System.
Specifically, each air-conditioner set data with validity are stored in each self-corresponding storage by data transmission module
After in file, if reach given threshold in the presence of the amount of storage of a storage file, deposited extracting in said one storage file
All or part of air-conditioner set data of storage, and by the above-mentioned all or part of air-conditioner set data extracted, be uploaded to specified
In distributed memory system.
In the embodiment of the present invention, distributed memory system can be Hadoop distributed file systems (Hadoop
Distribute File System, HDFS), or Google's file system (Google file system, GFS), or
Person is Lustre parallel distributive file systems, etc., for ease of understanding, in describing below, represents that distribution is deposited with HDFS
Storage system, certainly, during practical application, distributed memory system is not limited to HDFS.
For example, it is assumed that there is storage file 1, storage file 2 and storage file 3 in local file system, and each is deposited
The memory space for storing up file is all 500M, if being that the threshold value that each storage file is set is 450M in advance, when data transfer mould
Block is detected, when currently stored amount reaches 450M in storage file 1, then by all or part of air-conditioner set in storage file 1
Data are uploaded in specified HDFS.
Certainly, the memory space of each storage file can also be different, and the threshold value that each storage file is set also may be used
With difference, above-mentioned is only to illustrate, and can specifically see actual disposition.
Further, why, the air-conditioner set data that will be stored in local file system are uploaded in HDFS, are
Because the storage capacity of local file system is limited, and each air-conditioner set is producing substantial amounts of air-conditioner set number at any time
According to therefore, in the present embodiment, to mitigate the load of local file system, in the storage file that will be fully loaded with or will be fully loaded with
Air-conditioner set data, are uploaded in specified HDFS.
Certainly, in the specific implementation, data transmission module realizes that step 200 can be completed by data receiver program, and data are passed
Defeated module realizes that step 210 can be completed by data upload program.
Step 220：Data resolution module sends data requesting instructions to data management module.
Specifically, air-conditioner set data collecting system provided in an embodiment of the present invention, contains some data resolution modules,
When data resolution module needs to perform data parsing operation, actual demand can be based on and send request of data to data management module
Instruction, so as to which air-conditioner set data can be obtained, wherein, data requesting instructions can include No. ID of data resolution module, need to obtain
Data number taken, etc..
And when data resolution module needs to perform data parsing operation, data parsing operation it can complete above,
Or, analysis instruction has been obtained to judge.
For example, it is assumed that there is data resolution module 1, data resolution module 2, data parsing mould in air-conditioner set data system
Block 3 and data resolution module 4, if data resolution module 1 and data resolution module 4 need to perform data parsing operation, respectively can
Data requesting instructions A and data requesting instructions B is sent to data management module, wherein, data requesting instructions A is { data are parsed
Module 1,10 }, data requesting instructions B is { data resolution module 2,50 }.
Step 230：When data management module receives the data requesting instructions of data resolution module transmission, from specified distribution
Corresponding air-conditioner set data are extracted in formula storage system, and the air-conditioner set data extracted are sent to corresponding data solution
Analyse module.
Specifically, data management module receive each data resolution module transmission data requesting instructions when, from specify
Corresponding air-conditioner set data are extracted in HDFS, and based on each data requesting instructions received, by the air conditioner extracted
Group data return to corresponding data resolution module.
For example, continuing to illustrate with above-mentioned example, data management module receives the data of the transmission of data resolution module 1
Request instruction A, and the data requesting instructions B that data resolution module 2 is sent, by parsing, determine that data resolution module 1 needs
10 air-conditioner set data, data resolution module 2 needs 50 air-conditioner set data, then from specified HDFS, extracts 10 skies
Unit data are adjusted, and above-mentioned 10 air-conditioner set data are returned into data resolution module 1, and from specified HDFS, are extracted
50 air-conditioner set data, and above-mentioned 50 air-conditioner set data are returned into data resolution module 5.
Step 240：Data resolution module is received after the corresponding air-conditioner set data of data management module return, docking
The air-conditioner set data received are parsed, and analysis result is stored in into specified database.
Specifically, data resolution module is received after the corresponding air-conditioner set data of data management module return, docking
The air-conditioner set data received are parsed, and why data resolution module also needs the air-conditioner set to data management module return
Data are parsed, and are because WiFi module turns air-conditioner set data when sending air-conditioner set data to data transmission module
UDP message bag is changed to, therefore, data resolution module need to be based on default udp protocol, docking when receiving air-conditioner set data
The air-conditioner set data received are parsed.
Further, the air-conditioner set data instance received with a data resolution module, said one data
Parsing module, can be based on default udp protocol, each field that said one air-conditioner set packet contains be parsed, really
Fixed above-mentioned each self-corresponding actual parameter of each field, and each actual parameter of determination is stored in specified database.
For example, it is assumed that in default udp protocol, defining first character section and corresponding to air-conditioner set numbering, define second
The corresponding actual parameter of field is " room temperature ", and it is " operational mode " to define the 3rd corresponding actual parameter of field, wherein, first
Individual field includes 2 bytes, and second field includes 2 bytes, and the 3rd field includes 4 bytes；
If the air-conditioner set data A that data resolution module 1 is received, wherein, air-conditioner set data A for 78,18, system
It is cold }, the air-conditioner set data B that data resolution module 1 is received, wherein, air-conditioner set data B is { 03,22, refrigeration }；Data
The air-conditioner set data C that parsing module 2 is received, wherein, air-conditioner set data C is { 31,30, heat }；Data resolution module 3
The air-conditioner set data D received, wherein, air-conditioner set data D is { 09,25, heat }；
Through parsing, data resolution module 1 is determined, air-conditioner set 78 is currently at " refrigeration " state, and room temperature is " 18 " DEG C；
Data resolution module 1 determines that air-conditioner set 03 is currently at " refrigeration " state, and room temperature is " 22 " DEG C；Data resolution module 2 is true
Fixed, air-conditioner set 31 is currently at " heating " state, and room temperature is " 30 " DEG C；Data resolution module 3 determines, air-conditioner set 09
" heating " state is currently at, room temperature is " 25 " DEG C.
Further, data resolution module 1, data resolution module 2 and the meeting of data resolution module 3, respectively by each self-analytic data
Each obtained air-conditioner set data result is stored in the tables of data of database, and e.g., ginseng is shown in Table 1.
Table 1
| Air-conditioner set is numbered | Room temperature (DEG C) | Operational mode |
| 78 | 18 | Refrigeration |
| 03 | 22 | Refrigeration |
| 31 | 30 | Heat |
| 09 | 25 | Heat |
Further, due to there are multiple data resolution modules in this implementation, therefore, data management module can also be provided
Manage and safeguard the function of each data resolution module.
Certainly, unit data collecting system provided in an embodiment of the present invention, is not limited in using in air-conditioner set, on
State only by taking air-conditioner set as an example, other similar large scale equipments (e.g., being made up of multiple units or equipment) are also suitable.
In summary, in the embodiment of the present invention, unit data collecting system includes：Data transmission module, data management mould
Block and at least one data resolution module, wherein, data transmission module can be used for the udp port of preset monitored, and will listen to
Some unit data upload to specified HDFS from designated storage area, then by least one data resolution module, pass through data
Management module is parsed to the unit data extracted from above-mentioned specified HDFS, and analysis result is preserved to specified data
Storehouse, so, the unit data collecting system after improvement can support udp protocol, quickly can upload the unit data of magnanimity
To designated storage area, moreover, by above-mentioned multi-level storage machine system, positive acknowledgement can be carried out to all unit data listened to
Record.
Further, in the embodiment of the present invention, to avoid specifying HDFS to store too many ephemeral data, data management is utilized
Module realizes the scheduling to magnanimity unit data, and the scheduling to multiple data resolution modules, so, passes through data management mould
Block dispatches parsing of multiple data resolution modules completions to magnanimity unit data, and analysis result is stored in into specified database
In, data acquisition rate and data-handling efficiency are improved, so as to improve Consumer's Experience.
It should be understood by those skilled in the art that, embodiments of the invention can be provided as method, system or computer program
Product.Therefore, the present invention can be using the reality in terms of complete hardware embodiment, complete software embodiment or combination software and hardware
Apply the form of example.Moreover, the present invention can be used in one or more computers for wherein including computer usable program code
The computer program production that usable storage medium is implemented on (including but is not limited to magnetic disk storage, CD-ROM, optical memory etc.)
The form of product.
The present invention is the flow with reference to method according to embodiments of the present invention, equipment (system) and computer program product
Figure and/or block diagram are described.It should be understood that can be by every first-class in computer program instructions implementation process figure and/or block diagram
Journey and/or the flow in square frame and flow chart and/or block diagram and/or the combination of square frame.These computer programs can be provided
The processor of all-purpose computer, special-purpose computer, Embedded Processor or other programmable data processing devices is instructed to produce
A raw machine so that produced by the instruction of computer or the computing device of other programmable data processing devices for real
The device for the function of being specified in present one flow of flow chart or one square frame of multiple flows and/or block diagram or multiple square frames.
These computer program instructions, which may be alternatively stored in, can guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works so that the instruction being stored in the computer-readable memory, which is produced, to be included referring to
Make the manufacture of device, the command device realize in one flow of flow chart or multiple flows and/or one square frame of block diagram or
The function of being specified in multiple square frames.
These computer program instructions can be also loaded into computer or other programmable data processing devices so that in meter
Series of operation steps is performed on calculation machine or other programmable devices to produce computer implemented processing, thus in computer or
The instruction performed on other programmable devices is provided for realizing in one flow of flow chart or multiple flows and/or block diagram one
The step of function of being specified in individual square frame or multiple square frames.
, but those skilled in the art once know basic creation although preferred embodiments of the present invention have been described
Property concept, then can make other change and modification to these embodiments.So, appended claims are intended to be construed to include excellent
Select embodiment and fall into having altered and changing for the scope of the invention.
Obviously, those skilled in the art can carry out various changes and modification without departing from this hair to the embodiment of the present invention
The spirit and scope of bright embodiment.So, if these modifications and variations of the embodiment of the present invention belong to the claims in the present invention
And its within the scope of equivalent technologies, then the present invention is also intended to comprising including these changes and modification.
Claims (11)
1. a kind of unit data collecting system, it is characterised in that including：
Data transmission module, for the UDP udp port of preset monitored, and by some unit data listened to
It is stored in designated storage area, and determines that the amount of storage of the designated storage area is met when imposing a condition, is specified from described
Corresponding unit data are extracted in storage region, and the unit data extracted are uploaded to specified distributed memory system；
Data management module, for receive data resolution module transmission data requesting instructions when, from the specified distribution
Corresponding unit data are extracted in storage system, and the unit data extracted are sent to corresponding data resolution module；
At least one data resolution module, for receiving the corresponding unit data that the data management module is sent, and it is right
Corresponding unit data are parsed, and analysis result is preserved to specified database.
2. the system as claimed in claim 1, it is characterised in that shown designated storage area is local file system, wherein, institute
State local file system and include at least one storage file.
3. system as claimed in claim 1 or 2, it is characterised in that be stored in specify some unit data listened to and deposit
Before storage area domain, the data transmission module is used for：
Based on preset rules, Effective judgement is carried out to each the unit data listened to respectively；
Each unit data with validity are filtered out, and are determined in each comfortable local file system of each described unit data
Corresponding storage file；
It is described that some unit data listened to are stored in designated storage area, including：
Each unit data with validity are stored in each self-corresponding storage file.
4. system as claimed in claim 3, it is characterised in that determine that the amount of storage of the designated storage area meets setting bar
During part, corresponding unit data are extracted from the designated storage area, and the unit data extracted are uploaded to specified point
During cloth storage system, the data transmission module is used for：
Each storage file included respectively for local file system, performs following operate：
When determining that amount of storage reaches given threshold, all or part of unit data of storage are extracted；
The all or part of unit data extracted are uploaded to the specified distributed memory system.
5. system as claimed in claim 4, it is characterised in that corresponding unit data are parsed, and will parsing
When as a result preserving to specified database, the data resolution module is used for：
Respectively for each the unit data received, following operate is performed：
Based on default udp protocol, each field that unit packet contains is parsed；
It is determined that each self-corresponding actual parameter of each field；
Each actual parameter of determination is preserved to the specified database.
6. the system as claimed in claim 1, it is characterised in that the data management module is additionally operable to：
Manage and safeguard at least one described data resolution module.
7. a kind of unit collecting method, it is characterised in that including：
The UDP udp port of preset monitored, and some unit data listened to are stored in designated storage area
Domain；
When determining that the amount of storage satisfaction of the designated storage area imposes a condition, extract corresponding from the designated storage area
Unit data, and the unit data extracted are stored in specified distributed memory system；
Data requesting instructions based on generation, extract corresponding unit data from the specified distributed memory system, and
The unit data extracted are parsed, and analysis result is preserved to specified database.
8. method as claimed in claim 7, it is characterised in that shown designated storage area is local file system, wherein, institute
State local file system and include at least one storage file.
9. method as claimed in claim 7 or 8, it is characterised in that be stored in specify some unit data listened to and deposit
Before storage area domain, in addition to：
Based on preset rules, Effective judgement is carried out to each the unit data listened to respectively；
Each unit data with validity are filtered out, and are determined in each comfortable local file system of each described unit data
Corresponding storage file；
It is described that some unit data listened to are stored in designated storage area, including：
Each unit data with validity are stored in each self-corresponding storage file.
10. method as claimed in claim 9, it is characterised in that determine that the amount of storage of the designated storage area meets setting
During condition, corresponding unit data are extracted from the designated storage area, and the unit data extracted are stored in specified
Distributed memory system, including：
Each storage file included respectively for local file system, performs following operate：
When determining that amount of storage reaches given threshold, all or part of unit data of storage are extracted；
The all or part of unit data extracted are stored in the specified distributed memory system.
11. method as claimed in claim 10, it is characterised in that the unit data extracted are parsed, and will parsing
As a result preserve to specified database, including：
Respectively for each the unit data received, following operate is performed：
Based on default udp protocol, each field that unit packet contains is parsed；
It is determined that each self-corresponding actual parameter of each field；
Each actual parameter of determination is preserved to the specified database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710374654.4A CN107147654A (en) | 2017-05-24 | 2017-05-24 | Group data acquisition system and data acquisition method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710374654.4A CN107147654A (en) | 2017-05-24 | 2017-05-24 | Group data acquisition system and data acquisition method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107147654A true CN107147654A (en) | 2017-09-08 |
Family
ID=59780847
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710374654.4A Pending CN107147654A (en) | 2017-05-24 | 2017-05-24 | Group data acquisition system and data acquisition method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107147654A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109634943A (en) * | 2018-12-03 | 2019-04-16 | 珠海格力电器股份有限公司 | Air conditioner data transmission method and device for screening data and computer equipment |
| CN109855246A (en) * | 2019-03-15 | 2019-06-07 | 珠海格力电器股份有限公司 | Display method and device of air conditioner operation parameters and computer equipment |
| CN112583916A (en) * | 2020-12-11 | 2021-03-30 | 珠海格力电器股份有限公司 | Air conditioning unit and control method thereof |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104216963A (en) * | 2014-08-22 | 2014-12-17 | 南京邮电大学 | Mass network management data collection and storage method based on HBase |
| CN104850640A (en) * | 2015-05-26 | 2015-08-19 | 华北电力大学（保定） | HBase based storage and query method and system for power equipment status monitoring data |
| CN105629790A (en) * | 2016-01-29 | 2016-06-01 | 广州能迪能源科技股份有限公司 | Data management platform and method based on industrial IOT (Internet of Things) |
| CN105760459A (en) * | 2016-02-04 | 2016-07-13 | 四川嘉宝资产管理集团股份有限公司 | Distributed data processing system and method |
-
2017
- 2017-05-24 CN CN201710374654.4A patent/CN107147654A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104216963A (en) * | 2014-08-22 | 2014-12-17 | 南京邮电大学 | Mass network management data collection and storage method based on HBase |
| CN104850640A (en) * | 2015-05-26 | 2015-08-19 | 华北电力大学（保定） | HBase based storage and query method and system for power equipment status monitoring data |
| CN105629790A (en) * | 2016-01-29 | 2016-06-01 | 广州能迪能源科技股份有限公司 | Data management platform and method based on industrial IOT (Internet of Things) |
| CN105760459A (en) * | 2016-02-04 | 2016-07-13 | 四川嘉宝资产管理集团股份有限公司 | Distributed data processing system and method |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109634943A (en) * | 2018-12-03 | 2019-04-16 | 珠海格力电器股份有限公司 | Air conditioner data transmission method and device for screening data and computer equipment |
| CN109855246A (en) * | 2019-03-15 | 2019-06-07 | 珠海格力电器股份有限公司 | Display method and device of air conditioner operation parameters and computer equipment |
| CN109855246B (en) * | 2019-03-15 | 2020-08-07 | 珠海格力电器股份有限公司 | Display method and device of air conditioner operation parameters and computer equipment |
| CN112583916A (en) * | 2020-12-11 | 2021-03-30 | 珠海格力电器股份有限公司 | Air conditioning unit and control method thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106534333B (en) | A kind of two-way choice calculating discharging method based on MEC and MCC | |
| CN106850188B (en) | A kind of data transmission system based on multichannel isomery one-way transmission path | |
| CN108769121A (en) | Intelligent industrial equips the method for uploading of internet of things data acquisition system and gathered data | |
| CN106130796B (en) | SDN network topology traffic visualization monitoring method and control terminal | |
| CN104780210B (en) | Load-balancing method and device | |
| CN107347205B (en) | A kind of network slice selection method, apparatus and system | |
| CN107147654A (en) | Group data acquisition system and data acquisition method | |
| CN108924233A (en) | Electric power collection copies group system and method | |
| CN106302017B (en) | The small capaciated flow network velocity-measuring system of high concurrent and method | |
| CN105391634B (en) | A kind of message processing method, device and interchanger | |
| CN104301244B (en) | A kind of cluster communication system and method for large size distribution network system | |
| CN107404526A (en) | P2P remotely-sensed data network self-adapting dissemination systems and method based on SDN | |
| CN111600967A (en) | Access pressure-based load balancing system among block chain nodes | |
| CN108833267A (en) | Intelligent industrial gateway | |
| CN106797335A (en) | Data transmission method, data transmission device, electronic equipment and computer program product | |
| CN108243146A (en) | A kind of efficient information submits method | |
| CN111817917B (en) | Deep packet inspection method, device, server and storage medium | |
| CN109003005A (en) | A kind of green energy resource distribution method of the data back towards edge calculations | |
| CN105656709A (en) | Method and apparatus for predicting capacity of packet domain network | |
| CN104468190B (en) | A kind of wifi data packet snapping method, device and intelligent terminal | |
| CN109600402A (en) | Server load balancing method and device, storage medium, server, service system | |
| CN106210107A (en) | Animal heat acquisition method and system | |
| CN113949417B (en) | Power grid data transmission method and device based on hybrid communication | |
| CN115801828A (en) | Intelligent substation secondary data analysis system based on platform and terminal | |
| CN206962848U (en) | A kind of information O＆M device for mass data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170908 |