CN104102664A - Data processing method and system for physical machine resource information - Google Patents
Data processing method and system for physical machine resource information Download PDFInfo
- Publication number
- CN104102664A CN104102664A CN201310123750.3A CN201310123750A CN104102664A CN 104102664 A CN104102664 A CN 104102664A CN 201310123750 A CN201310123750 A CN 201310123750A CN 104102664 A CN104102664 A CN 104102664A
- Authority
- CN
- China
- Prior art keywords
- data
- information
- use amount
- accumulated value
- file system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 14
- 230000003068 static effect Effects 0.000 claims abstract description 27
- 238000000034 method Methods 0.000 claims abstract description 25
- 238000012545 processing Methods 0.000 claims abstract description 17
- 230000005540 biological transmission Effects 0.000 claims description 29
- 239000000284 extract Substances 0.000 claims description 16
- 238000005192 partition Methods 0.000 claims description 7
- 238000010586 diagram Methods 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
Abstract
The invention provides a data processing method and a data processing system for physical machine resource information. The method comprises the following steps of 1, acquiring the static information of physical machines and the dynamic information, which is represented in form of a current state value or an accumulated value at a current moment, of the physical machines; 2, re-acquiring the new dynamic information of the physical machines after 1 second, and subtracting the data of the dynamic information acquired in the previous second from data in the new dynamic information represented in form of an accumulated value to obtain information about resource utilization within a second; 3, creating a low-redundancy data table, wherein DTI is used for identifying data types, CI is used for identifying data positions, DRB is used for identifying subsequently-transmitted data items of the data, and DV is used for identifying a sequential arrangement result of values of the data items; 4, representing the static information, the new dynamic information and the information about resource utilization within a second by using the low-redundancy data table respectively, transmitting the low-redundancy data table to a destination node, repeating the steps, and acquiring the real-time resource information of an acquired node in real time by using the destination node.
Description
Technical field
The present invention relates to take numeral is the main large-scale data process field of transmitting data, relates in particular to a kind of data processing method and system of physical machine resource information.
Background technology
Extensibility is one of key challenge of extensive software systems.For large scale system, the network bandwidth is one of major limitation condition of its software systems extensibility.So optimization software system internal transmission number is that the presentation format of certificate can effectively reduce transmission volume, thus the extendability of raising software systems.
Extend markup language (Extensible Markup Language, XML) is the common language of data transmission, is to use a kind of data transmission format widely.Use XML can improve the independence between software.Yet XML defines two labels of each data item (start-tag and end-tag), and this has increased the redundant information of data greatly.Yet large-scale software systems need enhanced scalability, and the XML data representation form with more redundant information can increase transmitted data on network amount, so XML should not be used to the form of extensive software systems data transfer inside.
Chukwa:A large ?scale monitoring system.Jerome Boulon, Andy Konwinski and Runping Qi has proposed one and has operated in increase income distributed system and realized MapReduce thereon of mono-of Hadoop(: the extensive supervisory system a kind of programming model calculating for large-scale parallel).This internal system is transmitted with file, and this method is applicable to Hadoop and does post-processed, yet carry out internal system data transmittal and routing form with file, can increase unnecessary redundant information (such as, file attribute etc.When data are less, file attribute is sometimes more than useful information).
Aggregation of Real ?Time System Monitoring Data for Analyzing Large ?Scale Parallel and Distributed Computing Environments.S.
c.Engelmann, the form of the large-scale supervisory system data transfer inside that and S.L.Scott proposes is the value of the corresponding data item of a data item mark (identification, ID).The data of using this method representation than XML few a lot of redundant informations, yet for take numeral as main data, the value of data item and data item ID take essentially identical size, so the represented data of the method also have higher redundance.
Summary of the invention
For addressing the above problem, the invention provides a kind of data processing method and system of physical machine resource information, to reduce the redundant information in transmission data.
To achieve the above object of the invention, the invention provides a kind of data processing method of physical machine resource information, the method comprises:
Step 1 is obtained the multidate information that the static information of physical machine and current time represent with the form of current state value or accumulated value from the resource information of linux order and the file of proc file system;
Step 2, obtains the new multidate information of physical machine again after 1 second, the data in the new multidate information that the form with accumulated value is represented deduct the data of the multidate information obtaining upper one second, obtain the resource using information in 1 second;
Step 3, creates low-redundancy data form: DTI for identification data type, and CI is for identification data position, and DRB is used for the data item of identification data subsequent transmission, the result that DV arranges in order for the value of identification data item;
Step 4, described static information, described new multidate information and the resource using information in described 1 second are represented with low-redundancy data form respectively, then send it to destination node, repeat above-mentioned steps, obtain the real-time resource information of this destination node.
Further, the static information acquisition methods in described step 1 comprises:
Step 11: extract IP address the output information of the ifconfig order of the network card configuration information of checking providing from Linux;
Step 12, reads the cpuinfo file in proc file system, obtains check figure and the frequency of processor;
Step 13, reads the meminfo file in proc file system, obtains always size of internal memory;
Step 14, reads the partitions file in proc file system, obtains always size of disk;
Step 15,, the output information of the dmesg order of checking each unit status providing from Linux, obtain the network bandwidth;
Step 16, reads the stat file in proc file system, has been obtained from the accumulated value of the processor use amount since start;
Step 17, reads the net/dev file in proc file system, has been obtained from the accumulated value of the network bandwidth use amount since start.
Further, the multidate information acquisition methods in described step 1 comprises:
Step 111, reads the stat file in proc file system, has been obtained from the accumulated value of the processor use amount since start;
Step 112, reads the meminfo file in proc file system, obtains internal memory use amount;
Step 113, extracts disk use amount the output information of the df order of checking disk resource providing from Linux;
Step 114, reads the net/dev file in proc file system, has been obtained from the accumulated value of the network bandwidth use amount since start.
Further, described step 2 comprises:
Step 21, has deducted by the accumulated value of described processor use amount since start the processor use amount accumulated value obtaining upper one second, obtains the use amount of the processor in this second;
Step 22, has deducted by described network bandwidth use amount accumulated value since start the network bandwidth use amount accumulated value obtaining upper one second, obtains the use amount of the network bandwidth in this second.
Further, in described step 3:
If data type has N
dTIkind, use every data before
position is as DTI, wherein
represent to be not less than the smallest positive integral of m;
Described CI, uses a bit representation, if this position is 1, represents that these data are not the last item in data message, if this position is 0, represents that these data are the last item of data message;
The value of the data item in described DV is adjustable length character string, before this character string, adds the length of this character string of byte representation;
The data item number of every class data is at most N
dRB, at least N of every data that use is transmitted
dRBposition is as DRB, and each corresponding data item of described DRB, if certain position is 1, represents the value that DV comprises this corresponding data item, if this position is 0, DV does not comprise the value of this data item.
Further, in described step 3:
If the data of software systems internal transmission only have a class, DTI can be left out; If the data item of the data of each transmission is all identical, DRB can be left out.
For achieving the above object, the present invention also provides a kind of data handling system of physical machine resource information, and this system comprises:
Acquisition of information module is obtained the multidate information that the static information of physical machine and current time represent with the form of current state value or accumulated value from the resource information of linux order and the file of proc file system;
Message processing module, obtains the new multidate information of physical machine again after 1 second, the data in the new multidate information that the form with accumulated value is represented deduct the data of the multidate information obtaining upper one second, obtain the resource using information in 1 second;
Form creation module, creates low-redundancy data form: DTI for identification data type, and CI is for identification data position, and DRB is used for the data item of identification data subsequent transmission, the result that DV arranges in order for the value of identification data item;
Result acquisition module, described static information, described new multidate information and the resource using information in described 1 second are represented with low-redundancy data form respectively, then send it to destination node, repeat above-mentioned module, obtain the real-time resource information of this destination node.
Further, described acquisition of information module comprises:
Extract IP address module, the output information of the ifconfig order of the network card configuration information of checking providing from Linux, extract IP address;
First reads proc file system module, read the cpuinfo file in proc file system, obtain check figure and the frequency of processor, read the meminfo file in proc file system, obtain always size of internal memory, read the partitions file in proc file system, obtain always size of disk, read the stat file in proc file system, be obtained from the accumulated value of the processor use amount since start, read the net/dev file in proc file system, be obtained from the accumulated value of the network bandwidth use amount since start;
Operation dmesg command module, obtains the network bandwidth the output information of the dmesg order of checking each unit status providing from Linux.
Further, described acquisition of information module also comprises:
Second reads proc file system module, read the stat file in proc file system, be obtained from the accumulated value of the processor use amount since start, read the meminfo file in proc file system, obtain internal memory use amount, read the net/dev file in proc file system, be obtained from the accumulated value of the network bandwidth use amount since start;
Carry out df module, the output information of the df order of checking disk resource providing from Linux, extract disk use amount.
Further, it is characterized in that, described message processing module comprises:
The first processing module, has deducted by the accumulated value of described processor use amount since start the processor use amount accumulated value obtaining upper one second, obtains the use amount of the processor in this second;
The second processing module, has deducted by described network bandwidth use amount accumulated value since start the network bandwidth use amount accumulated value obtaining upper one second, obtains the use amount of the network bandwidth in this second.
Beneficial effect of the present invention is: greatly reduced the redundancy in data transmission.If the present invention is used in large-scale supervisory system, can be by reducing the Internet Transmission be produced by monitor data, thus increased the extendability of supervisory system.And the method for image data of the present invention is the data of directly utilizing operating system to provide, there is not the expense of using intermediate data sampling instrument to bring.
Below in conjunction with the drawings and specific embodiments, describe the present invention, but not as a limitation of the invention.
Accompanying drawing explanation
Fig. 1 is used complete data of data representation method representation of the present invention;
Fig. 2 is used datagram with many data of data representation method representation of the present invention;
Fig. 3 realizes and comprises a datagram with the static data of complete information by data representation method of the present invention;
Fig. 4 realizes and comprises a datagram with the dynamic data of complete information by data representation method of the present invention;
Fig. 5 realizes and comprises static data and a datagram with the dynamic data of complete information with complete information by data representation method of the present invention;
Fig. 6 realizes and comprises a datagram with the dynamic data of partial information by data representation method of the present invention;
Fig. 7 is the data processing method process flow diagram of physical machine resource information of the present invention;
Fig. 8 is the data handling system schematic diagram of physical machine resource information of the present invention.
Embodiment
Fig. 7 is the data processing method process flow diagram of physical machine resource information of the present invention.As shown in Figure 7, the method comprises:
S1 obtains the multidate information that the static information of physical machine and current time represent with the form of current state value or accumulated value from the resource information of linux order and the file of proc file system;
S2, obtains the new multidate information of physical machine again after 1 second, the data in the new multidate information that the form with accumulated value is represented deduct the data of the multidate information obtaining upper one second, obtain the resource using information in 1 second;
S3, creates low-redundancy data form: DTI for identification data type, and CI is for identification data position, and DRB is used for the data item of identification data subsequent transmission, the result that DV arranges in order for the value of identification data item;
S4, represents described static information, described new multidate information and the resource using information in described 1 second respectively with low-redundancy data form, then send it to destination node, repeats above-mentioned steps, obtains the real-time resource information of this destination node.
Further, the static information acquisition methods in described S1 comprises:
S11:, the output information of the ifconfig order of the information of checking network card configuration providing from Linux, extract IP address;
S12, reads the cpuinfo file in proc file system, obtains check figure and the frequency of processor;
S13, reads the meminfo file in proc file system, obtains always size of internal memory;
S14, reads the partitions file in proc file system, obtains always size of disk;
S15, obtains Netowrk tape the output information of the dmesg order of checking each unit status providing from Linux;
S16, reads the stat file in proc file system, has been obtained from the accumulated value of the processor use amount since start;
S17, reads the net/dev file in proc file system, has been obtained from the accumulated value of the network bandwidth use amount since start.
Further, the multidate information acquisition methods in described S1 comprises:
S111, reads the stat file in proc file system, has been obtained from the accumulated value of the processor use amount since start;
S112, reads the meminfo file in proc file system, obtains internal memory use amount;
S113,, the output information of the df order of checking disk resource providing from Linux, extract disk use amount;
S114, reads the net/dev file in proc file system, has been obtained from the accumulated value of the network bandwidth use amount since start.
Further, described S2 comprises:
S21, has deducted by the accumulated value of described processor use amount since start the processor use amount accumulated value obtaining upper one second, obtains the use amount of the processor in this second;
S22, has deducted by described network bandwidth use amount accumulated value since start the network bandwidth use amount accumulated value obtaining upper one second, obtains the use amount of the network bandwidth in this second.
Further, in described S3:
If data type has N
dTIkind, use every data before
position is as DTI, wherein
represent to be not less than the smallest positive integral of m;
Described CI, uses a bit representation, if this position is 1, represents that these data are not the last item in data message, if this position is 0, represents that these data are the last item of data message;
The value of the data item in described DV is adjustable length character string, before this character string, adds the length of this character string of byte representation;
The data item number of every class data is at most N
dRB, at least N of every data that use is transmitted
dRBposition is as DRB, and each corresponding data item of described DRB, if certain position is 1, represents the value that DV comprises this corresponding data item, if this position is 0, DV does not comprise the value of this data item.
Further, in described S3:
If the data of software systems internal transmission only have a class, DTI can be left out; If the data item of the data of each transmission is all identical, DRB can be left out.
Fig. 8 is the data handling system schematic diagram of physical machine resource information of the present invention.As shown in Figure 8, this system comprises:
Acquisition of information module 100 is obtained the multidate information that the static information of physical machine and current time represent with the form of current state value or accumulated value from the resource information of linux order and the file of proc file system;
Message processing module obtains the new multidate information of physical machine again after 200,1 seconds, the data in the new multidate information that the form with accumulated value is represented deduct the data of the multidate information obtaining upper one second, obtain the resource using information in 1 second;
Form creation module 300, creates low-redundancy data form: DTI for identification data type, and CI is for identification data position, and DRB is used for the data item of identification data subsequent transmission, the result that DV arranges in order for the value of identification data item;
Result acquisition module 400, described static information, described new multidate information and the resource using information in described 1 second are represented with low-redundancy data form respectively, then send it to destination node, repeat above-mentioned module, obtain the real-time resource information of this destination node.
Further, described acquisition of information module 100 comprises:
Extract IP address module, the output of the information ifconfig order of checking network card configuration providing from Linux, extract IP address;
First reads proc file system module, read the cpuinfo file in proc file system, obtain check figure and the frequency of processor, read the meminfo file in proc file system, obtain always size of internal memory, read the partitions file in proc file system, obtain always size of disk, read the stat file in proc file system, be obtained from the accumulated value of the processor use amount since start, read the net/dev file in proc file system, be obtained from the accumulated value of the network bandwidth use amount since start;
Operation dmesg command module, obtains the network bandwidth the dmesg order of checking each unit status providing from Linux.
Further, described acquisition of information module 100 also comprises:
Second reads proc file system module, read the stat file in proc file system, be obtained from the accumulated value of the processor use amount since start, read the meminfo file in proc file system, obtain internal memory use amount, read the net/dev file in proc file system, be obtained from the accumulated value of the network bandwidth use amount since start;
Carry out df module, the output information of the df order of checking disk resource providing from Linux, extract disk use amount.
Further, it is characterized in that, described message processing module 200 comprises:
The first processing module, has deducted by the accumulated value of described processor use amount since start the processor use amount accumulated value obtaining upper one second, obtains the use amount of the processor in this second;
The second processing module, has deducted by described network bandwidth use amount accumulated value since start the network bandwidth use amount accumulated value obtaining upper one second, obtains the use amount of the network bandwidth in this second.
Further, in described form creation module 300:
If data type has N
dTIkind, use every data before
position is as DTI, wherein
represent to be not less than the smallest positive integral of m;
Described CI, uses a bit representation, if this position is 1, represents that these data are not the last item in data message, if this position is 0, represents that these data are the last item of data message;
The value of the data item in described DV is adjustable length character string, before this character string, adds the length of this character string of byte representation;
The data item number of every class data is at most N
dRB, at least N of every data that use is transmitted
dRBposition is as DRB, and each corresponding data item of described DRB, if certain position is 1, represents the value that DV comprises this corresponding data item, if this position is 0, DV does not comprise the value of this data item.
Further, in described form creation module 300:
If the data of software systems internal transmission only have a class, DTI can be left out; If the data item of the data of each transmission is all identical, DRB can be left out.
Resource information acquisition method of the present invention towards be (SuSE) Linux OS, order and proc file system that information source provides for Linux, proc file system mainly comprises the service condition of total resources and resource, and resource service condition has been that the form with the accumulated value (as the load of CPU and transmission volume etc.) since start and current state (as the use amount of internal memory use amount and disk etc.) represents.
A data message can comprise one or more data.Use the data representation method in this invention, the expression of complete data is as Fig. 1, and the expression of a datagram with many data is as Fig. 2.
The supervisory system of a large scale system, the data of transmission are totally two classes: static data and dynamic data.The data item that every class data wherein comprise (size and the unit of supposing each data item occurrence, following numeral all represents with 10 systems) is as follows:
Static data (totally 6): the network address (10.10.10.10, account for 4 bytes), (8 of processor check figures, account for a byte), processor frequencies (2048MHZ, account for two bytes), the total size of internal memory (4192MB, account for two bytes), the total size of disk (140272MB, account for four bytes) and network interface card bandwidth (1000Mbps accounts for two bytes).
Dynamic data (totally 5): the network address (10.10.10.10 accounts for 4 bytes), processor load (123MHZ, account for two bytes), internal memory use amount (2001MB, account for two bytes), disk use amount (2013MB, account for four bytes) and network interface card bandwidth use amount (28Mbps accounts for two bytes).
Wherein processor load and network interface card bandwidth use amount are to have represented with the form of accumulated value since start, other all represent with the form of current state.Because the data of this software systems internal transmission are totally two classes, so the DTI of every data is
position, 1 represents static data, 0 represents dynamic data.DRB is 6: for static data, 6 (first to the 6th) of DRB are map network address, processor check figure, processor frequencies, the total size of internal memory, the total size of disk and network interface card bandwidth successively; For dynamic data, first 5 (first to the 5th) of DRB are map network address, processor load, internal memory use amount, disk use amount and network interface card bandwidth use amount successively.
For ease of realizing, by a byte, represent the 8th of DTI(), the 7th of CI() and first 6 of DRB().
The concrete steps of data acquisition are as follows:
Steps A: utilize the multidate information that the static information of file acquisition physical machine of number order that Linux provides and proc file system and this moment represent with the form of accumulated value;
Step B: the static information obtaining is processed into the data message of low redundancy, sends to the destination node that need to obtain this physical machine information;
Step C:1 after second from newly read the multidate information of physical machine from the file of proc file system;
Step D: the multidate information representing for the form with accumulated value, the data that step C is obtained deduct respectively the data of obtaining before 1 second, obtain the resource service condition in this 1 second;
Step e: the multidate information that take that step C the is obtained multidate information that current state is representation and step D calculate is processed into the data message of low redundancy, sends to the destination node that need to obtain this physical machine information;
Step F: repeating step C ?E, destination node just can obtain the real-time resource information of this node.
Wherein the concrete steps of steps A are:
Steps A 1: the information of checking network card configuration in operation ifconfig(Linux), extract the IP address (10.10.10.10) in output information;
Steps A 2: read the cpuinfo file in proc file system, obtain check figure (8) and the frequency (2048MHZ) of processor;
Steps A 3: read the meminfo file in proc file system, obtain the total size of internal memory (4192MB);
Steps A 4: read the partitions file in proc file system, obtain the total size of disk (140272MB);
Steps A 5: operation dmesg order (order of checking each unit status that Linux provides), obtain the network bandwidth (1000Mbps);
Steps A 6: read the stat file in proc file system, be obtained from the accumulated value of the processor use amount since start;
Steps A 7: read the net/dev file in proc file system, be obtained from the accumulated value of the network bandwidth use amount since start.
The data message that in step B, processing obtains is as Fig. 3, and wherein BF (scale-of-two is 10111111) is for representing 8 figure places of DTI, CI and DRB, and remaining represents respectively the value of each static data item.
Step C comprises again step:
Step C1: read the stat file in proc file system, be obtained from the accumulated value of the processor use amount since start;
Step C2: read the meminfo file in proc file system, obtain internal memory use amount (2001MB);
Step C3: carry out the order of checking disk resource that df(Linux provides), from this order output information, extract disk use amount (2013MB);
Step C4: read the net/dev file in proc file system, be obtained from the accumulated value of the network bandwidth use amount since start;
Step D comprises step:
Step D1: the processor use amount accumulated value that step C1 is obtained deducts the processor use amount accumulated value that upper one second, (from steps A 6 or step C1) obtained, obtains the use amount (123MHZ) of the processor in this second;
Step D2: the network bandwidth use amount accumulated value that step C4 is obtained deducts the network bandwidth use amount accumulated value that upper one second, (from steps A 7 or step C4) obtained, obtains the use amount (28Mbps) of the network bandwidth in this second;
The data message that step e processing obtains is as Fig. 4, and wherein 1F (scale-of-two is 00011111) is for representing 8 figure places of DTI, CI and DRB, and remaining represents respectively the value of each Dynamic Data Items.
If send together with the dynamic data that the static data that steps A is obtained obtains with step C, the data message these data processings being obtained is as Fig. 5, wherein FF (scale-of-two is 11111111) and 1F (scale-of-two is 00011111) are respectively and represent DTI, the CI of Static and dynamic data and front 8 figure places of DRB, the CI of static data is the 7th of 1(FF) represent to also have data (dynamic data) after these data, and the CI of dynamic data is 0(1F the 7th) represent that these data are the last item.
If destination node only needs to obtain the dynamic data of processor load, internal memory use amount and the disk use amount of this physical machine node, the datagram that step e processing obtains is as Fig. 6, and wherein 0F (scale-of-two is 00001111) is for representing 8 figure places of DTI, CI and DRB.DRB is 0F (scale-of-two is 01111), represents that the DV in the data of this data message only contains the value of the network address, processor load, internal memory use amount and disk use amount, does not comprise network interface card bandwidth use amount.
Use data representation method of the present invention need to know the data class number of software systems internal transmission and the number of the data item that every class packet contains.For value, be that digital data item need to be known this data item size.
The DTI of every data is for the type of identification data.The total N of data of a software systems internal transmission
dTIclass, use every data before
(
representing to be not less than the smallest positive integral of m) position is as DTI, represents the type of these data.
That follow DTI closely is CI, uses a bit representation.If this position is 1, represent that these data are not the last item in data message, otherwise, represent that these data are the last item of data message.
After CI, be DRB, for identifying its follow-up data item.The data item number of every class data is at most N
dRB, at least N of every data that use is transmitted
dRBposition is as DRB.Each corresponding data item of DRB.In DRB, if certain position is 1, represent the value that DV comprises this corresponding data item, otherwise DV does not comprise the value of this data item.
The result that last DV arranges the value of each data item for the order of formulating according to CI.If this value is adjustable length character string, before this character string, add the length of this character string of byte representation.
If the data of software systems internal transmission only have a class, DTI can be left out; If the data item of the data of each transmission is all identical, DRB can be left out.Thereby further reduce the redundancy of software inhouse transmission data.
Certainly; the present invention also can have other various embodiments; in the situation that not deviating from spirit of the present invention and essence thereof; those of ordinary skill in the art are when making according to the present invention various corresponding changes and distortion, but these corresponding changes and distortion all should belong to the protection domain of the appended claim of the present invention.
Claims (10)
1. a data processing method for physical machine resource information, is characterized in that, comprising:
Step 1 is obtained the multidate information that the static information of physical machine and current time represent with the form of current state value or accumulated value from the resource information of linux order and the file of proc file system;
Step 2, obtains the new multidate information of physical machine again after 1 second, the data in the new multidate information that the form with accumulated value is represented deduct the data of the multidate information obtaining upper one second, obtain the resource using information in 1 second;
Step 3, creates low-redundancy data form: DTI for identification data type, and CI is for identification data position, and DRB is used for the data item of identification data subsequent transmission, the result that DV arranges in order for the value of identification data item;
Step 4, described static information, described new multidate information and the resource using information in described 1 second are represented with low-redundancy data form respectively, then send it to destination node, repeat above-mentioned steps, obtain the real-time resource information of this destination node.
2. the data processing method of physical machine resource information as claimed in claim 1, is characterized in that, the static information acquisition methods in described step 1 comprises:
Step 11, extracts IP address the output information of the ifconfig order of the network card configuration information of checking providing from Linux;
Step 12, reads the cpuinfo file in proc file system, obtains check figure and the frequency of processor;
Step 13, reads the meminfo file in proc file system, obtains always size of internal memory;
Step 14, reads the partitions file in proc file system, obtains always size of disk;
Step 15, obtains the network bandwidth the output information of the dmesg order of checking each unit status providing from Linux;
Step 16, reads the stat file in proc file system, has been obtained from the accumulated value of the processor use amount since start;
Step 17, reads the net/dev file in proc file system, has been obtained from the accumulated value of the network bandwidth use amount since start.
3. the data processing method of physical machine resource information as claimed in claim 1, is characterized in that, the multidate information acquisition methods in described step 1 comprises:
Step 111, reads the stat file in proc file system, has been obtained from the accumulated value of the processor use amount since start;
Step 112, reads the meminfo file in proc file system, obtains internal memory use amount;
Step 113, extracts disk use amount the output information of the df order of checking disk resource providing from Linux;
Step 114, reads the net/dev file in proc file system, has been obtained from the accumulated value of the network bandwidth use amount since start.
4. the data processing method of physical machine resource information as claimed in claim 1, is characterized in that, described step 2 comprises:
Step 21, has deducted by the accumulated value of described processor use amount since start the processor use amount accumulated value obtaining upper one second, obtains the use amount of the processor in this second;
Step 22, has deducted by described network bandwidth use amount accumulated value since start the network bandwidth use amount accumulated value obtaining upper one second, obtains the use amount of the network bandwidth in this second.
5. the data processing method of physical machine resource information as claimed in claim 1, is characterized in that, in described step 3:
If data type has N
dTIkind, use every data before
position is as DTI, wherein
represent to be not less than the smallest positive integral of m;
Described CI, uses a bit representation, if this position is 1, represents that these data are not the last item in data message, if this position is 0, represents that these data are the last item of data message;
The value of the data item in described DV is adjustable length character string, before this character string, adds the length of this character string of byte representation;
The data item number of every class data is at most N
dRB, at least N of every data that use is transmitted
dRBposition is as DRB, and each corresponding data item of described DRB, if certain position is 1, represents the value that DV comprises this corresponding data item, if this position is 0, DV does not comprise the value of this data item.
6. the data processing method of physical machine resource information as claimed in claim 1, is characterized in that, in described step 3:
If the data of software systems internal transmission only have a class, DTI can be left out; If the data item of the data of each transmission is all identical, DRB can be left out.
7. a data handling system for physical machine resource information, is characterized in that, comprising:
Acquisition of information module is obtained the multidate information that the static information of physical machine and current time represent with the form of current state value or accumulated value from the resource information of linux order and the file of proc file system;
Message processing module, obtains the new multidate information of physical machine again after 1 second, the form with accumulated value is represented to the data in new multidate information deduct the data of the multidate information obtaining upper one second, obtain the resource using information in 1 second;
Form creation module, creates low-redundancy data form: DTI for identification data type, and CI is for identification data position, and DRB is used for the data item of identification data subsequent transmission, the result that DV arranges in order for the value of identification data item;
Result acquisition module, described static information, described new multidate information and the resource using information in described 1 second are represented with low-redundancy data form respectively, then send it to destination node, repeat above-mentioned module, obtain the real-time resource information of this destination node.
8. the data handling system of physical machine resource information as claimed in claim 7, is characterized in that, described acquisition of information module comprises:
Extract IP address module, the output information of the ifconfig order of the network card configuration information of checking providing from Linux, extract IP address;
First reads proc file system module, read the cpuinfo file in proc file system, obtain check figure and the frequency of processor, read the meminfo file in proc file system, obtain always size of internal memory, read the partitions file in proc file system, obtain always size of disk, read the stat file in proc file system, be obtained from the accumulated value of the processor use amount since start, read the net/dev file in proc file system, be obtained from the accumulated value of the network bandwidth use amount since start;
Operation dmesg command module, obtains the network bandwidth the output information of the dmesg order of checking each unit status providing from Linux.
9. the data handling system of physical machine resource information as claimed in claim 7, is characterized in that, described acquisition of information module also comprises:
Second reads proc file system module, read the stat file in proc file system, be obtained from the accumulated value of the processor use amount since start, read the meminfo file in proc file system, obtain internal memory use amount, read the net/dev file in proc file system, be obtained from the accumulated value of the network bandwidth use amount since start;
Carry out df module, the output information of the df order of checking disk resource providing from Linux, extract disk use amount.
10. the data handling system of physical machine resource information as claimed in claim 7, is characterized in that, described message processing module comprises:
The first processing module, has deducted by the accumulated value of described processor use amount since start the processor use amount accumulated value obtaining upper one second, obtains the use amount of the processor in this second;
The second processing module, has deducted by described network bandwidth use amount accumulated value since start the network bandwidth use amount accumulated value obtaining upper one second, obtains the use amount of the network bandwidth in this second.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310123750.3A CN104102664B (en) | 2013-04-10 | 2013-04-10 | Data processing method and system for physical machine resource information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310123750.3A CN104102664B (en) | 2013-04-10 | 2013-04-10 | Data processing method and system for physical machine resource information |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104102664A true CN104102664A (en) | 2014-10-15 |
| CN104102664B CN104102664B (en) | 2017-04-12 |
Family
ID=51670823
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310123750.3A Active CN104102664B (en) | 2013-04-10 | 2013-04-10 | Data processing method and system for physical machine resource information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104102664B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107203478A (en) * | 2017-06-21 | 2017-09-26 | 郑州云海信息技术有限公司 | A kind of method and device of data storage |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050120160A1 (en) * | 2003-08-20 | 2005-06-02 | Jerry Plouffe | System and method for managing virtual servers |
| CN102103518A (en) * | 2011-02-23 | 2011-06-22 | 运软网络科技(上海)有限公司 | System for managing resources in virtual environment and implementation method thereof |
| CN102271145A (en) * | 2010-06-04 | 2011-12-07 | 国云科技股份有限公司 | Virtual computer cluster and enforcement method thereof |
| CN102681899A (en) * | 2011-03-14 | 2012-09-19 | 金剑 | Virtual computing resource dynamic management system of cloud computing service platform |
-
2013
- 2013-04-10 CN CN201310123750.3A patent/CN104102664B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050120160A1 (en) * | 2003-08-20 | 2005-06-02 | Jerry Plouffe | System and method for managing virtual servers |
| CN102271145A (en) * | 2010-06-04 | 2011-12-07 | 国云科技股份有限公司 | Virtual computer cluster and enforcement method thereof |
| CN102103518A (en) * | 2011-02-23 | 2011-06-22 | 运软网络科技(上海)有限公司 | System for managing resources in virtual environment and implementation method thereof |
| CN102681899A (en) * | 2011-03-14 | 2012-09-19 | 金剑 | Virtual computing resource dynamic management system of cloud computing service platform |
Non-Patent Citations (1)
| Title |
|---|
| 张伟哲等: "云计算平台中多虚拟机内存协同优化策略研究", 《计算机学报》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107203478A (en) * | 2017-06-21 | 2017-09-26 | 郑州云海信息技术有限公司 | A kind of method and device of data storage |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104102664B (en) | 2017-04-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109254982B (en) | Stream data processing method, system, device and computer readable storage medium | |
| US10216960B2 (en) | Data masking systems and methods | |
| CN110287163B (en) | Method, device, equipment and medium for collecting and analyzing security log | |
| CN111683066B (en) | Heterogeneous system integration method, heterogeneous system integration device, computer equipment and storage medium | |
| CN107391770B (en) | Method, device and equipment for processing data and storage medium | |
| US11188443B2 (en) | Method, apparatus and system for processing log data | |
| CN111970353A (en) | Data processing method and device for heterogeneous equipment in Internet of things of cloud computing platform | |
| CN109194617A (en) | The automatically parsing of XML message, packaging method and device | |
| CN105868364B (en) | Byte stream-based structured data representation method | |
| CN103618733A (en) | Data filtering system and method applied to mobile internet | |
| CN104283891A (en) | Method and device for access of service function node to service link network | |
| CN104219298A (en) | Cluster system and data backup method thereof | |
| CN115567589A (en) | Compression transmission method, device, equipment and storage medium of JSON data | |
| CN109491660B (en) | Locomotive event data dynamic analysis method | |
| CN114416877A (en) | Data processing method, device and equipment and readable storage medium | |
| CN104102664A (en) | Data processing method and system for physical machine resource information | |
| KR101218087B1 (en) | Method for Extracting InputFormat for Binary Format Data in Hadoop MapReduce and Binary Data Analysis Using the Same | |
| CN117093619A (en) | Rule engine processing method and device, electronic equipment and storage medium | |
| CN109067649B (en) | Node processing method and device, storage medium and electronic equipment | |
| CN111130864A (en) | Method for collecting formatted network element data using template | |
| CN107968722B (en) | Method for converting interface control file into AFDX (avionics full Duplex switched Ethernet) network equipment configuration file | |
| CN113204683B (en) | Information reconstruction method and device, storage medium and electronic equipment | |
| KR20210000041A (en) | Method and apparatus for analyzing log data in real time | |
| CN114238335A (en) | Buried point data generation method and related equipment thereof | |
| CN113676521A (en) | Service request response method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20240320 Address after: Room 711C, Floor 7, Building A, Yard 19, Ronghua Middle Road, Daxing District, Beijing Economic-Technological Development Area, 100176 Patentee after: Beijing Zhongke Flux Technology Co.,Ltd. Guo jiahuodiqu after: Zhong Guo Address before: 100190 No. 6 South Road, Zhongguancun Academy of Sciences, Beijing, Haidian District Patentee before: Institute of Computing Technology, Chinese Academy of Sciences Guo jiahuodiqu before: Zhong Guo |