CN103235820A - Data storage method and device in cluster system - Google Patents
Data storage method and device in cluster system Download PDFInfo
- Publication number
- CN103235820A CN103235820A CN2013101535095A CN201310153509A CN103235820A CN 103235820 A CN103235820 A CN 103235820A CN 2013101535095 A CN2013101535095 A CN 2013101535095A CN 201310153509 A CN201310153509 A CN 201310153509A CN 103235820 A CN103235820 A CN 103235820A
- Authority
- CN
- China
- Prior art keywords
- data
- server
- mapping relation
- initial mapping
- relation table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The invention discloses a data storage method and device in a cluster system. The method comprises analyzing acquired server operation data; determining corresponding initial mapping relation tables according to analyzed server identification (ID) if the analyzed data are acquired based on an intelligent platform management interface (IPMI) protocol; executing a conversion step of data acquired based on the IPMI protocol to eradicate data isomerism; building a target mapping relation table between values of all acquired sensors and table field names; and generating structured query language (SQL) sentences to store the acquired server operation data in a data bank according to the target mapping relation table. According to the data storage method and device, the data isomerism is eradicated by setting of the initial mapping relation tables of different servers and building of the server target mapping relation table, and accordingly, operation data of all servers are stored integrally, and unified management and maintenance of a cluster are facilitated.
Description
Technical field
The present invention relates to the trunking communicating technology field, relate to date storage method and device in a kind of group system in particular.
Background technology
In order to obtain whether normally operation of server cluster in real time, need monitor server cluster, to server cluster monitor mainly comprise according to communication protocol acquisition server service data (be the encapsulation data, be convenient to transmission), whether the server service data of storage of collected is normal according to the server Operational Data Analysis cluster operation of storage.Wherein said service data further can comprise status data and performance data, and described status data refers to the data that some numerical value are fixing, such as the CPU model; Described performance data can comprise the data of some dynamic changes such as CPU usage, network traffics use information.
When the acquisition server service data, if the server model difference in the cluster has identical dimension based on the data of snmp protocol collection, can unify storage administration.But mainly by sensor image data in the server, the data dimension of collection is generally different based on the IPMI agreement, title difference such as sensor in the server, sensor kind difference, these can cause the isomerism of image data, and data store difficulty.What Internet firm adopted at present all is the IPMI management system that is provided by server production firm, come from a plurality of manufacturers if form the server of cluster, so, just have many cover IPMI management systems when server cluster is monitored, namely different model server service data is kept in the different databases.When the maintainer will detect the running status of cluster by the service data of Servers-all, need between different IPMI management systems, carry out transferring and comparison of data, when cluster scale is big, the server service data amount of storage is bigger, make that the maintenance difficulties of cluster is bigger, efficient is lower.
Summary of the invention
The embodiment of the present application provides date storage method and device in a kind of group system, to solve in the cluster that is constituted by the different model server in the prior art, the server service data exists isomerism can't unify storage, the technical matters that the management maintenance of the cluster integral body that causes disperses, for the data with isomerism provide a kind of unified data storage method, the server service data of all models is stored in same the table of database, improves the concentration degree of server service data and the maintenance efficiency of cluster integral body.
For solving the problems of the technologies described above, the invention provides date storage method and device in a kind of group system, the invention provides following technical scheme:
Date storage method in a kind of group system, described method comprises:
The server service data of gathering is resolved;
If the data that parse are based on the data that intelligent platform management IPMI agreement is gathered, determine the initial mapping relation table of described server correspondence according to the server identification ID that parses, described initial mapping relation table has recorded the mapping relations between at least one sensor name in the described server and the table field name, uses identical table field name in the initial mapping table of different server correspondence;
The data step of converting that execution is gathered based on the IPMI agreement is to remove the data isomerism, described data step of converting based on the collection of IPMI agreement comprises: judge whether all that parse are had record by the title of pick-up transducers at described initial mapping relation table, if then set up by the target mapping relations table between the numerical value of pick-up transducers and the table field name according to the initial mapping relation table; If not, in described initial mapping relation table, increase new table field name, and set up described new table field name and Unrecorded by the mapping relations between the pick-up transducers title; Carry out and describedly set up by the numerical value of pick-up transducers and the step of showing the target mapping relations table between the field name according to the initial mapping relation table;
According to described target mapping relations table, generate SQL statement and deposit described data based on the collection of IPMI agreement in database.
Preferably, the server identification ID that parses of described basis determines that the initial mapping relation table of described server correspondence comprises:
Determine the server model sign according to described server identification ID in the mode of database retrieval;
Determine the initial mapping relation table of described server correspondence according to described server model sign.
Preferably, described initial mapping relation table is set up in the mode of hash index table, then determine that according to the model of described server the initial mapping table of described server correspondence comprises: according to the server model corresponding identification, search the described initial mapping relation table of storage in advance by hash mode.
Preferably, described server service data to collection is resolved, and comprising:
Obtain the formation of being formed by the readable documents descriptor according to the Epoll mode;
From readable documents descriptor queue, obtain the filec descriptor of queue heads indication;
Filec descriptor according to the indication of described queue heads obtains server operation raw data, comprising: read the byte of default number from Socket, the length of the server service data of describing with the filec descriptor of determining described queue heads indication; According to established data length, from the file that described filec descriptor is described, read the data of this length value; From the data of described length, read the raw data of server according to the preset data form.
Preferably, if the data that parse are based on the data that the simple network management snmp protocol is gathered, carry out the storing step of the data of gathering based on snmp protocol;
The storing step of described data based on the snmp protocol collection comprises: set up the data that parse and the mapping relations table between the described data corresponding parameters title that parses; According to the status data in the described mapping relations table, upgrade the value of described status data in database by SQL statement; Deposit the performance data in the described mapping relations table in database by SQL statement.
Preferably, described database is relevant database.
The present invention also provides data storage device in a kind of group system, it is characterized in that, described device comprises:
Data resolution module is used for the server service data of gathering is resolved;
Initial mapping table acquisition module, when the data that are used for parsing are based on the data of IPMI agreement collection, determine the initial mapping relation table of described server correspondence according to the server identification ID that parses, described initial mapping relation table has recorded the mapping relations between at least one sensor name in the described server and the table field name, uses identical table field name in the initial mapping table of different server correspondence;
First execution module, be used for carrying out the conversion of the data of gathering based on the IPMI agreement to remove the isomerism of data, described first execution module comprises: judge submodule, be used for judging whether all are had record by the title of pick-up transducers at described initial mapping relation table; Generate submodule, the result who is used for described judgement submodule sets up by the numerical value of pick-up transducers and shows target mapping relations table between the field name according to the initial mapping relation table when being; The newly-increased module of table field name, the result who is used for described judgement submodule increase new table field name, and set up described new table field name and Unrecorded by the mapping relations between the pick-up transducers title for not the time in described initial mapping relation table; Trigger submodule, be used for triggering described generation submodule.
Data memory module is used for according to described target mapping relations table, generates SQL statement and deposits described data based on the collection of IPMI agreement in database.
Preferably, described initial mapping table acquisition module comprises:
The server model sign is determined submodule, is used for determining the server model sign according to the server identification ID that parses in the mode of database retrieval;
The initial mapping relation table is determined submodule, is used for determining according to described server model sign the initial mapping relation table of described server correspondence.
Preferably, described initial mapping relation table is set up in the mode of hash index table, and then the initial mapping relation table is determined submodule, concrete being used for according to the server model corresponding identification, searches the described initial mapping relation table of storage in advance by hash mode.
Preferably, described data resolution module comprises:
Descriptor queue obtains submodule, is used for obtaining the formation of being made up of the readable documents descriptor according to the Epoll mode;
The descriptor reading submodule is used for obtaining the filec descriptor that queue heads is indicated from readable documents descriptor queue;
The raw data reading submodule, being used for filec descriptor according to described queue heads indication obtains server operation raw data and specifically comprises: read the byte of default number from Socket, the length of the server service data of describing with the filec descriptor of determining described queue heads indication; According to established data length, from the file that described filec descriptor is described, read the data of this length value; From the data of described length, read the raw data of server according to the preset data form.
Preferably, described device also comprises:
When second execution module, the data that are used for parsing are based on the data of snmp protocol collection, carry out the storage of the data of gathering based on snmp protocol;
Described second execution module comprises: the mapping relations table is set up submodule, when the data that are used for parsing are based on the data of snmp protocol collection, and the data that foundation parses and the mapping relations table between the described data corresponding parameters title; The status data sub module stored is used for the status data according to described mapping relations table, upgrades the value of described status data in database by SQL statement; The performance data sub module stored is used for the performance data with described mapping relations table, deposits database in by SQL statement.
In the technical scheme of the present invention, each server has corresponding sensor title and the initial mapping relation table of table between the field name in the cluster, sets up target mapping relations table according to the numerical value of the sensor of described initial relation mapping table and collection then.What record in the described target mapping relations table is table field name and by the relation between the numerical value of pick-up transducers, by initial mapping relation table and target mapping relations table more as can be known, mapping relations between sensor value corresponding and the table field name are only arranged in the target mapping relations table that obtain this moment, do not had sensor name.Simultaneously, the literary name section name of using in the initial mapping table for each server correspondence among the present invention is identical, and so far, the data that record in the target mapping relations table have not had isomerism.Regard parameter name as if will show field name, corresponding sensor numerical value is exactly parameter value, and this is a kind of simple one-to-one relationship, when storage, as long as set up a table with above-mentioned table field name, leave the collection value of correspondence in corresponding literary name section then and get final product under one's name.Only for different servers, the physical significance of parameter value representative may be different.Target mapping relations table has removed by the different data isomerisms that bring with number of sensor name, at this moment, just can adopt unified SQL statement to deposit the server service data of described collection in database.
The uniform data of different model server based on the IPMI agreement can be stored in the table by technical scheme of the present invention, for the maintainer, when needing the running status of monitoring cluster integral body, only need in same table, carry out reading and comparison of data, even cluster scale is bigger, searching the speed that data compare in same table also is faster, has reduced the maintenance difficulties of cluster, has improved the maintenance efficiency of cluster.
Description of drawings
In order to be illustrated more clearly in the embodiment of the present application or technical scheme of the prior art, to do to introduce simply to the accompanying drawing of required use in embodiment or the description of the Prior Art below, apparently, the accompanying drawing that describes below only is some embodiment that put down in writing among the application, for those of ordinary skills, under the prerequisite of not paying creative work, can also obtain other accompanying drawing according to these accompanying drawings.
Fig. 1 is the process flow diagram of date storage method embodiment 1 in a kind of group system of the present invention;
Fig. 2 is the process flow diagram of date storage method embodiment 2 in a kind of group system of the present invention
Fig. 3 is the process flow diagram of date storage method embodiment 3 in a kind of group system of the present invention;
Fig. 4 is the structural representation of data storage device embodiment 1 in a kind of group system of the present invention;
Fig. 5 is the structural representation of first execution module 403 among apparatus of the present invention embodiment 1;
Fig. 6 is the structural representation of initial mapping table acquisition module 402 among apparatus of the present invention embodiment 1;
Fig. 7 is the structural representation of data storage device embodiment 2 in a kind of group system of the present invention;
Fig. 8 is the structural representation of data resolution module 401 among apparatus of the present invention embodiment 2;
Fig. 9 is a kind of structural drawing of the embodiment of the invention under the practical application scene.
Embodiment
In order to make those skilled in the art person understand the present invention program better, below in conjunction with the accompanying drawing in the embodiment of the invention, technical scheme in the embodiment of the invention is clearly and completely described, obviously, described embodiment only is the present invention's part embodiment, rather than whole embodiment.Based on the embodiment among the present invention, those of ordinary skills belong to the scope of protection of the invention not making the every other embodiment that obtains under the creative work prerequisite.
All comprise that with group system the different model server is that prerequisite describes among all embodiment of the present invention.With reference to shown in Figure 1, be the process flow diagram of date storage method embodiment 1 in a kind of group system provided by the invention, present embodiment specifically can comprise:
Step 101: resolve the server service data of gathering.
Under the practical application scene, the server service data is to collect according to certain acquisition protocols, gathered the server service data after, need the data that collect be encapsulated, encapsulation back data could be transmitted in channel.So after receiving the server service data, at first will carry out the parsing of data, with the original service data of the server that obtains collection.Parsing about data can repeat no more here with reference to the implementation of data parsing in the prior art.
Step 102: if the data that parse are based on the data that the IPMI agreement is gathered, determine the initial mapping relation table of described server correspondence according to the server identification ID that parses.
If the data that parse are based on IPMI agreement (Intelligent Platform Management Interface, IPMI) data of Cai Jiing, so there is isomerism in the data that collect based on the IPMI agreement of Dui Ying different model server, in order to remove the isomerism of data, the first step is exactly to obtain the initial mapping relation table of this server correspondence.Concrete, according to the data based on the collection of IPMI agreement that parse, can resolve the ID(sign of obtaining this server by data), obtain and this server ID corresponding server model (perhaps model ID) by the mode of database retrieval then, at last, find the initial mapping relation table of corresponding described model server by hash mode according to the server model of determining.
Preferably, described initial mapping relation table is set up in hash index table mode, for each server initial mapping relation table that can prestore, in practical operation, the initial mapping relation table of the correspondence that can prestore for the server of each model, and use identical table field name in the initial mapping relation table of different model server correspondence.Perhaps, after having resolved data, the mapping relations table between the sensor name that foundation parses and the table field name is as the initial mapping relation table at every turn.With the server of wherein two kinds of different models of comprising in the initial mapping relation table of the server correspondence of each model that prestores, the cluster for instance, for convenience of description described model is designated as first model and second model respectively, the initial mapping relation table A1 of the first model server correspondence of being set up by hash index table mode shown in the reference table 1; Table 2 is the initial mapping relation table A2 of the second model server correspondence set up by hash index table mode.
Table 1 initial mapping relation table A1
| The table field name | Sensor name | ? |
| Col1 | Name1 | Example |
| Col2 | Name2 | Example |
Table 2 initial mapping relation table A2
| The table field name | Sensor name | ? |
| Col1 | Name3 | Example |
| Col2 | Name4 | Example |
Sensor name in table 1 described here and the table 2 is with Name1, Name2, Name3 and Name4 sign.Sensor name in table 1 and the table 2 might not be the title of all the sensors that comprises in the corresponding model server, and may be the title of the several sensors in all the sensors in this concrete model server, this is that time of making the product difference by the corresponding a plurality of servers of a concrete model causes.Such as, for the server of first model, suppose that it number of sensors that comprises is 3, specifically be temperature sensor, voltage sensor and current sensor.But, might have situations such as server updating in actual use.Such as, increase new sensor on the original basis or delete certain sensor, at this moment, remain unchanged for the title of remaining original sensor, the number of sensors that same different server of the model time of making the product comprises may be different.And for the corresponding initial mapping table of the server of a model, so the title of the real sensor of each collection of server of the sensor name in this initial mapping table and this model is possible incomplete same.
Reference table 1 and table 2 as can be known, for the server of first model, second model, the literary name section name of using in its corresponding initial mapping table is identical, only same table field name corresponding sensor title difference.
Step 103: carry out the data step of converting gathered based on the IPMI agreement to remove the data isomerism, generate target mapping relations table.
Because there is isomerism in the data based on the collection of IPMI agreement, for the ease of the uniform data storage, before the storage of carrying out data, at first to carry out the conversion of data structure to data.The conversion process of concrete data structure is as follows:
After parsing the original service data of server, set up the mapping relations table (form with the data Hash table in the present embodiment provides) between the sensor name and corresponding collection value in the server of actual acquisition according to the data that parse.For the ease of understanding, be that example carry out the schematically illustrating of this step for certain of corresponding second model by the data Hash table B2 of acquisition server by the data Hash table B1 of acquisition server, table 4 with certain of corresponding first model of table 3.
Table 3 data Hash table B1
| The sensor name | The collection value | ? |
| Name1 | 10 | Example |
| Name2 | 20 | Example |
Table 4 data Hash table B2
| The sensor name | The collection value | ? |
| Name3 | 30 | Example |
| Name4 | 40 | Example |
| Name5 | 50 | ? |
After obtaining the data Hash table of server correspondence, judge whether the sensor name set that comprises in the table 1 comprises the sensor name set that comprises in the table 3, whether all the sensors title that comprises in the table 3 in other words all exists in table 1, if then be called table 3 and covered fully by table 1.If table 3 is covered fully by table 1, directly carry out the merging of table 1 and table 3, set up the target mapping table between table field name and the collection value, server for described first model, shown in the reference table 5, be the target mapping relations table that obtains after table 1 and table 3 merge, i.e. the target mapping relations table of certain server of being gathered of first model.
The target mapping relations table of the table 5 first model server
| The table field name | The collection value | ? |
| Col1 | 10 | Example |
| Col2 | 20 | Example |
For the described server of being gathered of first model, its initial mapping table and data Hash table cover fully, merge fairly simple.From table 2 and table 4 as can be seen, for certain server of second model of being gathered, initial mapping table and data Hash table that it is corresponding do not cover fully.In this case, set up and the newly-increased mapping relations of showing between the field name for the sensor name that is not comprised by the initial mapping relation table.In practical operation, because there is unpredictability in server updating, so can set up the set of a table field name, when running into the sensor name that the initial mapping relation table do not comprise, obtain from described table field name set that the table field name that is not used is set up and the sensor name that do not comprise between mapping relations.Such as, use the collection value of the corresponding Name5 of Col3, so, shown in the reference table 6, the target mapping relations table of certain server of gathering for the quilt of the correspondence that obtains and second model.
The target mapping relations table of table 6 second model
| The table field name | The collection value | ? |
| Col1 | 30 | Example |
| Col2 | 40 | Example |
| Col3 | 50 | Example |
In the described step 103 be a server of gathering be that example describes, when other server datas were handled, principle was identical.
Step 104: generate SQL statement according to described target mapping relations table and deposit described data based on the collection of IPMI agreement in database.
After generating target mapping relations table, deposit described data based on the collection of IPMI agreement in database by generating SQL statement.Wherein, the SQL statement of data correspondence comprises that insert inserts statement and update upgrades statement in the target mapping relations table, concrete, state type data in the target mapping relations table are saved in database by upgrading statement, performance data is saved in database by inserting statement.
Insert statement form:
insert?into?table_name(x1,x2,x3,x4,...)values(y1,y2,y3,y4,...)
Xn wherein, yn table field name and the collection value in the corresponding target mapping relations table respectively are right.
Update statement form:
update?table_name?set?x1=y1,x2=y2,x3=y3,...where...
Xn wherein, yn table field name and the collection value in the corresponding target mapping relations table respectively are right, and traversal target mapping relations table gets final product during data loading, and time complexity is O (N).
In the technical scheme of present embodiment, each server has corresponding sensor title and the initial mapping relation table of table between the field name in the cluster, sets up target mapping relations table according to the numerical value of the sensor of described initial relation mapping table and collection then.What record in the described target mapping relations table is table field name and by the relation between the numerical value of pick-up transducers, by initial mapping relation table and target mapping relations table more as can be known, mapping relations between sensor value corresponding and the table field name are only arranged in the target mapping relations table that obtain this moment, do not had sensor name.Simultaneously, the literary name section name of using in the initial mapping table for each server correspondence among the present invention is identical, and so far, the data that record in the target mapping relations table have not had isomerism.Regard parameter name as if will show field name, corresponding sensor numerical value is exactly parameter value, and this is a kind of simple one-to-one relationship, when storage, as long as set up a table with above-mentioned table field name, leave the collection value of correspondence in corresponding literary name section then and get final product under one's name.Only for different servers, the physical significance of parameter value representative may be different.Target mapping relations table has removed by the different data isomerisms that bring with number of sensor name, at this moment, just can adopt unified SQL statement to deposit the server service data of described collection in database.
The uniform data of different model server based on the collection of IPMI agreement can be stored in the table by technical scheme of the present invention, for the maintainer, when needing the running status of monitoring cluster integral body, only need in same table, carry out reading and comparison of data, even cluster scale is bigger, searching the speed that data compare in same table also is faster, has reduced the maintenance difficulties of cluster, has improved the maintenance efficiency of cluster.
Among the described embodiment 1 the data storage based on the collection of IPMI agreement is described, when being resolved, data adopt existing data analytic technique, when but available data is resolved, be that the service data that will receive in real time is placed directly in buffer memory in the formation, when needing resolution data, from data queue, directly read data.Because data are buffer memory in formation directly, can take more operating-system resources, and capacity of queue is limited, and is not suitable for the storage of high concurrent data.In order to solve the storage of high concurrency data, the invention provides date storage method embodiment 2 in a kind of group system, adopt a kind of new data analysis mode, for high concurrency data provide storage mode.Be to provide a kind of storage mode efficiently based on IPMI the agreement data of gathering and the service data of gathering based on snmp protocol (Simple Network Management Protocol, Simple Network Management Protocol) simultaneously.
With reference to shown in Figure 2, process flow diagram for date storage method embodiment 2 in a kind of group system provided by the invention, on the basis of embodiment 1, the example that is stored as with the data of snmp protocol collection in the present embodiment is elaborated, though under the background of the data of snmp protocol collection, introduce new data analytical form, but only being for analysis mode is described, is not that this analysis mode only is applicable to the data that snmp protocol is gathered.With reference to the step in above-described embodiment, present embodiment repeats no more for the implementation of step same as the previously described embodiments.Present embodiment specifically can comprise:
Step 201: the initial mapping relation table of in database, storing every kind of model server correspondence in advance.
Initial mapping relation table described here mainly is to arrange at the data of gathering based on the IPMI agreement.
Step 202: the server service data of gathering is resolved.
Adopt the Epoll interface to receive data in the present embodiment, the Epoll interface is the enhancing version of multiplexed IO interface select/poll under the Linux, its program that can significantly improve has only the system CPU utilization factor under a small amount of situation about enlivening in a large amount of concurrent connections, because the multiplexing filec descriptor of its meeting is gathered to transmit the result and need not force the filec descriptor that all must prepare again to be intercepted before the each waiting event of developer to be gathered; Another reason is exactly when obtaining event, and it need not travel through whole descriptor set of being intercepted, as long as those descriptors that added formation by asynchronous the waking up of kernel IO event of traversal have been gathered.Described formation is the linear list of a kind of " first in first out " (FIFO-first in first out).
Receive the mode of data based on Epoll, the data of carrying out are resolved detailed process: the mode by Epoll at first, from system, obtain the readable filec descriptor of current time, and the readable documents descriptor that obtains put into readable documents descriptor queue, read the readable documents descriptor in the queue heads.And from corresponding file, read the original service data of server (the not server service data before the encapsulation) according to the readable documents descriptor in the queue heads.When obtaining the original service data of server according to filec descriptor, at first from Socket, read a default byte, what store in the described default byte is the length of encapsulation of data in the file represented of the readable documents descriptor in the described queue heads; According to this length value, from this document descriptor, read the data of this length in the corresponding file, from the data of described length, understand the original service data of server according to the form of communicating pair agreement then.When carrying out the data parsing, the readable documents descriptor that all reads the queue heads correspondence carries out data and resolves at every turn.If the data that parse are based on the data of IPMI agreement, the data storage method described in the reference example 1 is stored, if the data that parse are based on the data of snmp protocol, enters step 203.
Step 203: determine based on the data of snmp protocol collection and the mapping relations table between the described data corresponding parameters title.
If the data that parse are based on the data that snmp protocol is gathered, at first need to determine based on the data of snmp protocol collection and the mapping relations table between the described data corresponding parameters title.The data that just can enter step 204 after the mapping relations table is determined have been stored.
Step 204: store status data into relevant database.
For the status data that collects, its numerical value that repeatedly collects may be identical, so the status data that there is no need to gather all stores separately at every turn, needs only the status data in each more new database.In the practical operation, the status data based on the snmp protocol collection can be upgraded the value of these state type data in database by SQL statement.
Step 205: store performance data into described relevant database.
Preferably, can from described mapping relations table, filter described status data, deposit remaining performance data in database by SQL statement, namely finish the storage based on the snmp protocol image data.The SQL statement form that uses when storing based on the data loading of IPMI agreement collection is applicable to the storage of the data that snmp protocol is gathered.In practical operation, status data can separate with performance data and is stored in the different tables, separately is stored in the different tables with data based on the snmp protocol collection based on the data of IPMI agreement collection simultaneously.
As shown in the above, simple more than the data storage of gathering based on the IPMI agreement based on the data storage that snmp protocol is gathered, because be identical for the server of different model based on the data dimension of snmp protocol collection, namely the number of parameters of Cai Jiing is identical with parameter name, as long as the mode that has this moment parameter name and the mapping relations of table between the field name just can adopt Hash table to merge is obtained the collection value and shown target Hash table between the field name, and be identical for the server corresponding parameters title of different model with the mapping relations of showing between the field name.
In the technical scheme of present embodiment, used a kind of data analysis mode unlike the prior art, read the original service data of server by filec descriptor, because the operating file descriptor does not take operating-system resources, but application program itself has reduced the influence to operating system.In addition, what place in the formation is filec descriptor, if high concurrency data are arranged, can not take too much resource as long as the data corresponding descriptor is buffered in the formation this moment, effectively solved the storage problem of high concurrent data.
When in the prior art cluster being monitored, the data that collect are generally used RRD(Round Robin Database, ring-type database) store, so-called RRD resemble a periphery indicate a little annulus--these points are exactly the position of storing according to the time.Store data at an annulus, do not have starting point and terminal point.After long-time, data new among the RRD can cover legacy data, and namely the persistence of data is relatively poor.Accordingly, adopt relevant database to store data in the present embodiment, can keep the persistence of data.
Though for described separately by different embodiment with the data storage based on the snmp protocol collection based on the collection of IPMI agreement, in actual applications, be simultaneous, because for a server, some the time, should be based on the snmp protocol image data, simultaneously also will be based on IPMI agreement image data, now embodiment 1 is integrated with the flow process of embodiment 2, with reference to shown in Figure 3, be the complete realization flow figure of data storage in the cluster, wherein the realization of correlation step can reference example 1 and embodiment 2 in the implementation of correlation step.
Corresponding to method embodiment, corresponding, the present invention also provides data storage device in a kind of group system, with reference to shown in Figure 4, is the structural representation of data storage device embodiment 1 in a kind of group system provided by the invention, and described device comprises:
Initial mapping table acquisition module 402, when the data that are used for parsing are based on the data of IPMI agreement collection, determine the initial mapping relation table of described server correspondence according to the server identification ID that parses, described initial mapping relation table has recorded the mapping relations between at least one sensor name in the described server and the table field name, uses identical table field name in the initial mapping table of different server correspondence;
Wherein, preferred, with reference to shown in Figure 5, be the structural representation of described first execution module 403, described first execution module 403 comprises:
Generate submodule 502, the result who is used for described judgement submodule sets up by the numerical value of pick-up transducers and shows target mapping relations table between the field name according to the initial mapping relation table when being;
The newly-increased submodule 503 of table field name is used for increasing new table field name at described initial mapping relation table, and sets up described new table field name and Unrecorded by the mapping relations between the pick-up transducers title;
Trigger submodule, be used for triggering described generation submodule.
Preferably, with reference to shown in Figure 6, be the structural representation of described initial mapping table acquisition module 402, described initial mapping relation table acquisition module 402 comprises:
The server model sign is determined submodule 601, is used for determining the server model sign according to the server identification that parses in the mode of database retrieval;
The initial mapping relation table is determined submodule 602, is used for determining according to described server model sign the initial mapping relation table of described server correspondence.
Described initial mapping relation table is set up in hash index table mode, and then the initial mapping relation table is determined submodule 603, concrete being used for according to the server model corresponding identification, searches the described initial mapping relation table of storage in advance by hash mode.
The function that each module realizes in the device is corresponding with method operation steps among the method embodiment 1, repeats no more here.
The structural representation of data storage device embodiment 2 in a kind of group system provided by the invention, with reference to shown in Figure 7, be the structural representation of data storage device embodiment 2 in a kind of group system provided by the invention, except module shown in Figure 4, described device can also comprise:
When second execution module 701, the data that are used for parsing are based on the data of snmp protocol collection, carry out the storage of the data of gathering based on snmp protocol;
Described second execution module 701 comprises: the mapping relations table is set up submodule 7011, when the data that are used for parsing are based on the data of snmp protocol collection, and the data that foundation parses and the mapping relations table between the described data corresponding parameters title;
Status data sub module stored 7012 is used for the status data according to described mapping relations table, upgrades the value of described status data in database by SQL statement;
Performance data sub module stored 7013 is used for the performance data with described mapping relations table, deposits database in by SQL statement.
Preferably, with reference to shown in Figure 8, be described data resolution module 401 structural representation in the present embodiment, described data resolution module 401 comprises:
Descriptor queue obtains submodule 801, is used for obtaining the formation of being made up of the readable documents descriptor according to the Epoll mode;
Raw data reading submodule 803, being used for filec descriptor according to described queue heads indication obtains server operation raw data and specifically comprises: read the byte of default number from Socket, the length of the server service data of describing with the filec descriptor of determining described queue heads indication; According to established data length, from the file that described filec descriptor is described, read the data of this length value; From the data of described length, read the raw data of server according to the preset data form.
The function that each module realizes in the device is corresponding with method operation steps among the method embodiment 2, repeats no more here.
With reference to shown in Figure 9, be a kind of realization of the embodiment of the invention under the practical application scene, main thread is put into readable documents descriptor queue (shown in the A1) by Epoll with available filec descriptor.The thread that contains some in the worker thread pond is responsible for taking out the filec descriptor (shown in the A2) in the queue heads from readable documents descriptor queue, therefrom read data, resolution data and storage data (shown in the A3), finishes the unified storage of IPMI data.In the design of database, according to the ageing separately storage of difference of image data.Be each last look of opening each field of form stores of table design simultaneously, can get access to the last look of every data during accessing database rapidly.
Need to prove, term " comprises ", " comprising " or its any other variant are intended to contain comprising of nonexcludability, thereby make and comprise that process, method, article or the equipment of a series of key elements not only comprise those key elements, but also comprise other key elements of clearly not listing, or also be included as the intrinsic key element of this process, method, article or equipment.Do not having under the situation of more restrictions, the key element that is limited by statement " comprising ... ", and be not precluded within process, method, article or the equipment that comprises described key element and also have other identical element.
For system embodiment, because it corresponds essentially to method embodiment, so relevant part gets final product referring to the part explanation of method embodiment.System embodiment described above only is schematic, wherein said unit as the separating component explanation can or can not be physically to separate also, the parts that show as the unit can be or can not be physical locations also, namely can be positioned at a place, perhaps also can be distributed on a plurality of network element.Can select wherein some or all of module to realize the purpose of present embodiment scheme according to the actual needs.Those of ordinary skills namely can understand and implement under the situation of not paying creative work.
The above only is the specific embodiment of the present invention; should be pointed out that for those skilled in the art, under the prerequisite that does not break away from the principle of the invention; can also make some improvements and modifications, these improvements and modifications also should be considered as protection scope of the present invention.
Claims (11)
1. date storage method in the group system is characterized in that described method comprises:
The server service data of gathering is resolved;
If the data that parse are based on the data that intelligent platform management IPMI agreement is gathered, determine the initial mapping relation table of described server correspondence according to the server identification ID that parses, described initial mapping relation table has recorded the mapping relations between at least one sensor name in the described server and the table field name, uses identical table field name in the initial mapping table of different server correspondence;
The data step of converting that execution is gathered based on the IPMI agreement is to remove the data isomerism, described data step of converting based on the collection of IPMI agreement comprises: judge whether all that parse are had record by the title of pick-up transducers at described initial mapping relation table, if then set up by the target mapping relations table between the numerical value of pick-up transducers and the table field name according to the initial mapping relation table; If not, in described initial mapping relation table, increase new table field name, and set up described new table field name and Unrecorded by the mapping relations between the pick-up transducers title; Carry out and describedly set up by the numerical value of pick-up transducers and the step of showing the target mapping relations table between the field name according to the initial mapping relation table;
According to described target mapping relations table, generate SQL statement and deposit described data based on the collection of IPMI agreement in database.
2. method according to claim 1 is characterized in that, the server identification ID that described basis parses determines that the initial mapping relation table of described server correspondence comprises:
Determine the server model sign according to described server identification ID in the mode of database retrieval;
Determine the initial mapping relation table of described server correspondence according to described server model sign.
3. method according to claim 2, it is characterized in that, described initial mapping relation table is set up in the mode of hash index table, then determine that according to the model of described server the initial mapping table of described server correspondence comprises: according to the server model corresponding identification, search the described initial mapping relation table of storage in advance by hash mode.
4. method according to claim 1 is characterized in that, described server service data to collection is resolved, and comprising:
Obtain the formation of being formed by the readable documents descriptor according to the Epoll mode;
From readable documents descriptor queue, obtain the filec descriptor of queue heads indication;
Filec descriptor according to the indication of described queue heads obtains server operation raw data, comprising: read the byte of default number from Socket, the length of the server service data of describing with the filec descriptor of determining described queue heads indication; According to established data length, from the file that described filec descriptor is described, read the data of this length value; From the data of described length, read the raw data of server according to the preset data form.
5. method according to claim 1 is characterized in that, if the data that parse are based on the data that the simple network management snmp protocol is gathered, carries out the storing step of the data of gathering based on snmp protocol;
The storing step of described data based on the snmp protocol collection comprises: set up the data that parse and the mapping relations table between the described data corresponding parameters title that parses; According to the status data in the described mapping relations table, upgrade the value of described status data in database by SQL statement; Deposit the performance data in the described mapping relations table in database by SQL statement.
6. according to each described method of claim 1-5, it is characterized in that described database is relevant database.
7. data storage device in the group system is characterized in that described device comprises:
Data resolution module is used for the server service data of gathering is resolved;
Initial mapping table acquisition module, when the data that are used for parsing are based on the data of IPMI agreement collection, determine the initial mapping relation table of described server correspondence according to the server identification ID that parses, described initial mapping relation table has recorded the mapping relations between at least one sensor name in the described server and the table field name, uses identical table field name in the initial mapping table of different server correspondence;
First execution module, be used for carrying out the conversion of the data of gathering based on the IPMI agreement to remove the isomerism of data, described first execution module comprises: judge submodule, be used for judging whether all are had record by the title of pick-up transducers at described initial mapping relation table; Generate submodule, the result who is used for described judgement submodule sets up by the numerical value of pick-up transducers and shows target mapping relations table between the field name according to the initial mapping relation table when being; The newly-increased module of table field name, the result who is used for described judgement submodule increase new table field name, and set up described new table field name and Unrecorded by the mapping relations between the pick-up transducers title for not the time in described initial mapping relation table; Trigger submodule, be used for triggering described generation submodule.
Data memory module is used for according to described target mapping relations table, generates SQL statement and deposits described data based on the collection of IPMI agreement in database.
8. device according to claim 7 is characterized in that, described initial mapping table acquisition module comprises:
The server model sign is determined submodule, is used for determining the server model sign according to the server identification ID that parses in the mode of database retrieval;
The initial mapping relation table is determined submodule, is used for determining according to described server model sign the initial mapping relation table of described server correspondence.
9. device according to claim 8, it is characterized in that, described initial mapping relation table is set up in the mode of hash index table, then the initial mapping relation table is determined submodule, concrete being used for according to the server model corresponding identification, search the described initial mapping relation table of storage in advance by hash mode.
10. device according to claim 7 is characterized in that, described data resolution module comprises:
Descriptor queue obtains submodule, is used for obtaining the formation of being made up of the readable documents descriptor according to the Epoll mode;
The descriptor reading submodule is used for obtaining the filec descriptor that queue heads is indicated from readable documents descriptor queue;
The raw data reading submodule, being used for filec descriptor according to described queue heads indication obtains server operation raw data and specifically comprises: read the byte of default number from Socket, the length of the server service data of describing with the filec descriptor of determining described queue heads indication; According to established data length, from the file that described filec descriptor is described, read the data of this length value; From the data of described length, read the raw data of server according to the preset data form.
11. device according to claim 7 is characterized in that, described device also comprises:
When second execution module, the data that are used for parsing are based on the data of snmp protocol collection, carry out the storage of the data of gathering based on snmp protocol;
Described second execution module comprises: the mapping relations table is set up submodule, when the data that are used for parsing are based on the data of snmp protocol collection, and the data that foundation parses and the mapping relations table between the described data corresponding parameters title; The status data sub module stored is used for the status data according to described mapping relations table, upgrades the value of described status data in database by SQL statement; The performance data sub module stored is used for the performance data with described mapping relations table, deposits database in by SQL statement.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310153509.5A CN103235820B (en) | 2013-04-27 | 2013-04-27 | Date storage method and device in a kind of group system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310153509.5A CN103235820B (en) | 2013-04-27 | 2013-04-27 | Date storage method and device in a kind of group system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103235820A true CN103235820A (en) | 2013-08-07 |
| CN103235820B CN103235820B (en) | 2016-10-05 |
Family
ID=48883861
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310153509.5A Active CN103235820B (en) | 2013-04-27 | 2013-04-27 | Date storage method and device in a kind of group system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103235820B (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106302554A (en) * | 2016-11-08 | 2017-01-04 | 郑州云海信息技术有限公司 | A kind of socket communication means, device and storage device |
| CN106330990A (en) * | 2015-06-17 | 2017-01-11 | 阿里巴巴集团控股有限公司 | B/S structure performance monitoring analysis system and method |
| CN106529195A (en) * | 2016-12-02 | 2017-03-22 | 郑州云海信息技术有限公司 | Data acquisition and analysis method and device |
| CN106909557A (en) * | 2015-12-23 | 2017-06-30 | 中国电信股份有限公司 | The storage method and device of main memory cluster, the read method and device of main memory cluster |
| CN107169075A (en) * | 2017-05-10 | 2017-09-15 | 深圳大普微电子科技有限公司 | Data access method, storage device and the storage system of feature based analysis |
| CN107229838A (en) * | 2017-07-17 | 2017-10-03 | 成都嘉逸科技有限公司 | A kind of collecting method and data acquisition unit for medical monitoring equipment |
| CN107463618A (en) * | 2017-07-04 | 2017-12-12 | 北京奇艺世纪科技有限公司 | A kind of index creation method and apparatus |
| CN108228426A (en) * | 2018-01-19 | 2018-06-29 | 郑州云海信息技术有限公司 | A kind of method and apparatus for recording SDR information |
| CN108595121A (en) * | 2018-04-17 | 2018-09-28 | 湖北文理学院 | Date storage method and device |
| CN111274051A (en) * | 2020-01-19 | 2020-06-12 | 中信银行股份有限公司 | Data processing method and device, electronic equipment and computer readable storage medium |
| CN111538775A (en) * | 2020-04-28 | 2020-08-14 | 中车株洲电力机车有限公司 | Train running state data analysis method and system and rail transit vehicle |
| CN111738909A (en) * | 2020-06-11 | 2020-10-02 | 杭州海康威视数字技术股份有限公司 | Image generation method and device |
| CN113992658A (en) * | 2021-10-28 | 2022-01-28 | 科大讯飞股份有限公司 | Data transmission method and related device, equipment, system and storage medium |
| CN116352330A (en) * | 2023-06-01 | 2023-06-30 | 广州富士汽车整线集成有限公司 | Switching control device of heavy-load slewing mechanism and automobile processing production line system |
| CN117312435A (en) * | 2023-11-23 | 2023-12-29 | 首都信息发展股份有限公司 | Data acquisition method and device and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030055955A1 (en) * | 2001-09-20 | 2003-03-20 | Heung-For Cheng | System and method for interfacing to different implementations of the intelligent platform management interface |
| CN101212355A (en) * | 2006-12-29 | 2008-07-02 | 佛山市顺德区顺达电脑厂有限公司 | Server monitoring method |
| CN101916260A (en) * | 2010-07-21 | 2010-12-15 | 中国测绘科学研究院 | Method for establishing semantic mapping between disaster body and relational database |
-
2013
- 2013-04-27 CN CN201310153509.5A patent/CN103235820B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030055955A1 (en) * | 2001-09-20 | 2003-03-20 | Heung-For Cheng | System and method for interfacing to different implementations of the intelligent platform management interface |
| CN101212355A (en) * | 2006-12-29 | 2008-07-02 | 佛山市顺德区顺达电脑厂有限公司 | Server monitoring method |
| CN101916260A (en) * | 2010-07-21 | 2010-12-15 | 中国测绘科学研究院 | Method for establishing semantic mapping between disaster body and relational database |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106330990A (en) * | 2015-06-17 | 2017-01-11 | 阿里巴巴集团控股有限公司 | B/S structure performance monitoring analysis system and method |
| CN106330990B (en) * | 2015-06-17 | 2020-02-11 | 阿里巴巴集团控股有限公司 | Performance monitoring and analyzing system and method of B/S structure |
| CN106909557A (en) * | 2015-12-23 | 2017-06-30 | 中国电信股份有限公司 | The storage method and device of main memory cluster, the read method and device of main memory cluster |
| CN106302554A (en) * | 2016-11-08 | 2017-01-04 | 郑州云海信息技术有限公司 | A kind of socket communication means, device and storage device |
| CN106529195A (en) * | 2016-12-02 | 2017-03-22 | 郑州云海信息技术有限公司 | Data acquisition and analysis method and device |
| CN107169075A (en) * | 2017-05-10 | 2017-09-15 | 深圳大普微电子科技有限公司 | Data access method, storage device and the storage system of feature based analysis |
| CN107463618A (en) * | 2017-07-04 | 2017-12-12 | 北京奇艺世纪科技有限公司 | A kind of index creation method and apparatus |
| CN107229838A (en) * | 2017-07-17 | 2017-10-03 | 成都嘉逸科技有限公司 | A kind of collecting method and data acquisition unit for medical monitoring equipment |
| CN108228426A (en) * | 2018-01-19 | 2018-06-29 | 郑州云海信息技术有限公司 | A kind of method and apparatus for recording SDR information |
| CN108595121A (en) * | 2018-04-17 | 2018-09-28 | 湖北文理学院 | Date storage method and device |
| CN111274051A (en) * | 2020-01-19 | 2020-06-12 | 中信银行股份有限公司 | Data processing method and device, electronic equipment and computer readable storage medium |
| CN111274051B (en) * | 2020-01-19 | 2023-08-22 | 中信银行股份有限公司 | Data processing method and device, electronic equipment and computer readable storage medium |
| CN111538775A (en) * | 2020-04-28 | 2020-08-14 | 中车株洲电力机车有限公司 | Train running state data analysis method and system and rail transit vehicle |
| CN111738909A (en) * | 2020-06-11 | 2020-10-02 | 杭州海康威视数字技术股份有限公司 | Image generation method and device |
| CN111738909B (en) * | 2020-06-11 | 2023-09-26 | 杭州海康威视数字技术股份有限公司 | Image generation method and device |
| CN113992658A (en) * | 2021-10-28 | 2022-01-28 | 科大讯飞股份有限公司 | Data transmission method and related device, equipment, system and storage medium |
| CN113992658B (en) * | 2021-10-28 | 2023-09-26 | 科大讯飞股份有限公司 | Data transmission method, related device, equipment, system and storage medium |
| CN116352330A (en) * | 2023-06-01 | 2023-06-30 | 广州富士汽车整线集成有限公司 | Switching control device of heavy-load slewing mechanism and automobile processing production line system |
| CN116352330B (en) * | 2023-06-01 | 2023-08-01 | 广州富士汽车整线集成有限公司 | Switching control device of heavy-load slewing mechanism and automobile processing production line system |
| CN117312435A (en) * | 2023-11-23 | 2023-12-29 | 首都信息发展股份有限公司 | Data acquisition method and device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103235820B (en) | 2016-10-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103235820A (en) | Data storage method and device in cluster system | |
| US11138252B2 (en) | System for organizing and fast searching of massive amounts of data | |
| CN111258989B (en) | Database migration evaluation method and device, storage medium and computer equipment | |
| CN102236581B (en) | Mapping reduction method and system thereof for data center | |
| US9081829B2 (en) | System for organizing and fast searching of massive amounts of data | |
| US9081834B2 (en) | Process for gathering and special data structure for storing performance metric data | |
| US7606804B2 (en) | System and method for information management in a distributed network | |
| CN105005618A (en) | Data synchronization method and system among heterogeneous databases | |
| CN102638378B (en) | Mass storage system monitoring method integrating heterogeneous storage devices | |
| US20200257681A1 (en) | System for storing and searching big data in real-time | |
| JP6060312B2 (en) | Data organization and high-speed search | |
| US7765219B2 (en) | Sort digits as number collation in server | |
| CN111752959A (en) | Real-time database cross-database SQL interaction method and system | |
| CN104679500A (en) | Automatic generation realizing method and device for entity classes | |
| CN112699183A (en) | Data processing method, system, readable storage medium and computer equipment | |
| CN111475505B (en) | Data acquisition method and device | |
| US20090164482A1 (en) | Methods and systems for optimizing projection of events | |
| CN116501726B (en) | Information creation cloud platform data operation system based on GraphX graph calculation | |
| CN109324951A (en) | The acquisition methods and device of hard disk information in server | |
| CN116644039B (en) | Automatic acquisition and analysis method for online capacity operation log based on big data | |
| CN113553320B (en) | Data quality monitoring method and device | |
| CN112818049B (en) | Information system performance full-link analysis system and method thereof | |
| CN114238698A (en) | Database index creation method based on artificial intelligence and related equipment | |
| CN117591515A (en) | Centralized processing method and device for airplane flight test data | |
| CN109359008A (en) | The management method and device of system log |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CP02 | Change in the address of a patent holder | ||
| CP02 | Change in the address of a patent holder |
Address after: 100086 12, 1201, 3 building, 2 South Road, Haidian District Academy of Sciences, Beijing. Patentee after: Beijing Sohu New Media Information Technology Co., Ltd. Address before: 100084 Beijing Haidian District Zhongguancun East Road 1 hospital 9 building Sohu cyber building 15 Floor Patentee before: Beijing Sohu New Media Information Technology Co., Ltd. |