Summary of the invention
The present invention cannot be directed to the disposable operation of more library multilists for the prior art, carry out asking for integration across database statistics
Topic, proposes a kind of statistical method based on MongoDB database.
Firstly, the present invention proposes a kind of statistical method based on MongoDB database, comprising the following steps:
S100) pass through instruction definition one or more servers to be visited and/IP address of database, port numbers, user
Name, password and database collection title to be counted;
S200) client connects remote server by IP address, port numbers, username and password and accesses one or more
Database in a server;
S300 all library names of database in one or more servers) are obtained, and traverse all libraries, are obtained each
The relation on attributes data of the database collection comprising all database collection titles to be counted under library, and have identical
The corresponding relation data of database collection added up to obtain the corresponding relation data of database collection and stored and filled with storage
In setting;Wherein, the corresponding relationship include at least one of following items, data volume, file size, file size accounting,
Size of data, size of data accounting, storage size, storage size accounting, mean object size, index quantity and index space
Size;
S400) the corresponding relation data of display data library set.
Further, in the above method proposed by the present invention, step S200 further includes following sub-step:
S201) client sends access request to server and/or database using communication means;
S202) server and/or database receive access request that client is sent and judge access request whether include
Authorization message sends not awarding for the first authentication data generated with server and/or database to client if not including
Information is weighed, if comprising allowing client access server and/or database;
S203) client receives the unauthorized information from server and/or database and extracts the first certification number
According to close using the first algorithm and the first authentication data, server and/or database user name and server and/or database
Code generates the second authentication data, and the second authentication data is sent to server and/database;
S204) server and/or database receive after the second authentication data of client, using the first algorithm and
First authentication data, server and/or database user name and server and/or database password generate third authentication data,
And judge whether third authentication data is consistent with the second authentication data, and it is such as consistent, then allow client access server and/or number
Such as inconsistent according to library, then denied access is requested.
Specifically, in doing the above method that invention proposes, first algorithm is Message Digest 5.
Specifically, in doing the above method that invention proposes, the Message Digest 5 is at least one of following algorithm:
MD algorithm, SHA algorithm and MAC algorithm.
Further, in the above method proposed by the present invention, first authentication data is that server and/database generate
Random data.
Further, in the above method proposed by the present invention, the communication means includes at least following one kind: wire communication
Method, wireless communications method.
Further, in the above method proposed by the present invention, the wireless communications method is including at least following one kind: 2G,
3G, 4G, 5G and Wi-Fi.
Further, in the above method proposed by the present invention, step S300 further includes following sub-step:
One in the corresponding relationship is specified, and with specified corresponding relationship, according to the method pair of ascending or descending order
The corresponding relation data of database collection is ranked up.
Secondly, the present invention proposes a kind of statistic device based on MongoDB database, comprise the following modules:
Definition module, for passing through instruction definition one or more servers to be visited and/IP address of database, port
Number, user name, password and database collection title to be counted;
Access modules connect remote server by IP address, port numbers, username and password for client and access
Database in one or more servers;
Module is obtained, for obtaining all library names of the database in one or more servers, and traverses all libraries,
The relation on attributes data of the database collection comprising all database collection titles to be counted under each library are obtained, and are had
The identical corresponding relation data of database collection is added up to obtain the corresponding relation data of database collection and storage
In storage device;Wherein, the corresponding relationship includes at least one of following items, and data volume, file size, file are big
Small accounting, size of data, size of data accounting, storage size, storage size accounting, mean object size, index quantity and
Index space size;
Display module, the corresponding relation data for display data library set.
Finally, the present invention proposes a kind of computer readable storage medium, it is stored thereon with computer instruction, described instruction is used
In the above-mentioned method of execution.
Beneficial achievement of the invention is: the method proposed through the invention can be for multiple libraries in remote data base
In multiple tables, a key obtains its each statistical data for occupying resource, and obtains similar report, referred to for research and development or DBA,
Can amount of user data observation, also can be used as the foundation of data scrubbing.
Specific embodiment
It is carried out below with reference to technical effect of the embodiment and attached drawing to design of the invention, specific structure and generation clear
Chu, complete description, to be completely understood by the purpose of the present invention, scheme and effect.It should be noted that the case where not conflicting
Under, the features in the embodiments and the embodiments of the present application can be combined with each other.The identical attached drawing mark used everywhere in attached drawing
Note indicates the same or similar part.
It should be noted that unless otherwise specified, when a certain feature referred to as " fixation ", " connection " are in another feature,
It can directly fix, be connected to another feature, and can also fix, be connected to another feature indirectly.In addition, this
The descriptions such as upper and lower, left and right used in application are only the mutual alignment pass relative to each component part of the application in attached drawing
For system.In the application and the "an" of singular used in the attached claims, " described " and "the" also purport
It is including most forms, unless the context clearly indicates other meaning.
In addition, unless otherwise defined, the technology of all technical and scientific terms used herein and the art
The normally understood meaning of personnel is identical.Term used in the description is intended merely to description specific embodiment herein, without
It is to limit the application.Term as used herein "and/or" includes the arbitrary of one or more relevant listed items
Combination.
A kind of network topology for statistical method based on MongoDB database that the application shown in referring to Fig.1 is proposed
Scheme, shows the N number of server for being connected to communication network by router or interchanger in figure, be server 1, server respectively
2 and server N and same-router or interchanger are connected to the client 1 and client 2 of network, wherein the communication
Network can be local area network, be also possible to internet, be also possible to enterprise's special line, wherein in the storage device in N number of server
It is stored with database and database collection, on the other hand, client can be fixed equipment, such as desktop computer, can also be with
It is mobile terminal, such as mobile phone or tablet computer, laptop etc., communication connection mode can be wire communication and be also possible to
Wireless communication, if using mobile phone or tablet computer etc. without network cable jack terminal device when, need using wireless communication
Means are connected to communication network, and the wireless communication means include at least following one kind: 2G, 3G, 4G, 5G and Wi-Fi.
Referring to a kind of the first implementation of statistical method based on MongoDB database that the application shown in Fig. 2 is proposed
Example flow chart, the flow diagrams illustrate the method and steps of the set by access server statistics MongoDB database, comprising:
S100) pass through the IP address of instruction definition one or more servers and/database to be visited, port numbers, user
Name, password and database collection title to be counted;
S200) client passes through IP address, port numbers, the simultaneously access one or more of username and password connection remote server
Database in a server;
S300 all library names of database in one or more servers) are obtained, and traverse all libraries, are obtained each
The relation on attributes data of the database collection comprising all database collection titles to be counted under library, and have identical
The corresponding relation data of database collection added up to obtain the corresponding relation data of database collection and stored and filled with storage
In setting;Wherein, the corresponding relationship includes at least one of following items, data volume, file size, file size accounting,
Size of data, size of data accounting, storage size, storage size accounting, mean object size, index quantity and index space
Size;
S400) the corresponding relation data of display data library set.
Further, step S300 further includes following sub-step:
One in the corresponding relationship is specified, and with specified corresponding relationship, according to the method pair of ascending or descending order
The corresponding relation data of database collection is ranked up.
Specifically, in the above method proposed by the present invention, the specified mode includes at least the one of following manner
Kind, system default is artificial to be arranged.
Further, step S300 further includes following sub-step:
Judge whether the data that corresponding relationship is included can all show that, if cannot all show, branch is aobvious in a line
Show.
Specifically, MongoDB is write by C Plus Plus, is the PostgreSQL database based on distributed document storage
System.In high-load situations, this, more nodes are added, it is ensured that server performance.
MongoDB stores data as a document, and data structure is by key assignments (key=> value) to forming.MongoDB
Document is similar to JSON object.Field value may include other documents, array and document array.Set is exactly the text of MongoDB
Shelves group, is similar in RDBMS (relational database management system: Relational Database Management System)
Table, set is present in database, gathers the structure do not fixed, it means that not apposition is can be inserted to set in you
The data of formula and type, but the data that we are inserted into set under normal conditions can all have certain relevance.Similarly,
Database table in the corresponding SQL term of the set of MongoDB database.Also, it can establish multiple data in a MongoDB
Library.
Further, the problem of relating generally to access security due to server, client when accessing server,
Need to introduce authentication means, to guarantee the safety of the data in server.The one kind proposed referring to the application shown in Fig. 3
The flow chart of the second embodiment of statistical method based on MongoDB database is shown in figure and a kind of is stepped on using user name password
The server authorizing procedure of record mode, comprising the following steps:
S201) client sends access request to server and/or database using communication means;
S202) server and/or database receive access request that client is sent and judge access request whether include
Authorization message sends not awarding for the first authentication data generated with server and/or database to client if not including
Information is weighed, if comprising allowing client access server and/or database;
S203) client receives the unauthorized information from server and/or database and extracts the first certification number
According to close using the first algorithm and the first authentication data, server and/or database user name and server and/or database
Code generates the second authentication data, and the second authentication data is sent to server and/database;
S204) server and/or database receive after the second authentication data of client, using the first algorithm and
First authentication data, server and/or database user name and server and/or database password generate third authentication data,
And judge whether third authentication data is consistent with the second authentication data, and it is such as consistent, then allow client access server and/or number
Such as inconsistent according to library, then denied access is requested.
Specifically, in doing the above method that invention proposes, first algorithm is Message Digest 5.
Specifically, in doing the above method that invention proposes, the Message Digest 5 is at least one of following algorithm:
MD algorithm (Message Digest 5), SHA algorithm (Secure Hash Algorithm) and MAC algorithm (message authentication code algorithm).
Specifically, MD algorithm family includes MD2, MD3, MD4 and MD5 algorithm, they require to obtain a random-length
Information and generate one 128 informative abstracts.If by this 128 binary system summary info be converted into 16 into
System, available one 32 character strings, it is hexadecimal character that the digital finger-print of daily major part MD5 algorithm, which is all 32,
String.
Secure Hash Algorithm (Secure Hash Algorithm) is primarily adapted for use in digital signature standard (Digital
Signature Standard DSS) Digital Signature Algorithm (the Digital Signature Algorithm that defines of the inside
DSA).It is less than 2^64 message for length, SHA1 can generate one 160 eap-message digests.In MAC algorithm, message
Hashed value is controlled by the privacy key K for only having communicating pair to know.Hash value is referred to as MAC at this time.
Further, one kind that above-mentioned communication process can refer to that the application shown in Fig. 4 is proposed is based on MongoDB data
The second embodiment information flow direction figure of the statistical method in library:
Step 1, user end to server initiate access request;
Step 2, server receive access request, judge the client whether Internet access, if do not have permission to access, to
Client initiates authorizing procedure, specially authenticates to the requirement that client sends the first authentication data generated with server
Information;
Step 3 after client receives the first authentication data, calculates the second authentication data and sends to server;
Step 4 after server receives the second authentication data, calculates third authentication data using same method and with
Two authentication datas compare, and allow client initiating business request if consistent, if inconsistent, refuse the visit of the client
Ask request.
Further, in the above method proposed by the present invention, first authentication data is that server and/database generate
Random data.
Further, in the above method proposed by the present invention, the communication means includes at least following one kind: wire communication
Method, wireless communications method.
Further, in the above method proposed by the present invention, the wireless communications method is including at least following one kind: 2G,
3G, 4G, 5G and Wi-Fi.
Further, it if database is provided with user name and password, also needs authenticate accordingly to access accordingly
Database, specific authentication process can refer to the authentication process of server.
Further, other than the method for using authentication, the measure of some communication securitys can also be used, to guarantee data
The safety of transmission, such as pass through IPSEC VPN.
Specifically, the communication process of an IPSec is as follows:
1. user's first (on host A) sends message to user's second (on host B);
2. the IPSec driver on host A checks IP screening washer, check whether data packet needs protected and needs
By which kind of protection;
3. driver notice IKE starts security negotiation;
4. the IKE on host B receives request security negotiation notice;
5. two hosts establish first stage SA, " master key " note is shared in each self-generating: if two machines before this communication in
It has been set up first stage SA, then can directly carry out second stage SA negotiation;
6. SA couples of second stage: inbound SA and outbound SA are established in negotiation.SA includes key and SPI;
7. IPSec driver is signed (integrity checking) to data packet and/or is added using outbound SA on host A
It is close.
(starting to send data packet);
8. data packet is submitted IP layers by driver, then forwards the packet to host B by IP layers;
9. host B network adapter driver receives data packet and submits to IPSec driver;
10. the IPSec driver on host B is signed using inbound SA checked for integrity and/or is solved to data packet
It is close;
11. the data packet after decryption is submitted upper TCP/IP driver by driver, then by TCP/IP driver
Data packet is submitted to the reception application program of host B.
After above-mentioned authentication measure and setting up safe communication connection, so that it may safely in server
Data access.
A kind of third for statistical method based on MongoDB database that the application referring to Figure 5 is proposed is implemented
Example statistical form, the statistical form is by executing to issue orders to obtain:
python MongoDB_stat_tool.py-h 10.11.81.62-u root-p root-c crash_raw_
data。
Wherein, in the order, " crash_raw_data " is indicated, the title for the database collection for needing to count, the life
The data cases that the name set for counting all libraries on 10.11.81.62 is had to the set of " crash_raw_data " are enabled,
And count the service condition in each library.The technology based on python, used of receiving orders is mainly python, pymongo, psutil
Deng
Specifically, pymongo is driver, so that python program is able to use Mongodb database, uses python
It writes, needs in general to be arranged using the MongoClient inside the library pymongo when connecting MongoDB database
The IP address of MongoDB and port, first parameter are address host, and second parameter is port port, if port
Not passing default is 27017.
import pymongo
Client=pymongo.MongoClient (host='localhost', port=27017)
The connecting object of a MongoDB can be created by mentioned order.In addition first of MongoClient
Parameter host can also directly pass the connection string of MongoDB, be started with MongoDB,
It is also divided into database one by one in MongoDB, needs to specify database to be used in a program.It is entitled to access
For the database of data, by issue orders,
Db=client.data
By calling the test attribute of client to can return to test database;
Each database of MongoDB contains many set Collection again, also with regard to similar and relevant database
In table, thus need also to specify the set to be operated, specifically need using order below
Collection=db.Collection, wherein Collection is exactly the set for needing to access.
Specifically, in Fig. 5 table, it can be seen that gauge outfit field includes: name set, quantity, size of data accounting
(%), size of data, mean object size, index quantity, index size and storing data size, wherein size of data with
Storing data size is simultaneously identical, to explain the difference of the two, need a kind of base proposed referring to the application shown in fig. 6
In the data file structure of the MongoDB database of the statistical method of MongoDB database, according to the description of front, one
MongoDB database includes multiple set Collection, and Fig. 6 shows only the data of one of set of some database
File structure, to achieve the purpose that brief description.
In the data file of the set, it can be seen that the data file is made of two different parts Extent,
MongoDB is inside the data of the BSON document stored and b-tree indexed tissue to logic container " Extent ", also, one
File can have multiple parts Extent, each Extent can only include the data or index of a set, such as Fig. 6
In, Extent1 is index file, and Extent2 is data file, and the data or index of the same set can be distributed in multiple
In Extent.These Extent can also be distributed in multiple files, and the same Extent will not have not only data but also have index.
Storage has multiple records inside each Extent, includes a record head and MongoDB in each record
BSON document and some additional spaces Padding, as shown in Figure 6.The space Padding is that MongoDB is recorded in insertion
When some vacant spaces of additional allocation, such document in future is unlikely to that document is needed to move to other places when becoming larger.Record
Head is started with the size entirely recorded, oneself position and the previous position recorded and the latter records are recorded including this.
Thus there is the record of different size of data.
Size of data (dataSize) is closest to a parameter of truthful data size, this parameter can be used to check
Data how many.This size includes the summation of every record of database (or set).Pay attention to every record in addition to
There are also these overheads of header and Padding outside BSON document.The meeting so actual size can take up space than authentic data
It is slightly larger.
When deleting document, this parameter can accordingly become smaller because it is the size summation of all number of files.If
Your document is not deleted, and only the field of inside documents is deleted or reduces, then will not be had an impact to size of data.Reason is just
It is also to exist because being recorded where document, and whole record takes up space and has no change, only the vacant space in record becomes
It is more.
Storage size (storageSize) this parameter is equal to database or some gathers all data used
The summation of Extents.Pay attention to staying after this number can be greater than data because having some deletion documents inside Extent
Fragment, i.e., it is shown in fig. 6 to have deleted part.
File size (fileSize) this parameter only on the database effectively, refers to using in actual file system
The size of file.It includes the summation of all data Extents, indexes the summation of Extent and some unassigned
Space.
Referring to the last rows of Fig. 5 statistical form generated, it is seen then that single database occupancy alreadys exceed 20G, single to collect
It closes more than 5G, individual optimization processing will be carried out for these library tables in this case.
In addition, psutil is the process and system availability that a cross-platform library can easily realize the operation of acquisition system
(including CPU, memory, disk, network etc.) information.It is mainly used to do system monitoring, performance evaluation, management of process.It is realized
The function that same command-line tool provides, as ps, top, lsof, netstat, ifconfig, who, df, kill, free,
Nice, ionice, iostat, iotop, uptime, pidof, tty, taskset, pmap etc..32 and 64 are supported at present
The operating systems such as Linux, Windows, OS X, FreeBSD and Sun Solaris.
Referring to a kind of the 4th implementation of statistical method based on MongoDB database that the application shown in Fig. 7 is proposed
Example statistical form, as can be seen that the table is to carry out descending sort, the application according to the file size accounting of each set in table
The statistical method proposed can be supported to carry out any static fields ascending or descending order arrangement and be shown, for example, such as figure figure
A kind of 5th embodiment statistical form of statistical method based on MongoDB database that the application shown in 8 is proposed is exactly logical
It crosses with storing data size accounting, is arranged by ascending order.In addition, also supporting branch's display in table and with different units
It is shown.
Referring to a kind of the 5th implementation of statistical method based on MongoDB database that the application shown in Fig. 9 is proposed
Example statistics pie chart, the statistical method that the application is proposed can also support pie chart to export, and automatically too small to accounting be integrated into pair
Figure output.
A kind of frame diagram for statistic device based on MongoDB database that the application shown in 0 is proposed referring to Fig.1,
It comprises the following modules:
Definition module, for the IP address by instruction definition one or more servers and/database to be visited, port
Number, user name, password and database collection title to be counted;
Access modules pass through IP address, port numbers for client, and username and password connection remote server simultaneously accesses
Database in one or more servers;
Module is obtained, for obtaining all library names of the database in one or more servers, and traverses all libraries,
The relation on attributes data of the database collection comprising all database collection titles to be counted under each library are obtained, and are had
The identical corresponding relation data of database collection is added up to obtain the corresponding relation data of database collection and storage
In storage device;Wherein, the corresponding relationship includes at least one of following items, data volume, file size, and file is big
Small accounting, size of data, size of data accounting, storage size, storage size accounting, mean object size, index quantity and
Index space size;
Display module, the corresponding relation data for display data library set.
Further, the problem of relating generally to access security due to server, client when accessing server,
Need to introduce authentication means, to guarantee the safety of the data in server.Therefore, access modules may also include following submodule
Block:
Request module sends access request to server and/database using communication means for client;
Judgment module, the access request sent for server and/database reception client simultaneously judge that access request is
No includes authorization message, if not including, sends the first authentication data generated with server and/database to client
Unauthorized information, if comprising allowing client access server and/database;
Authentication module receives from the unauthorized information of server and/database for client and extracts described first
Authentication data, using the first algorithm and the first authentication data, server and/database user name and server and/database
Password generates the second authentication data, and the second authentication data is sent to server and/database;
Authorization module receives after the second authentication data of client for server and/database, calculates using first
Method and the first authentication data, server and/database user name and server and/database password generate third and authenticate number
According to, and judge whether third authentication data consistent with the second authentication data, it is such as consistent, then allow client access server and/
Database, such as inconsistent, then denied access is requested.
Specifically, in above-mentioned apparatus, first authentication data is the random data that server and/database generate.
Further, in above-mentioned apparatus, the communication means includes at least following one kind: wire communication method, channel radio
Letter method.
Further, in above-mentioned apparatus, the wireless communications method includes at least following one kind: 2G, 3G, 4G, 5G and Wi-
Fi。
Further, obtaining module further includes following submodule:
Sorting module, for specifying one in the corresponding relationship, and with specified corresponding relationship, according to ascending order or
The method of descending is ranked up the corresponding relation data of database collection.
Specifically, in above-mentioned module, the specified mode includes at least one of following manner, system default,
Artificial setting.
Further, obtaining module further includes following submodule:
Branch's display module, for judging whether the data that corresponding relationship is included can all show in a line, if cannot
All displays, then branch shows.
Finally, the present invention proposes a kind of computer readable storage medium, it is stored thereon with computer instruction, described instruction is used
In the above-mentioned method of execution.
It will be appreciated that though various elements may be described in this application using term first, second, third, etc., but
These elements should not necessarily be limited by these terms.These terms are only used to for same type of element being distinguished from each other out.For example, not taking off
In the case where the application range, first element can also be referred to as second element, and similarly, second element can also be referred to as
First element.Depending on context, word as used in this " if " can be construed to " ... when " or " when ...
When ".
It should be appreciated that the embodiment of the present invention can be by computer hardware, the combination of hardware and software or by depositing
The computer instruction in non-transitory computer-readable memory is stored up to be effected or carried out.Standard volume can be used in the method
Journey technology include realized in computer program configured with the non-transitory computer-readable storage media of computer program, wherein
Configured in this way storage medium make computer operated in a manner of specific and is predefined-according in a particular embodiment describing
Method and attached drawing.Each program can be realized with the programming language of level process or object-oriented with logical with computer system
Letter.However, if desired, the program can be realized with compilation or machine language.Under any circumstance, the language can be compiling or
The language of explanation.In addition, the program can be run on the specific integrated circuit of programming for this purpose.
Further, this method can be realized in being operably coupled to suitable any kind of computing platform, including
But it is not limited to PC, mini-computer, main frame, work station, network or distributed computing environment, individual or integrated
It computer platform or is communicated with charged particle tool or other imaging devices etc..Each aspect of the present invention can be with storage
No matter machine readable code on non-transitory storage medium or equipment is moveable or is integrated to calculate and put down to realize
Platform, such as hard disk, optically read and/or write-in storage medium, RAM, ROM, so that it can be read by programmable calculator, when depositing
Storage media or equipment can be used for configuration and operation computer to execute process described herein when being read by computer.In addition,
Machine readable code, or part thereof can be transmitted by wired or wireless network.When such media include in conjunction with microprocessor or
When other data processors realize the instruction or program of steps described above, invention as described herein includes that these and other are different
The non-transitory computer-readable storage media of type.When methods and techniques according to the present invention programming, the present invention is also
Including computer itself.
This document describes embodiment of the disclosure, become known for executing optimal mode of the invention including inventor.It is readding
After having read foregoing description, the variation of these embodiments will be apparent those skilled in the art.Inventor wishes skill
Art personnel optionally use such modification, and inventor be intended to by be different from it is as specifically described herein in a manner of practice this public affairs
The embodiment opened.Therefore, through applicable legal permission, the scope of the present disclosure includes describing in this appended claims
The all modifications and equivalent of theme.In addition, the scope of the present disclosure covers any of the above-mentioned element in its all possible modification
Combination, unless herein in addition instruction or otherwise significantly with contradicted by context.
Although description of the invention is quite detailed and especially several embodiments are described, it is not
Any of these details or embodiment or any specific embodiments are intended to be limited to, but should be considered as is by reference to appended
A possibility that claim provides broad sense in view of the prior art for these claims explanation, to effectively cover the present invention
Preset range.In addition, with the foreseeable embodiment of inventor, present invention is described above, its purpose is to be provided with
Description, and those still unforeseen at present change to unsubstantiality of the invention can still represent equivalent modifications of the invention.
Therefore, should be with descriptive sense rather than restrictive sense understands the specification and drawings.However, by apparent
It is:, can be to the application in the case where not departing from the broader spirit and scope of the application as described in claims
Make various modifications and change.
Other modifications are in spirit herein.Therefore, although disclosed technology may be allowed various modifications and substitution structure
It makes, but has shown that in the accompanying drawings and its some embodiments shown in being described in detail above.It will be appreciated, however, that not
It is intended to for the application to be confined to disclosed one or more concrete forms;On the contrary, its intention covers such as the appended claims
Defined in fall in all modifications, alternative constructions and equivalent in spirit and scope.