CN112380163A - S3 file system space occupation monitoring method and system - Google Patents
S3 file system space occupation monitoring method and system Download PDFInfo
- Publication number
- CN112380163A CN112380163A CN202011122246.8A CN202011122246A CN112380163A CN 112380163 A CN112380163 A CN 112380163A CN 202011122246 A CN202011122246 A CN 202011122246A CN 112380163 A CN112380163 A CN 112380163A
- Authority
- CN
- China
- Prior art keywords
- file
- attribute data
- monitoring
- service
- space occupation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 57
- 238000000034 method Methods 0.000 title claims abstract description 37
- 238000007619 statistical method Methods 0.000 claims abstract description 13
- 230000008569 process Effects 0.000 claims abstract description 9
- 230000002452 interceptive effect Effects 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 3
- 230000009286 beneficial effect Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 8
- 238000007726 management method Methods 0.000 description 7
- 230000008859 change Effects 0.000 description 4
- 238000013480 data collection Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 230000036961 partial effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000002829 reductive effect Effects 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
Abstract
The invention discloses a method and a system for monitoring space occupation of an S3 file system, wherein the method comprises the following steps: s100, constructing a plurality of monitoring processes based on an access configuration table of the S3 file service, wherein the monitoring processes acquire file attribute data of the S3 file service and push the file attribute data to corresponding acquired data pipelines of the monitoring processes; s200, acquiring the file attribute data in the data acquisition pipeline, performing statistical analysis to obtain a space occupation analysis result, and storing the space occupation analysis result in a database. The invention has at least the following beneficial effects: the configuration of the splitting process and the data pipeline monitors the space occupation of the S3 file service, can acquire the S3 file attribute data distributed at different geographic positions, reduces the complexity of data acquisition, and effectively avoids congestion and delay.
Description
Technical Field
The invention relates to the technical field of game server test, in particular to a method and a system for monitoring space occupation of an S3 file system.
Background
S3 is an abbreviation of Simple Storage Service, which has many advantages such as high availability, distributed, cross-region replication, event notification, encryption, flexible Storage options, etc. As the application of the S3 storage service becomes more and more widespread, it follows that monitoring of its system becomes more and more important. The S3 storage service provides only simple client and partial system-dependent api, is not very friendly to monitor, and currently there is no monitoring service specific to S3.
Disclosure of Invention
The present invention is directed to solving at least one of the problems of the prior art. Therefore, the invention provides a method for monitoring the space occupation of the S3 file system, which can monitor the S3 storage service.
The invention also provides an S3 file system space occupation monitoring system with the S3 file system space occupation monitoring method.
The invention also provides a computer readable storage medium with the S3 file system space occupation monitoring method.
The method for monitoring the space occupation of the S3 file system according to the first embodiment of the invention comprises the following steps: s100, constructing a plurality of monitoring processes based on an access configuration table of the S3 file service, wherein the monitoring processes acquire file attribute data of the S3 file service and push the file attribute data to corresponding acquired data pipelines of the monitoring processes; s200, acquiring the file attribute data in the data acquisition pipeline through a distributed process, performing statistical analysis to obtain a space occupation analysis result, and storing the space occupation analysis result in a database.
The method for monitoring the space occupation of the S3 file system according to the embodiment of the invention at least has the following beneficial effects: the space occupation of the S3 file service is monitored according to the configuration splitting process and the data pipeline, the S3 file attribute data distributed at different geographic positions can be obtained, the complexity of data acquisition is reduced, and congestion and delay are effectively avoided.
According to some embodiments of the invention, the access configuration table comprises: service name, service area, service access address, access ID, and access password.
According to some embodiments of the present invention, the monitoring process collects the file attribute data at regular time according to a preset configuration.
According to some embodiments of the invention, the file attribute data comprises: the name of a Bucket, the occupied space of the Bucket, the name of a file under the Bucket and the occupied space of the file.
According to some embodiments of the invention, the S200 comprises: and acquiring the total number of buckets, the number of files under the buckets, the total number of the files and the total space occupation according to the file attribute data.
According to some embodiments of the invention, the S200 further comprises: the method comprises the steps of obtaining a first number of buckets according to the occupied space from large to small, and obtaining a second number of files under a single Bucket according to the occupied space from large to small.
According to some embodiments of the invention, further comprising: and generating a graphic and text report according to the space occupation analysis result based on a preset display type.
An S3 file system space usage monitoring system according to an embodiment of the second aspect of the present invention includes: the monitoring acquisition module is used for acquiring the file attribute data of the S3 file service and pushing the file attribute data to a corresponding data acquisition pipeline of the monitoring acquisition module; the general control module is used for managing a plurality of monitoring acquisition modules based on the access configuration table of the S3 file service; and the statistical analysis module is used for acquiring the file attribute data in the data acquisition pipeline through a distributed process, performing statistical analysis to obtain a space occupation analysis result and storing the space occupation analysis result in a database.
The S3 file system space occupation monitoring system according to the embodiment of the invention has at least the following beneficial effects: the configuration of the splitting process and the data pipeline monitors the space occupation of the S3 file service, can acquire the S3 file attribute data distributed at different geographic positions, reduces the complexity of data acquisition, and effectively avoids congestion and delay.
According to some embodiments of the invention, further comprising: the configuration management module is used for providing an interactive interface to edit the access attribute data of the S3 file service and storing the access attribute data into the access configuration table; and the report generation module is used for generating a graphic and text report according to the space occupation analysis result based on a preset display type.
The computer-readable storage medium according to an embodiment of the third aspect of the present invention has stored thereon a computer program which, when executed by a processor, implements the method of an embodiment of the first aspect of the present invention.
The computer-readable storage medium according to the embodiment of the invention has at least the following advantages: the configuration of the splitting process and the data pipeline monitors the space occupation of the S3 file service, can acquire the S3 file attribute data distributed at different geographic positions, reduces the complexity of data acquisition, and effectively avoids congestion and delay.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
The above and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
FIG. 1 is a schematic flow chart of a method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of data interaction in a method according to an embodiment of the invention;
FIG. 3 is a block diagram of the modules of the system of an embodiment of the present invention.
Reference numerals:
the system comprises a monitoring acquisition module 100, a master control module 200, a statistical analysis module 300, a configuration management module 400 and a report generation module 500.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention.
In the description of the present invention, the meaning of a plurality of means is one or more, the meaning of a plurality of means is two or more, and more than, less than, more than, etc. are understood as excluding the present number, and more than, less than, etc. are understood as including the present number. If the first and second are described for the purpose of distinguishing technical features, they are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated or implicitly indicating the precedence of the technical features indicated.
Referring to fig. 1, a method of an embodiment of the present invention includes: s100, constructing a plurality of monitoring processes based on an access configuration table of the S3 file service, acquiring file attribute data of the S3 file service by the monitoring processes, and pushing the file attribute data to corresponding acquired data pipelines of the monitoring processes; and S200, acquiring file attribute data in the data acquisition pipeline, performing statistical analysis to obtain a space occupation analysis result, and storing the space occupation analysis result in a database.
In one embodiment of the present invention, an access configuration table is required to be configured in advance, the table may include access attributes of a plurality of S3 file services, and the access attribute of each S3 file service includes a service name, an area to which the service belongs, an access address, an access ID and an access password for connecting to the S3 file service, and the like. Referring to fig. 2, the monitoring program starts a plurality of monitoring processes according to the access configuration table, each monitoring process accesses a corresponding S3 file service according to the configuration (each monitoring process accesses a corresponding S3 file service), collects file attribute data of the S3 file service, and the file attribute data is pushed to a corresponding data collection pipeline of the monitoring process. The file attribute data includes: the names of all buckets under the S3 file service, the occupation space of a single Bucket, the names of all files under a single Bucket, and the occupation space of each file. In the embodiment of the present invention, the monitoring process may set configurations such as a start-stop time (or a work period) of work, an acquisition frequency, and the like, and periodically perform acquisition of file attribute data according to a preset configuration. It should be understood that multiple monitoring processes of embodiments of the present invention may also push collected data to the same data collection pipeline; for example, two monitoring operations in the same area, but without overlapping operating times, may push data to the same data acquisition pipeline. The distributed process acquires file attribute data from the data acquisition pipeline, and obtains the following statistical contents according to the file attribute data: the total number of buckets, the number of files under the buckets, the total number of the files and the total occupied space; and according to the sequence of space occupation from large to small, a first number of buckets (for example: Bucket occupation space top10) are obtained, and a second number of files under a single Bucket (for example: occupation space top10 of files under a single Bucket) are obtained. The space occupation analysis results are stored in a database and used for providing graphic and text report display according to needs. For example: the size change of the occupied space of a single Bucket along with the change of time is shown in a graph form; the change of the number of files of a single Bucket along with the change of time is shown in a graph form; the total number of buckets in the S3 service is shown graphically as a function of time, among other things. For another example: displaying the space occupation proportion of each Bucket in the S3 service by a pie chart; the footprint top10 of a file under a single Bucket is shown in a bar chart. It should be understood that an interactive interface can be provided for presetting the display type, selecting the viewed statistical items, and generating a graphic report according to the space occupation analysis result. In some embodiments of the present invention, a presentation style is also provided, including a presentation type of each analysis result; selecting the style generates various chart analysis results required by the user. In some embodiments of the present invention, a file attribute data downloading function is also provided, and the file attribute data is downloaded locally in json format for further analysis and use.
The system of the embodiment of the present invention, referring to fig. 3, includes: a monitoring acquisition module 100, a total control module 200 and a statistical analysis module 300. The monitoring acquisition module 100 is used for acquiring file attribute data of the S3 file service and pushing the file attribute data to a corresponding acquired data pipeline of the monitoring acquisition module; the general control module 200 is used for managing a plurality of monitoring acquisition modules based on an access configuration table of the S3 file service; and the statistical analysis module 300 is configured to obtain file attribute data in the collected data pipeline, perform statistical analysis, obtain a space occupation analysis result, and store the space occupation analysis result in the database. The master control module 200 controls and generates a plurality of monitoring acquisition modules 100, and the statistical analysis module 300 analyzes the file attribute data acquired by the monitoring acquisition modules 100.
Referring to fig. 3, the system of the embodiment of the present invention further includes: a configuration management module 400 and a report generation module 500. The configuration management module 400 is used for providing access attribute data of the interactive interface editing S3 file service and saving the access attribute data to an access configuration table; and the report generation module 500 is configured to generate a graphic report according to the space occupation analysis result based on the preset display type. The configuration management module 400 can store different access attribute data into one access configuration table uniformly and synchronize the access configuration table to all users, so that statistics and management are facilitated. In other embodiments of the present invention, at least one of the configuration management module 400 or the report generation module 500 is not included.
Although specific embodiments have been described herein, those of ordinary skill in the art will recognize that many other modifications or alternative embodiments are equally within the scope of this disclosure. For example, any of the functions and/or processing capabilities described in connection with a particular device or component may be performed by any other device or component. In addition, while various illustrative implementations and architectures have been described in accordance with embodiments of the present disclosure, those of ordinary skill in the art will recognize that many other modifications of the illustrative implementations and architectures described herein are also within the scope of the present disclosure.
Certain aspects of the present disclosure are described above with reference to block diagrams and flowchart illustrations of systems, methods, systems, and/or computer program products according to example embodiments. It will be understood that one or more blocks of the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, respectively, can be implemented by executing computer-executable program instructions. Also, according to some embodiments, some blocks of the block diagrams and flow diagrams may not necessarily be performed in the order shown, or may not necessarily be performed in their entirety. In addition, additional components and/or operations beyond those shown in the block diagrams and flow diagrams may be present in certain embodiments.
Accordingly, blocks of the block diagrams and flowchart illustrations support combinations of means for performing the specified functions, combinations of elements or steps for performing the specified functions and program instruction means for performing the specified functions. It will also be understood that each block of the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, can be implemented by special purpose hardware-based computer systems that perform the specified functions, elements or steps, or combinations of special purpose hardware and computer instructions.
Program modules, applications, etc. described herein may include one or more software components, including, for example, software objects, methods, data structures, etc. Each such software component may include computer-executable instructions that, in response to execution, cause at least a portion of the functionality described herein (e.g., one or more operations of the illustrative methods described herein) to be performed.
The software components may be encoded in any of a variety of programming languages. An illustrative programming language may be a low-level programming language, such as assembly language associated with a particular hardware architecture and/or operating system platform. Software components that include assembly language instructions may need to be converted by an assembler program into executable machine code prior to execution by a hardware architecture and/or platform. Another exemplary programming language may be a higher level programming language, which may be portable across a variety of architectures. Software components that include higher level programming languages may need to be converted to an intermediate representation by an interpreter or compiler before execution. Other examples of programming languages include, but are not limited to, a macro language, a shell or command language, a job control language, a scripting language, a database query or search language, or a report writing language. In one or more exemplary embodiments, a software component containing instructions of one of the above programming language examples may be executed directly by an operating system or other software component without first being converted to another form.
The software components may be stored as files or other data storage constructs. Software components of similar types or related functionality may be stored together, such as in a particular directory, folder, or library. Software components may be static (e.g., preset or fixed) or dynamic (e.g., created or modified at execution time).
The embodiments of the present invention have been described in detail with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the gist of the present invention.
Claims (10)
1. A method for monitoring space occupation of an S3 file system is characterized by comprising the following steps:
s100, constructing a plurality of monitoring processes based on an access configuration table of the S3 file service, wherein the monitoring processes acquire file attribute data of the S3 file service and push the file attribute data to corresponding acquired data pipelines of the monitoring processes;
s200, acquiring the file attribute data in the data acquisition pipeline through a distributed process, performing statistical analysis to obtain a space occupation analysis result, and storing the space occupation analysis result in a database.
2. The S3 file system space usage monitoring method of claim 1, wherein the access configuration table includes: service name, service area, service access address, access ID, and access password.
3. The S3 file system space usage monitoring method of claim 1, wherein the monitoring process collects the file attribute data periodically according to a preset configuration.
4. The S3 file system space usage monitoring method of claim 1, wherein the file attribute data includes: the name of a Bucket, the occupied space of the Bucket, the name of a file under the Bucket and the occupied space of the file.
5. The S3 file system space usage monitoring method of claim 1, wherein the S200 includes: and acquiring the total number of buckets, the number of files under the buckets, the total number of the files and the total space occupation according to the file attribute data.
6. The S3 file system space usage monitoring method of claim 1, wherein the S200 further includes: the method comprises the steps of obtaining a first number of buckets according to the occupied space from large to small, and obtaining a second number of files under a single Bucket according to the occupied space from large to small.
7. The S3 file system space usage monitoring method of claim 5, further comprising: and generating a graphic and text report according to the space occupation analysis result based on a preset display type.
8. An S3 file system space usage monitoring system, using the method of any of claims 1 to 7, comprising:
the monitoring acquisition module is used for acquiring the file attribute data of the S3 file service and pushing the file attribute data to a corresponding data acquisition pipeline of the monitoring acquisition module;
the general control module is used for managing a plurality of monitoring acquisition modules based on the access configuration table of the S3 file service;
and the statistical analysis module is used for acquiring the file attribute data in the data acquisition pipeline through a distributed process, performing statistical analysis to obtain a space occupation analysis result and storing the space occupation analysis result in a database.
9. The S3 file system space usage monitoring system of claim 8, further comprising:
the configuration management module is used for providing an interactive interface to edit the access attribute data of the S3 file service and storing the access attribute data into the access configuration table;
and the report generation module is used for generating a graphic and text report according to the space occupation analysis result based on a preset display type.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011122246.8A CN112380163A (en) | 2020-10-20 | 2020-10-20 | S3 file system space occupation monitoring method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011122246.8A CN112380163A (en) | 2020-10-20 | 2020-10-20 | S3 file system space occupation monitoring method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112380163A true CN112380163A (en) | 2021-02-19 |
Family
ID=74581673
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011122246.8A Pending CN112380163A (en) | 2020-10-20 | 2020-10-20 | S3 file system space occupation monitoring method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112380163A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130041872A1 (en) * | 2011-08-12 | 2013-02-14 | Alexander AIZMAN | Cloud storage system with distributed metadata |
| CN104390660A (en) * | 2014-11-10 | 2015-03-04 | 深圳市利恩信息技术有限公司 | Remote container monitoring system and method |
| CN105122243A (en) * | 2013-03-15 | 2015-12-02 | 亚马逊科技公司 | Scalable analysis platform for semi-structured data |
| CN107911410A (en) * | 2017-10-17 | 2018-04-13 | 珠海金山网络游戏科技有限公司 | Distributed Services process resource consumes statistical method and device |
| CN109857703A (en) * | 2019-01-15 | 2019-06-07 | 珠海金山网络游戏科技有限公司 | A kind of space bucket accounting statistical method and its device based on minio file system |
| CN110377595A (en) * | 2019-07-24 | 2019-10-25 | 浙江吉利汽车研究院有限公司 | A kind of vehicle data management system |
| CN110968482A (en) * | 2019-12-18 | 2020-04-07 | 上海良鑫网络科技有限公司 | Enterprise service and application intelligent monitoring system |
| CN111488321A (en) * | 2020-03-05 | 2020-08-04 | 北京联创信安科技股份有限公司 | Management system for storage volume |
-
2020
- 2020-10-20 CN CN202011122246.8A patent/CN112380163A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130041872A1 (en) * | 2011-08-12 | 2013-02-14 | Alexander AIZMAN | Cloud storage system with distributed metadata |
| CN105122243A (en) * | 2013-03-15 | 2015-12-02 | 亚马逊科技公司 | Scalable analysis platform for semi-structured data |
| CN104390660A (en) * | 2014-11-10 | 2015-03-04 | 深圳市利恩信息技术有限公司 | Remote container monitoring system and method |
| CN107911410A (en) * | 2017-10-17 | 2018-04-13 | 珠海金山网络游戏科技有限公司 | Distributed Services process resource consumes statistical method and device |
| CN109857703A (en) * | 2019-01-15 | 2019-06-07 | 珠海金山网络游戏科技有限公司 | A kind of space bucket accounting statistical method and its device based on minio file system |
| CN110377595A (en) * | 2019-07-24 | 2019-10-25 | 浙江吉利汽车研究院有限公司 | A kind of vehicle data management system |
| CN110968482A (en) * | 2019-12-18 | 2020-04-07 | 上海良鑫网络科技有限公司 | Enterprise service and application intelligent monitoring system |
| CN111488321A (en) * | 2020-03-05 | 2020-08-04 | 北京联创信安科技股份有限公司 | Management system for storage volume |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8707275B2 (en) | Simulation environment for distributed programs | |
| US20070156698A1 (en) | Systems and methods for finding log files generated by a distributed computer | |
| CN107040578B (en) | Data synchronization method, device and system | |
| US10120886B2 (en) | Database integration of originally decoupled components | |
| CN105468720A (en) | Method for integrating distributed data processing systems, corresponding systems and data processing method | |
| US20100332904A1 (en) | Testing of Distributed Systems | |
| CN112527848B (en) | Report data query method, device and system based on multiple data sources and storage medium | |
| US9575772B2 (en) | Model-based user interface | |
| CN112073511B (en) | System and method for generating application page | |
| CN112711512A (en) | Prometheus monitoring method, device and equipment | |
| CN112035207A (en) | Page menu dynamic loading method and device based on database | |
| CN112380163A (en) | S3 file system space occupation monitoring method and system | |
| CN110909072B (en) | Data table establishment method, device and equipment | |
| US9059992B2 (en) | Distributed mobile enterprise application platform | |
| CN111459484B (en) | Method for simulating back-end data during separate development of front end and back end of webpage | |
| CN111506644B (en) | Application data processing method and device and electronic equipment | |
| CN110928895B (en) | Data query and data table establishment method, device and equipment | |
| CN112507193A (en) | Data updating method, device, equipment and storage medium | |
| CN112395350A (en) | Method and device for visualizing monitoring data of multiple data sources | |
| CN117762967A (en) | Data redistribution initiation method and device, electronic equipment and storage medium | |
| CN111221551B (en) | Database upgrading method and device | |
| CN116340363B (en) | Data storage and loading method based on relational database and related device | |
| CN117725079A (en) | Data storage method, medium and computer equipment of database | |
| CN112231378A (en) | Data processing method, system, server and storage medium | |
| CN117827266A (en) | Method and system for inquiring version and path of rpm packet in yum source |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 4th Floor, No. 6, 8, Jianzhong Road, Tianhe District, Guangzhou City, Guangdong Province, 510000 Room No. 406 Applicant after: Guangzhou Xishanju Network Technology Co.,Ltd. Address before: 3-04-1, No. 309, Huangpu Avenue, Tianhe District, Guangzhou, Guangdong 510000 Applicant before: GUANGZHOU SEASUN ENTERTAINMENT NETWORK TECHNOLOGY Co.,Ltd. |