CN112463725A - Cloud architecture log file batch processing method and device and storage medium - Google Patents
Cloud architecture log file batch processing method and device and storage medium Download PDFInfo
- Publication number
- CN112463725A CN112463725A CN202011302079.5A CN202011302079A CN112463725A CN 112463725 A CN112463725 A CN 112463725A CN 202011302079 A CN202011302079 A CN 202011302079A CN 112463725 A CN112463725 A CN 112463725A
- Authority
- CN
- China
- Prior art keywords
- log
- log file
- target host
- processed
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 18
- 238000000034 method Methods 0.000 claims abstract description 28
- 230000008569 process Effects 0.000 claims abstract description 21
- 230000003993 interaction Effects 0.000 claims abstract description 16
- 238000004140 cleaning Methods 0.000 claims description 28
- 238000004590 computer program Methods 0.000 claims description 10
- 230000006835 compression Effects 0.000 claims description 4
- 238000007906 compression Methods 0.000 claims description 4
- 230000008531 maintenance mechanism Effects 0.000 abstract description 4
- 238000012423 maintenance Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 3
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/122—File system administration, e.g. details of archiving or snapshots using management policies
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/162—Delete operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a cloud architecture log file batch processing method, a cloud architecture log file batch processing device and a storage medium, wherein the method comprises the following steps: the target host reads the tasks to be processed from the queue and sends an interaction request to the cloud server, wherein the interaction request carries a target host password; and the cloud server logs in the target host according to the target host password, performs task processing on the tasks to be processed, and acquires the next task to be processed from the target host and performs task processing when the task processing is completed until all the tasks to be processed in the target host are processed. The cloud server manages the target host in a cloud framework mode, can log in the target host according to the password of the target host to be managed, processes the log file through the task to be processed, is convenient for modifying the log file for the second time, constructs a cloud log maintenance mechanism, and strengthens the management of the life cycle of the log.
Description
Technical Field
The invention mainly relates to the technical field of file batch processing, in particular to a cloud architecture log file batch processing method, a cloud architecture log file batch processing device and a storage medium.
Background
In the cloud architecture, the number of service bearing hosts is increased, each service maintenance log is independent, a log maintenance script needs to be deployed to each host, and the secondary modification is very complicated; meanwhile, most logs are lost and maintained, and the life cycle management of the logs of the whole system is influenced. At present, no good solution to this problem exists.
Disclosure of Invention
The invention aims to solve the technical problem of the prior art and provides a method, a device and a storage medium for batch processing of cloud architecture log files.
The technical scheme for solving the technical problems is as follows: a cloud architecture log file batch processing method comprises the following steps:
the target host reads the tasks to be processed from the queue and sends an interaction request to the cloud server, wherein the interaction request carries a target host password;
the cloud server logs in a target host according to the target host password, and the task to be processed is processed;
and when the task processing is finished, acquiring the next task to be processed from the target host and processing the task until all the tasks to be processed in the target host are processed.
Another technical solution of the present invention for solving the above technical problems is as follows: a cloud architecture log file batch processing device comprises a target host and a cloud server;
the target host reads the tasks to be processed from the queue and sends an interaction request to a cloud server, wherein the interaction request carries a target host password;
and the cloud server logs in the target host according to the target host password, performs task processing on the tasks to be processed, and acquires the next task to be processed from the target host and performs task processing when the task processing is completed until all the tasks to be processed in the target host are processed.
Another technical solution of the present invention for solving the above technical problems is as follows: a cloud architecture log file batch processing device comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, and when the computer program is executed by the processor, the cloud architecture log file batch processing method is realized.
Another technical solution of the present invention for solving the above technical problems is as follows: a computer-readable storage medium storing a computer program which, when executed by a processor, implements a cloud architecture log file batch processing method as described above.
The invention has the beneficial effects that: the cloud server manages the target host in a cloud framework mode, can log in the target host according to a target host password to be managed, processes the log file through a task to be processed, is convenient for secondarily modifying the log file, constructs a cloud log maintenance mechanism, and strengthens management of a log life cycle.
Drawings
Fig. 1 is a schematic flow chart of a method for batch processing of cloud architecture log files according to an embodiment of the present invention;
fig. 2 is a functional module schematic diagram of a cloud architecture log file batch processing apparatus according to an embodiment of the present invention;
fig. 3 is a schematic diagram illustrating a connection between a cloud server and a plurality of target hosts according to an embodiment of the present invention.
Detailed Description
The principles and features of this invention are described below in conjunction with the following drawings, which are set forth by way of illustration only and are not intended to limit the scope of the invention.
Fig. 1 is a schematic flow chart of a cloud architecture log file batch processing method according to an embodiment of the present invention.
As shown in fig. 1, a method for batch processing of cloud architecture log files includes the following steps:
the target host reads the tasks to be processed from the queue and sends an interaction request to the cloud server, wherein the interaction request carries a target host password;
the cloud server logs in the target host according to the target host password and performs task processing on the task to be processed;
and when the task processing is finished, acquiring the next task to be processed from the target host and processing the task until all the tasks to be processed in the target host are processed.
Specifically, an expect automatic interaction function is used for writing a script of a main program, the main program controls business logic, and an expect built-in command is used, for example: spawn, expect, send, interact, etc. to realize automatic interaction with the target host.
When automatically interacting with the target host, the processing rule of the target host file is stored in the configuration file and is separated from the program logic. The target host login password, etc. is saved in another configuration file. The configuration file is stored in the cloud server.
Fig. 3 is a schematic diagram illustrating a connection between a cloud server and a plurality of target hosts according to an embodiment of the present invention.
It should be understood that, as shown in fig. 3, the cloud server is not limited to interact with only one target host, and the cloud server can simultaneously interact with n target hosts through target host passwords of different hosts.
In the embodiment, the cloud server manages the target host in a cloud framework mode, can log in the target host according to the password of the target host to be managed, and processes the log file through the task to be processed, so that the log file is modified for the second time, a cloud log maintenance mechanism is constructed, and the management of the life cycle of the log is enhanced.
Optionally, as an embodiment of the present invention, a process of the cloud server logging in the target host according to the target host password includes:
and matching the password of the target host through a pre-stored password configuration file, and logging in the target host if the matching is successful.
In the embodiment, the target host is quickly and safely logged in by the password of the target host, so that different target hosts can be managed conveniently.
Optionally, as an embodiment of the present invention, the process of performing task processing on the task to be processed includes:
and searching a corresponding log cleaning strategy configuration file according to the task to be processed, and cleaning the log file in the target host according to the log cleaning strategy configuration file.
In the embodiment, the log maintenance strategy is managed uniformly, maintenance and upgrading are easy, expansibility is good, and most log maintenance requirements are met.
Optionally, as an embodiment of the present invention, the process of performing a cleaning process on the log file in the target host according to the log cleaning policy configuration file includes:
if the task to be processed is a direct log deleting instruction, obtaining a log file name from the direct log deleting instruction, searching a log file storage path from the log cleaning strategy configuration file according to the log file name, searching a log file according to the log file storage path, and deleting the log file and the log file storage path together.
In the above embodiment, the log file corresponding to the designated log file name can be deleted specifically.
Optionally, as an embodiment of the present invention, the process of performing a cleaning process on the log file in the target host according to the log cleaning policy configuration file further includes:
if the task to be processed is a fuzzy matching log deleting instruction, acquiring partial log file names from the fuzzy matching log deleting instruction, searching a plurality of associated log file storage paths from the log cleaning strategy configuration file according to the partial log file names, respectively searching corresponding log files according to the plurality of log file storage paths, and deleting the searched log files and the corresponding log file storage paths together.
In the embodiment, the log files corresponding to a plurality of log file names meeting the fuzzy matching condition can be deleted in batch, and the log processing efficiency can be improved.
Optionally, as an embodiment of the present invention, the process of performing a cleaning process on the log file in the target host according to the log cleaning policy configuration file further includes:
if the task to be processed is a log compression instruction, obtaining a log file name from the log compression instruction, searching a log file storage path from the log cleaning strategy configuration file according to the log file name, searching a log file according to the log file storage path, and compressing the log file.
In the above embodiment, the log files can be compressed uniformly, and the log processing efficiency can be improved.
Optionally, as an embodiment of the present invention, the process of performing a cleaning process on the log file in the target host according to the log cleaning policy configuration file further includes:
if the task to be processed is a log clearing instruction according to time, acquiring time information from the log clearing instruction, searching a matched log file name from the log clearing strategy configuration file according to the time information, searching a log file storage path according to the log file name, searching a log file according to the log file storage path, and deleting the searched log file and the corresponding log file storage path together.
In the embodiment, the log files corresponding to a plurality of log file names meeting the time condition can be deleted in batch, and the log processing efficiency can be improved.
Fig. 2 is a functional module schematic diagram of a cloud architecture log file batch processing apparatus according to an embodiment of the present invention.
Optionally, as an embodiment of the present invention, as shown in fig. 2, a cloud-based architecture log file batch processing apparatus includes a target host and a cloud server;
the target host reads the tasks to be processed from the queue and sends an interaction request to a cloud server, wherein the interaction request carries a target host password;
and the cloud server logs in the target host according to the target host password, performs task processing on the tasks to be processed, and acquires the next task to be processed from the target host and performs task processing when the task processing is completed until all the tasks to be processed in the target host are processed.
Fig. 3 is a schematic diagram illustrating a connection between a cloud server and a plurality of target hosts according to an embodiment of the present invention.
It should be understood that, as shown in fig. 3, the cloud server is not limited to interact with only one target host, and the cloud server can simultaneously interact with n target hosts through target host passwords of different hosts. The efficiency of processing logs can be improved.
Optionally, as another embodiment of the present invention, a cloud architecture log file batch processing apparatus includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the cloud architecture log file batch processing method as described above is implemented.
Alternatively, as another embodiment of the present invention, a computer-readable storage medium stores a computer program which, when executed by a processor, implements the cloud architecture log file batch processing method as described above.
The cloud server manages the target host in a cloud framework mode, can log in the target host according to the password of the target host to be managed, processes the log file through the task to be processed, is convenient for modifying the log file for the second time, constructs a cloud log maintenance mechanism, and strengthens the management of the life cycle of the log. The log maintenance strategy is managed uniformly, maintenance and upgrading are easy, expansibility is good, and most log maintenance requirements are met.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (10)
1. A cloud architecture log file batch processing method is characterized by comprising the following steps:
the target host reads a task to be processed for processing the log file from the queue and sends an interaction request to the cloud server, wherein the interaction request carries a password of the target host;
the cloud server logs in the target host according to the target host password and performs task processing on the task to be processed;
and when the task processing is finished, acquiring the next task to be processed from the target host and processing the task until all the tasks to be processed in the target host are processed.
2. The method of claim 1, wherein the process of the cloud server logging in the target host according to the target host password comprises:
and matching the password of the target host through a pre-stored password configuration file, and logging in the target host if the matching is successful.
3. The cloud architecture log file batch processing method according to claim 1, wherein the task processing for the task to be processed includes:
and searching a corresponding log cleaning strategy configuration file according to the task to be processed, and cleaning the log file in the target host according to the log cleaning strategy configuration file.
4. The cloud architecture log file batch processing method according to claim 3, wherein the process of cleaning the log file in the target host according to the log cleaning policy configuration file includes:
if the task to be processed is a direct log deleting instruction, obtaining a log file name from the direct log deleting instruction, searching a log file storage path from the log cleaning strategy configuration file according to the log file name, searching a log file according to the log file storage path, and deleting the log file and the log file storage path together.
5. The cloud architecture log file batch processing method according to claim 3, wherein the process of cleaning the log file in the target host according to the log cleaning policy configuration file further comprises:
if the task to be processed is a fuzzy matching log deleting instruction, acquiring partial log file names from the fuzzy matching log deleting instruction, searching a plurality of associated log file storage paths from the log cleaning strategy configuration file according to the partial log file names, respectively searching corresponding log files according to the plurality of log file storage paths, and deleting the searched log files and the corresponding log file storage paths together.
6. The cloud architecture log file batch processing method according to claim 3, wherein the process of cleaning the log file in the target host according to the log cleaning policy configuration file further comprises:
if the task to be processed is a log compression instruction, obtaining a log file name from the log compression instruction, searching a log file storage path from the log cleaning strategy configuration file according to the log file name, searching a log file according to the log file storage path, and compressing the log file.
7. The cloud architecture log file batch processing method according to claim 3, wherein the process of cleaning the log file in the target host according to the log cleaning policy configuration file further comprises:
if the task to be processed is a log clearing instruction according to time, acquiring time information from the log clearing instruction, searching a matched log file name from the log clearing strategy configuration file according to the time information, searching a log file storage path according to the log file name, searching a log file according to the log file storage path, and deleting the searched log file and the corresponding log file storage path together.
8. A cloud architecture log file batch processing device is characterized by comprising a target host and a cloud server;
the target host reads a task to be processed for processing the log file from the queue and sends an interaction request to the cloud server, wherein the interaction request carries a password of the target host;
and the cloud server logs in a target host according to the target host password, performs task processing on the tasks to be processed, and acquires the next task to be processed from the target host and performs task processing when the task processing is completed until all the tasks to be processed in the target host are processed and completed.
9. A cloud architecture log file batch processing apparatus comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the cloud architecture log file batch processing method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the cloud architecture log file batch processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011302079.5A CN112463725B (en) | 2020-11-19 | 2020-11-19 | Cloud architecture log file batch processing method, cloud architecture log file batch processing device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011302079.5A CN112463725B (en) | 2020-11-19 | 2020-11-19 | Cloud architecture log file batch processing method, cloud architecture log file batch processing device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112463725A true CN112463725A (en) | 2021-03-09 |
| CN112463725B CN112463725B (en) | 2024-05-14 |
Family
ID=74836653
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011302079.5A Active CN112463725B (en) | 2020-11-19 | 2020-11-19 | Cloud architecture log file batch processing method, cloud architecture log file batch processing device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112463725B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115967614A (en) * | 2021-10-08 | 2023-04-14 | 大唐移动通信设备有限公司 | Network element log extraction method and device, electronic equipment and storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103326999A (en) * | 2012-12-14 | 2013-09-25 | 无锡华御信息技术有限公司 | File safety management system based on cloud service |
| CN105184154A (en) * | 2015-09-15 | 2015-12-23 | 中国科学院信息工程研究所 | System and method for providing cryptogrammic operation service in virtualized environment |
| CN106372859A (en) * | 2016-09-20 | 2017-02-01 | 中国银联股份有限公司 | Timing task control analysis method and system, and master control server |
| CN109600257A (en) * | 2018-12-07 | 2019-04-09 | 泰康保险集团股份有限公司 | Dispositions method, device and the storage medium of tool |
| CN110209562A (en) * | 2019-05-13 | 2019-09-06 | 深圳前海微众银行股份有限公司 | A kind of log analysis method and Analysis server |
| CN110290133A (en) * | 2019-06-25 | 2019-09-27 | 常熟市飞梦信息技术有限公司 | A kind of website cloud means of defence and device |
| CN110888778A (en) * | 2019-11-14 | 2020-03-17 | 西安雷风电子科技有限公司 | Cloud desktop-based log file monitoring system and method |
| CN111143286A (en) * | 2019-12-13 | 2020-05-12 | 无锡华云数据技术服务有限公司 | Cloud platform log management method and system |
| CN111324356A (en) * | 2018-12-17 | 2020-06-23 | 北京数安鑫云信息技术有限公司 | Software automation deployment method and system |
| CN111352691A (en) * | 2018-12-21 | 2020-06-30 | 北京神州泰岳软件股份有限公司 | Log collection method and system |
-
2020
- 2020-11-19 CN CN202011302079.5A patent/CN112463725B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103326999A (en) * | 2012-12-14 | 2013-09-25 | 无锡华御信息技术有限公司 | File safety management system based on cloud service |
| CN105184154A (en) * | 2015-09-15 | 2015-12-23 | 中国科学院信息工程研究所 | System and method for providing cryptogrammic operation service in virtualized environment |
| CN106372859A (en) * | 2016-09-20 | 2017-02-01 | 中国银联股份有限公司 | Timing task control analysis method and system, and master control server |
| CN109600257A (en) * | 2018-12-07 | 2019-04-09 | 泰康保险集团股份有限公司 | Dispositions method, device and the storage medium of tool |
| CN111324356A (en) * | 2018-12-17 | 2020-06-23 | 北京数安鑫云信息技术有限公司 | Software automation deployment method and system |
| CN111352691A (en) * | 2018-12-21 | 2020-06-30 | 北京神州泰岳软件股份有限公司 | Log collection method and system |
| CN110209562A (en) * | 2019-05-13 | 2019-09-06 | 深圳前海微众银行股份有限公司 | A kind of log analysis method and Analysis server |
| CN110290133A (en) * | 2019-06-25 | 2019-09-27 | 常熟市飞梦信息技术有限公司 | A kind of website cloud means of defence and device |
| CN110888778A (en) * | 2019-11-14 | 2020-03-17 | 西安雷风电子科技有限公司 | Cloud desktop-based log file monitoring system and method |
| CN111143286A (en) * | 2019-12-13 | 2020-05-12 | 无锡华云数据技术服务有限公司 | Cloud platform log management method and system |
Non-Patent Citations (1)
| Title |
|---|
| 刘川意 等: "基于小云审大云的云平台可信评测体系结构与技术研究", 网络与信息安全学报, vol. 2, no. 10, 15 October 2016 (2016-10-15), pages 36 - 47 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115967614A (en) * | 2021-10-08 | 2023-04-14 | 大唐移动通信设备有限公司 | Network element log extraction method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112463725B (en) | 2024-05-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107665234B (en) | Service processing method, device, server and storage medium | |
| RU2581560C2 (en) | Method of scanning files, client computer and server | |
| CN110781143B (en) | Method and device for querying and extracting server logs | |
| US9715588B2 (en) | Method of detecting a malware based on a white list | |
| CN101923617B (en) | Cloud-based sample database dynamic maintaining method | |
| KR20160125960A (en) | Virus processing method, apparatus, system and device, and computer storage medium | |
| US10122815B1 (en) | Dynamic class mapping in messaging middleware | |
| CN105893225A (en) | Automatic error processing method and device | |
| CN104536792A (en) | Method and device for eliminating application program residual files | |
| CN105871587A (en) | Log uploading method and device | |
| CN107426041B (en) | Method and device for analyzing command | |
| CN111159520B (en) | Sample identification method, device and safety emergency response system | |
| CN110968478A (en) | Log collection method, server and computer storage medium | |
| CN104331376A (en) | Method and device for clearing installation package stored in mobile terminal | |
| CN105718307A (en) | Process management method and process management device | |
| CN112463725B (en) | Cloud architecture log file batch processing method, cloud architecture log file batch processing device and storage medium | |
| CN111736876A (en) | Client update control method, device, storage medium and server | |
| US7987225B2 (en) | Method for remembering resource allocation in grids | |
| CN103473350B (en) | Document handling method and equipment | |
| CN107463487B (en) | Distributed host performance acquisition system and performance acquisition method | |
| CN111538542B (en) | System configuration method and related device | |
| US20180004626A1 (en) | Non-transitory computer-readable storage medium, evaluation method, and evaluation device | |
| JP2019144872A (en) | System having computation model for machine learning, and machine learning method | |
| CN110263060B (en) | ERP electronic accessory management method and computer equipment | |
| CN114880157B (en) | Fault injection method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |