CN112463725B - Cloud architecture log file batch processing method, cloud architecture log file batch processing device and storage medium - Google Patents

Cloud architecture log file batch processing method, cloud architecture log file batch processing device and storage medium Download PDF

Info

Publication number
CN112463725B
CN112463725B CN202011302079.5A CN202011302079A CN112463725B CN 112463725 B CN112463725 B CN 112463725B CN 202011302079 A CN202011302079 A CN 202011302079A CN 112463725 B CN112463725 B CN 112463725B
Authority
CN
China
Prior art keywords
log
log file
target host
task
processed
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011302079.5A
Other languages
Chinese (zh)
Other versions
CN112463725A (en
Inventor
路广岩
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Si Tech Information Technology Co Ltd
Original Assignee
Beijing Si Tech Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Si Tech Information Technology Co Ltd filed Critical Beijing Si Tech Information Technology Co Ltd
Priority to CN202011302079.5A priority Critical patent/CN112463725B/en
Publication of CN112463725A publication Critical patent/CN112463725A/en
Application granted granted Critical
Publication of CN112463725B publication Critical patent/CN112463725B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/11File system administration, e.g. details of archiving or snapshots
    • G06F16/122File system administration, e.g. details of archiving or snapshots using management policies
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/16File or folder operations, e.g. details of user interfaces specifically adapted to file systems
    • G06F16/162Delete operations
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/1805Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815Journaling file systems
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The invention provides a cloud architecture log file batch processing method, a cloud architecture log file batch processing device and a storage medium, wherein the cloud architecture log file batch processing method comprises the following steps: the target host reads the task to be processed from the queue, and sends an interaction request to the cloud server, wherein the interaction request carries a target host password; and the cloud server logs in the target host according to the target host password, processes the task to be processed, and when the task processing is completed, acquires the next task to be processed from the target host, processes the task until all the tasks to be processed in the target host are processed. According to the cloud server, the target host is managed in a cloud architecture mode, the target host can be logged in according to the target host password to be managed, the log file is processed through the task to be processed, the log file is convenient to modify secondarily, a cloud log maintenance mechanism is built, and management of log life cycle is enhanced.

Description

Cloud architecture log file batch processing method, cloud architecture log file batch processing device and storage medium
Technical Field
The invention mainly relates to the technical field of file batch processing, in particular to a cloud architecture log file batch processing method, a cloud architecture log file batch processing device and a storage medium.
Background
In the cloud architecture, the number of service-bearing hosts is increased, each service maintenance log is independent, log maintenance scripts are required to be deployed to each host, and the operation of secondary modification is very complicated; meanwhile, most logs lack maintenance mechanisms to influence the whole system log life cycle management. At present, no good solution to this problem is available.
Disclosure of Invention
The invention aims to solve the technical problem of overcoming the defects of the prior art and provides a cloud architecture log file batch processing method, a cloud architecture log file batch processing device and a storage medium.
The technical scheme for solving the technical problems is as follows: a cloud architecture log file batch processing method comprises the following steps:
The target host reads the task to be processed from the queue, and sends an interaction request to the cloud server, wherein the interaction request carries a target host password;
the cloud server logs in a target host according to the target host password, and the task to be processed performs task processing;
when the task processing is completed, the next task to be processed is acquired from the target host and the task processing is performed until the processing is completed on all the tasks to be processed in the target host.
The other technical scheme for solving the technical problems is as follows: a cloud architecture log file batch processing device comprises a target host and a cloud server;
The target host reads a task to be processed from the queue, and sends an interaction request to the cloud server, wherein the interaction request carries a target host password;
And the cloud server logs in the target host according to the target host password, processes the task to be processed, and when the task processing is completed, acquires the next task to be processed from the target host, processes the task until the processing is completed on all the tasks to be processed in the target host.
The other technical scheme for solving the technical problems is as follows: the cloud architecture log file batch processing device comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the cloud architecture log file batch processing method is realized when the processor executes the computer program.
The other technical scheme for solving the technical problems is as follows: a computer readable storage medium storing a computer program which, when executed by a processor, implements a cloud architecture log file batch processing method as described above.
The beneficial effects of the invention are as follows: the cloud server manages the target host in a cloud architecture mode, can log in the target host according to a target host password to be managed, processes the log file through a task to be processed, is convenient for modifying the log file secondarily, builds a cloud log maintenance mechanism, and strengthens management of log life cycle.
Drawings
Fig. 1 is a flow chart of a batch processing method for log files of a cloud architecture according to an embodiment of the present invention;
fig. 2 is a schematic functional module diagram of a batch processing device for log files of a cloud architecture according to an embodiment of the present invention;
Fig. 3 is a schematic connection diagram of a cloud server and a plurality of target hosts according to an embodiment of the present invention.
Detailed Description
The principles and features of the present invention are described below with reference to the drawings, the examples are illustrated for the purpose of illustrating the invention and are not to be construed as limiting the scope of the invention.
Fig. 1 is a flow chart of a batch processing method for log files of a cloud architecture according to an embodiment of the present invention.
As shown in fig. 1, a batch processing method for cloud architecture log files includes the following steps:
The target host reads the task to be processed from the queue, and sends an interaction request to the cloud server, wherein the interaction request carries a target host password;
The cloud server logs in the target host according to the target host password and performs task processing on the task to be processed;
when the task processing is completed, the next task to be processed is acquired from the target host and the task processing is performed until the processing is completed on all the tasks to be processed in the target host.
Specifically, script of the main program is written using expect automatic interaction function, the main program controls business logic, and commands built in using expect such as: spawn, expect, send, interact, etc. to enable automatic interaction with the target host.
Processing rules for the target host file when automatically interacting with the target host are stored in the configuration file and split with the program logic. The target host login password, etc. are stored in another configuration file. The configuration file is stored in the cloud server.
Fig. 3 is a schematic connection diagram of a cloud server and a plurality of target hosts according to an embodiment of the present invention.
It should be understood that, as shown in fig. 3, the cloud server is not limited to the case of interacting with only one target host, and the cloud server can interact with n target hosts simultaneously through the target host passwords of different hosts.
In the above embodiment, the cloud server manages the target host in a cloud architecture manner, can log in the target host according to the target host password to be managed, and processes the log file through the task to be processed, so that the log file is convenient to modify for the second time, a cloud log maintenance mechanism is constructed, and the management of the log life cycle is enhanced.
Optionally, as an embodiment of the present invention, the process of logging in the target host by the cloud server according to the target host password includes:
And matching the password of the target host through a pre-stored password configuration file, and logging in the target host if the matching is successful.
In the above embodiment, the target host logs in the target host quickly and safely by means of the target host password, so that management is convenient for different target hosts.
Optionally, as an embodiment of the present invention, the task processing process for the task to be processed includes:
Searching a corresponding log cleaning strategy configuration file according to the task to be processed, and cleaning the log files in the target host according to the log cleaning strategy configuration file.
In the embodiment, the log maintenance strategy is uniformly managed, is easy to maintain and upgrade, has good expansibility, and meets most of log maintenance requirements.
Optionally, as an embodiment of the present invention, the process of cleaning the log file in the target host according to the log cleaning policy configuration file includes:
If the task to be processed is a direct deleting log instruction, acquiring a log file name from the direct deleting log instruction, searching a log file storage path from the log cleaning strategy configuration file according to the log file name, searching a log file according to the log file storage path, and deleting the log file and the log file storage path together.
In the above embodiment, the deletion processing can be performed on the log file corresponding to the specified log file name in a targeted manner.
Optionally, as an embodiment of the present invention, the process of cleaning the log file in the target host according to the log cleaning policy configuration file further includes:
If the task to be processed is a fuzzy matching deleting log instruction, acquiring a part of log file names from the fuzzy matching deleting log instruction, searching a plurality of associated log file storage paths from the log cleaning strategy configuration file according to the part of log file names, searching corresponding log files according to the plurality of log file storage paths respectively, and deleting the searched log files and the corresponding log file storage paths together.
In the above embodiment, the log files corresponding to the log file names meeting the fuzzy matching condition can be deleted in batch, so that the log processing efficiency can be improved.
Optionally, as an embodiment of the present invention, the process of cleaning the log file in the target host according to the log cleaning policy configuration file further includes:
if the task to be processed is a compressed log instruction, acquiring a log file name from the compressed log instruction, searching a log file storage path from the log cleaning strategy configuration file according to the log file name, searching a log file according to the log file storage path, and compressing the log file.
In the above embodiment, the log files can be uniformly compressed, so that the log processing efficiency can be improved.
Optionally, as an embodiment of the present invention, the process of cleaning the log file in the target host according to the log cleaning policy configuration file further includes:
If the task to be processed is a time-based log cleaning instruction, acquiring time information from the time-based log cleaning instruction, searching a matched log file name from the log cleaning strategy configuration file according to the time information, searching a log file storage path according to the log file name, searching a log file according to the log file storage path, and deleting the searched log file and the corresponding log file storage path together.
In the above embodiment, the log files corresponding to the log file names meeting the time conditions can be deleted in batch, so that the log processing efficiency can be improved.
Fig. 2 is a schematic functional module diagram of a batch processing device for log files with a cloud architecture according to an embodiment of the present invention.
Optionally, as an embodiment of the present invention, as shown in fig. 2, a batch processing device for a log file of a cloud architecture includes a target host and a cloud server;
The target host reads a task to be processed from the queue, and sends an interaction request to the cloud server, wherein the interaction request carries a target host password;
And the cloud server logs in the target host according to the target host password, processes the task to be processed, and when the task processing is completed, acquires the next task to be processed from the target host, processes the task until the processing is completed on all the tasks to be processed in the target host.
Fig. 3 is a schematic connection diagram of a cloud server and a plurality of target hosts according to an embodiment of the present invention.
It should be understood that, as shown in fig. 3, the cloud server is not limited to the case of interacting with only one target host, and the cloud server can interact with n target hosts simultaneously through the target host passwords of different hosts. The efficiency of processing logs can be improved.
Alternatively, as another embodiment of the present invention, a batch processing device for a log file of a cloud architecture includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the batch processing method for a log file of a cloud architecture is implemented when the processor executes the computer program.
Alternatively, as another embodiment of the present invention, a computer-readable storage medium storing a computer program that when executed by a processor implements the cloud architecture log file batch processing method as described above.
According to the cloud server, the target host is managed in a cloud architecture mode, the target host can be logged in according to the target host password to be managed, the log file is processed through the task to be processed, the log file is convenient to modify secondarily, a cloud log maintenance mechanism is built, and management of log life cycle is enhanced. The log maintenance strategy is uniformly managed, is easy to maintain and upgrade, has good expansibility and meets most of log maintenance requirements.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
The foregoing description of the preferred embodiments of the invention is not intended to limit the invention to the precise form disclosed, and any such modifications, equivalents, and alternatives falling within the spirit and scope of the invention are intended to be included within the scope of the invention.

Claims (7)

1. The cloud architecture log file batch processing method is characterized by comprising the following steps of:
The target host reads a task to be processed for processing the log file from the queue, and sends an interaction request to the cloud server, wherein the interaction request carries a target host password;
The cloud server logs in the target host according to the target host password and performs task processing on the task to be processed;
When the task processing is completed, acquiring a next task to be processed from the target host and performing task processing until all the tasks to be processed in the target host are processed;
The task processing process for the task to be processed comprises the following steps:
Searching a corresponding log cleaning strategy configuration file according to the task to be processed, and cleaning the log files in the target host according to the log cleaning strategy configuration file;
the process of cleaning the log file in the target host according to the log cleaning policy configuration file further includes:
if the task to be processed is a fuzzy matching deleting log instruction, acquiring a part of log file names from the fuzzy matching deleting log instruction, searching a plurality of associated log file storage paths from the log cleaning strategy configuration file according to the part of log file names, respectively searching corresponding log files according to the plurality of log file storage paths, and deleting the searched log files and the corresponding log file storage paths together;
the process of cleaning the log file in the target host according to the log cleaning policy configuration file further includes:
If the task to be processed is a time-based log cleaning instruction, acquiring time information from the time-based log cleaning instruction, searching a matched log file name from the log cleaning strategy configuration file according to the time information, searching a log file storage path according to the log file name, searching a log file according to the log file storage path, and deleting the searched log file and the corresponding log file storage path together.
2. The batch processing method of cloud architecture log files according to claim 1, wherein the process of logging in the target host by the cloud server according to the target host password comprises the following steps:
And matching the password of the target host through a pre-stored password configuration file, and logging in the target host if the matching is successful.
3. The batch processing method of the log files of the cloud architecture according to claim 1, wherein the process of cleaning the log files in the target host according to the log cleaning policy configuration file includes:
If the task to be processed is a direct deleting log instruction, acquiring a log file name from the direct deleting log instruction, searching a log file storage path from the log cleaning strategy configuration file according to the log file name, searching a log file according to the log file storage path, and deleting the log file and the log file storage path together.
4. The batch processing method of log files of cloud architecture according to claim 1, wherein the process of cleaning log files in the target host according to the log cleaning policy configuration file further comprises:
if the task to be processed is a compressed log instruction, acquiring a log file name from the compressed log instruction, searching a log file storage path from the log cleaning strategy configuration file according to the log file name, searching a log file according to the log file storage path, and compressing the log file.
5. The cloud architecture log file batch processing device is characterized by comprising a target host and a cloud server;
the target host reads a task to be processed for processing the log file from the queue, and sends an interaction request to the cloud server, wherein the interaction request carries a target host password;
The cloud server logs in a target host according to the target host password, performs task processing on the task to be processed, and when the task processing is completed, acquires the next task to be processed from the target host, and performs task processing until all the tasks to be processed in the target host are processed;
The task processing process for the task to be processed comprises the following steps:
Searching a corresponding log cleaning strategy configuration file according to the task to be processed, and cleaning the log files in the target host according to the log cleaning strategy configuration file;
the process of cleaning the log file in the target host according to the log cleaning policy configuration file further includes:
if the task to be processed is a fuzzy matching deleting log instruction, acquiring a part of log file names from the fuzzy matching deleting log instruction, searching a plurality of associated log file storage paths from the log cleaning strategy configuration file according to the part of log file names, respectively searching corresponding log files according to the plurality of log file storage paths, and deleting the searched log files and the corresponding log file storage paths together;
the process of cleaning the log file in the target host according to the log cleaning policy configuration file further includes:
If the task to be processed is a time-based log cleaning instruction, acquiring time information from the time-based log cleaning instruction, searching a matched log file name from the log cleaning strategy configuration file according to the time information, searching a log file storage path according to the log file name, searching a log file according to the log file storage path, and deleting the searched log file and the corresponding log file storage path together.
6. A cloud architecture log file batch processing device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the cloud architecture log file batch processing method according to any one of claims 1 to 4 is implemented when the computer program is executed by the processor.
7. A computer readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the cloud architecture log file batch processing method of any one of claims 1 to 4.
CN202011302079.5A 2020-11-19 2020-11-19 Cloud architecture log file batch processing method, cloud architecture log file batch processing device and storage medium Active CN112463725B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011302079.5A CN112463725B (en) 2020-11-19 2020-11-19 Cloud architecture log file batch processing method, cloud architecture log file batch processing device and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011302079.5A CN112463725B (en) 2020-11-19 2020-11-19 Cloud architecture log file batch processing method, cloud architecture log file batch processing device and storage medium

Publications (2)

Publication Number Publication Date
CN112463725A CN112463725A (en) 2021-03-09
CN112463725B true CN112463725B (en) 2024-05-14

Family

ID=74836653

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011302079.5A Active CN112463725B (en) 2020-11-19 2020-11-19 Cloud architecture log file batch processing method, cloud architecture log file batch processing device and storage medium

Country Status (1)

Country Link
CN (1) CN112463725B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115967614A (en) * 2021-10-08 2023-04-14 大唐移动通信设备有限公司 Network element log extraction method and device, electronic equipment and storage medium

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103326999A (en) * 2012-12-14 2013-09-25 无锡华御信息技术有限公司 File safety management system based on cloud service
CN105184154A (en) * 2015-09-15 2015-12-23 中国科学院信息工程研究所 System and method for providing cryptogrammic operation service in virtualized environment
CN106372859A (en) * 2016-09-20 2017-02-01 中国银联股份有限公司 Timing task control analysis method and system, and master control server
CN109600257A (en) * 2018-12-07 2019-04-09 泰康保险集团股份有限公司 Dispositions method, device and the storage medium of tool
CN110209562A (en) * 2019-05-13 2019-09-06 深圳前海微众银行股份有限公司 A kind of log analysis method and Analysis server
CN110290133A (en) * 2019-06-25 2019-09-27 常熟市飞梦信息技术有限公司 A kind of website cloud means of defence and device
CN110888778A (en) * 2019-11-14 2020-03-17 西安雷风电子科技有限公司 Cloud desktop-based log file monitoring system and method
CN111143286A (en) * 2019-12-13 2020-05-12 无锡华云数据技术服务有限公司 Cloud platform log management method and system
CN111324356A (en) * 2018-12-17 2020-06-23 北京数安鑫云信息技术有限公司 Software automation deployment method and system
CN111352691A (en) * 2018-12-21 2020-06-30 北京神州泰岳软件股份有限公司 Log collection method and system

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103326999A (en) * 2012-12-14 2013-09-25 无锡华御信息技术有限公司 File safety management system based on cloud service
CN105184154A (en) * 2015-09-15 2015-12-23 中国科学院信息工程研究所 System and method for providing cryptogrammic operation service in virtualized environment
CN106372859A (en) * 2016-09-20 2017-02-01 中国银联股份有限公司 Timing task control analysis method and system, and master control server
CN109600257A (en) * 2018-12-07 2019-04-09 泰康保险集团股份有限公司 Dispositions method, device and the storage medium of tool
CN111324356A (en) * 2018-12-17 2020-06-23 北京数安鑫云信息技术有限公司 Software automation deployment method and system
CN111352691A (en) * 2018-12-21 2020-06-30 北京神州泰岳软件股份有限公司 Log collection method and system
CN110209562A (en) * 2019-05-13 2019-09-06 深圳前海微众银行股份有限公司 A kind of log analysis method and Analysis server
CN110290133A (en) * 2019-06-25 2019-09-27 常熟市飞梦信息技术有限公司 A kind of website cloud means of defence and device
CN110888778A (en) * 2019-11-14 2020-03-17 西安雷风电子科技有限公司 Cloud desktop-based log file monitoring system and method
CN111143286A (en) * 2019-12-13 2020-05-12 无锡华云数据技术服务有限公司 Cloud platform log management method and system

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
基于小云审大云的云平台可信评测体系结构与技术研究;刘川意 等;网络与信息安全学报;20161015;第2卷(第10期);36-47 *

Also Published As

Publication number Publication date
CN112463725A (en) 2021-03-09

Similar Documents

Publication Publication Date Title
US8108456B2 (en) Method and apparatus for migrating the system environment on which the applications depend
CN110765026B (en) Automatic test method, device, storage medium and equipment
CN108845930B (en) Interface operation test method and device, storage medium and electronic device
CN102970272B (en) Method, device and cloud server for detesting viruses
KR20160125960A (en) Virus processing method, apparatus, system and device, and computer storage medium
US10122815B1 (en) Dynamic class mapping in messaging middleware
CN105871587A (en) Log uploading method and device
CN110188543A (en) White list library, white list program library update method and industrial control system
CN111159520B (en) Sample identification method, device and safety emergency response system
CN104536792A (en) Method and device for eliminating application program residual files
US20200097842A1 (en) Method and system for predicting and preempting patching failures
CN107368343A (en) A kind of starting up of terminal method, terminal and storage medium based on Android system
CN110502366A (en) Case executes method, apparatus, equipment and computer readable storage medium
CN112463725B (en) Cloud architecture log file batch processing method, cloud architecture log file batch processing device and storage medium
US20200241996A1 (en) Distributed web page performance monitoring methods and systems
CN112953983A (en) SFTP transmission method and device
CN108255735B (en) Associated environment testing method, electronic device and computer readable storage medium
CN105718307A (en) Process management method and process management device
CN111736876A (en) Client update control method, device, storage medium and server
CN110677390B (en) Abnormal account identification method and device, electronic equipment and storage medium
CN114880157B (en) Fault injection method and device
CN116303320A (en) Real-time task management method, device, equipment and medium based on log file
CN105868957A (en) Continuous integration method and device
CN113553595B (en) Vulnerability scanning method, vulnerability scanning device, vulnerability scanning equipment and storage medium
CN111209051B (en) Optimization acceleration method and device based on domestic operating system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant