CN108255665B - Resource monitoring method, device and equipment for computing task and readable storage medium - Google Patents
Resource monitoring method, device and equipment for computing task and readable storage medium Download PDFInfo
- Publication number
- CN108255665B CN108255665B CN201710900536.2A CN201710900536A CN108255665B CN 108255665 B CN108255665 B CN 108255665B CN 201710900536 A CN201710900536 A CN 201710900536A CN 108255665 B CN108255665 B CN 108255665B
- Authority
- CN
- China
- Prior art keywords
- page
- task execution
- resource
- data
- unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention provides a resource monitoring method, a resource monitoring device, a resource monitoring equipment and a computer readable storage medium for a computing task. The resource monitoring method of the computing task comprises the following steps: acquiring data of a page where a running task execution unit of a resident service in a computing task is located by utilizing a webpage crawler technology; analyzing the number of the running task execution units according to the data of the page; calculating and storing the resource usage amount of the resident service according to the linear relation between the amount of the running task execution units and the resource usage amount; counting the resource usage number in a preset time period; and showing the counted resource usage amount through the view. The embodiment of the invention visually shows the resource use state of the resident service in the computing task, can realize the resource monitoring of the resident service in the computing task, and can reasonably distribute the resources used by the resident service.
Description
Technical Field
The present invention relates to the field of information processing technologies, and in particular, to a method, an apparatus, a device, and a computer-readable storage medium for monitoring resources of a computing task.
Background
Distributed computing systems (Spark) are fast, general-purpose computing engines designed specifically for large-scale data processing. In big data processing, when a Spark executes a computing task, resources used, such as the number of CPU cores needed to be used, are correspondingly allocated to resident services (continuously running services) related to the computing task, such as Spark client (service for providing database connection), Spark streaming (service for processing data volume), and the like. However, the resource usage state of the resident service in the computing task is difficult to count, so that the resident service in the computing task has the following problems: 1. the resource utilization rate of resident services in the computing task is difficult to obtain, and a 'black box' phenomenon exists in the resource utilization rate; 2. since the resource allocation of the resident service in the computing task is difficult to predict, when the resource used by the resident service is allocated, the resource is allocated as much as possible to ensure that the application can normally run, but the resource allocation is excessive, and the phenomenon of resource waste exists.
Disclosure of Invention
The embodiment of the invention provides a resource monitoring method, a resource monitoring device, equipment and a computer readable storage medium for a computing task, which visually show the resource use state of a resident service in the computing task, can realize the resource monitoring of the resident service in the computing task, and can reasonably allocate the resource used by the resident service.
In a first aspect, an embodiment of the present invention provides a method for monitoring resources of a computing task, where the method includes:
acquiring data of a page where a running task execution unit of a resident service in a computing task is located by utilizing a webpage crawler technology;
analyzing the number of the running task execution units according to the data of the page;
calculating and storing the resource usage amount of the resident service according to the linear relation between the amount of the running task execution units and the resource usage amount;
counting the resource usage number in a preset time period;
and showing the counted resource usage amount through the view.
In a second aspect, an embodiment of the present invention provides a resource monitoring apparatus for a computing task, where the apparatus includes a unit configured to perform resource monitoring for the computing task according to the first aspect.
In a third aspect, an embodiment of the present invention further provides an apparatus, where the apparatus includes a memory and a processor connected to the memory;
the memory is configured to store program data for implementing resource monitoring of a computing task, and the processor is configured to execute the program data stored in the memory to perform the resource monitoring method for a computing task according to the first aspect.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where one or more program data are stored, and the one or more program data are executable by one or more processors to implement the resource monitoring method for a computing task according to the first aspect.
In the embodiment of the invention, the data of the page where the running task execution unit of the resident service in the computing task is located is obtained by utilizing the webpage crawler technology; analyzing the number of the running task execution units according to the data of the page; calculating and storing the resource usage amount of the resident service according to the linear relation between the amount of the running task execution units and the resource usage amount; counting the resource usage number in a preset time period; and showing the counted resource usage amount through the view. The embodiment of the invention visually shows the resource use state of the resident service in the computing task, can realize the resource monitoring of the resident service in the computing task, and can reasonably distribute the resources used by the resident service.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic structural component diagram of a distributed computing system according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a resource monitoring method for a computing task according to an embodiment of the present invention;
FIG. 3 is a schematic sub-flow chart of the method of FIG. 2 according to an embodiment of the present invention;
FIG. 4 is a schematic illustration of another sub-flow chart of the method of FIG. 2 according to an embodiment of the present invention;
FIG. 5 is a schematic illustration of another sub-flow chart of the method of FIG. 2 according to an embodiment of the present invention;
FIG. 6 is a flowchart illustrating a resource monitoring method for a computing task according to another embodiment of the present invention;
FIG. 7 is a schematic block diagram of a resource monitoring apparatus for computing tasks according to an embodiment of the present invention;
FIG. 8 is a schematic block diagram of a page retrieval unit provided by an embodiment of the present invention;
FIG. 9 is a schematic block diagram of a parsing unit provided by an embodiment of the invention;
FIG. 10 is a schematic block diagram of a statistics unit provided by an embodiment of the present invention;
FIG. 11 is a schematic block diagram of a resource monitoring apparatus for computing tasks according to another embodiment of the present invention;
FIG. 12 is a schematic block diagram of a resource monitoring device for computing tasks according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It should also be understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
Fig. 1 is a schematic diagram of an architecture composition of a distributed computing system (Apache Spark, Spark for short). The principle of operation of a computational task in a distributed computing system will now be briefly described by means of figure 1. When a computing task is submitted in a distributed computing system, the computing task starts a corresponding master control process (Driver Program). The main control process can occupy a certain amount of internal memory and CPU core according to the set parameters. One thing the master process needs to do is to apply for an execution process (execution) that needs to be used to run the computing task from a Cluster resource Manager (a Cluster Manager may specifically be a spare standard Cluster, and the deployment mode of the spare standard Cluster is the simplest one in the Cluster mode, or may be another resource management Cluster, such as a YARN resource management Cluster). The cluster resource manager starts a certain number of execution processes (executors) on each work Node (Worker Node, which may be understood as a physical Node) according to the resource parameters set for the computing task application, where the resource parameters include the resource usage number of the resident service, and each execution process occupies a certain number of memories and CPU cores.
After the computing task has applied for the resources needed for execution, the host process will begin scheduling and executing the code for the computing task. The master control process divides the written calculation task code into a plurality of stages (stages), each stage executes a part of code fragments, creates a batch of task execution units (tasks) for each stage, and then distributes the task execution units to the execution processes for execution. The task execution unit (task) is the smallest computational unit. The execution speed of the task execution unit is directly related to the number of CPU cores executing each process. A CPU core can only execute one thread at a time. And the plurality of task execution units distributed on each execution process are simultaneously operated in a multi-thread mode in a mode of one task execution unit and one thread. It is understood that a CPU core is required for a task execution unit to execute. If the number of CPU cores is sufficient and the number of assigned task execution units is reasonable, then generally speaking, these task execution unit threads can be executed relatively quickly and efficiently.
However, it is difficult to count the resource usage status of a computing task, such as the resource usage status of a resident service, and thus it is difficult to count the resource usage of a resident service in a computing task. Since the resource utilization rate of the resident service in a computing task is difficult to count, when the resource used by the resident service is allocated, the application can normally run only by allocating as much as possible, which may cause excessive resource allocation and resource waste.
It should be noted that, without specific description, the distributed computing system referred to in this application is referred to as Spark, the master process is referred to as Driver Program, the Cluster resource Manager is referred to as Cluster Manager, the worker node is referred to as WorkerNode, the execution process is referred to as execution, the phase is referred to as stage, the task execution unit is referred to as task, and the running task execution unit is referred to as running task. The resident service is a service which is continuously operated after being opened, and has occupied resources all the time. In a distributed computing system, for example, sparkthriftserver (a service providing database connection), spark streaming (a service processing data volume), and the like are included.
Fig. 2 is a schematic flowchart of a resource monitoring method for a computing task according to an embodiment of the present invention. The method is premised on that: when the distributed computing system is started, the web UI service corresponding to the distributed computing system is also started, and the running task execution unit of the task scheduling is displayed on a page (web page user interface) provided by the web UI service. The method operates in a server in which a distributed computing system is installed. As shown in fig. 1, the method comprises the following steps S201-S205.
S201, acquiring data of a page where a running task execution unit of a resident service in a computing task is located by utilizing a webpage crawler technology. The web crawler technology is a technology for automatically capturing a program or script of web information according to a certain rule. The web crawler technology is widely used for an internet search engine or other similar websites, and can automatically collect all page contents which can be accessed by the websites so as to acquire or update the contents and retrieval modes of the websites. In the system framework of the web crawler, the main process consists of three parts, namely a controller, a resolver and a resource library. The main task of the controller is to be responsible for assigning work tasks to each crawler thread in multiple threads. The parser mainly downloads the webpage and processes the webpage, mainly processes some JS script tags, CSS code content, space characters, HTML tags and other content, extracts the content of special HTML tags and analyzes data in HTML, and the basic work of the crawler is completed by the parser. The resource library is used to store downloaded web page resources, and generally adopts a large database storage, such as an Oracle database, and establishes an index for the database storage. In the present application, the functions of the parser in the web crawler technology are mainly used.
Specifically, as shown in fig. 3, the data of the page where the running task execution unit of the service resident in the computing task is located is obtained by using the web crawler technology, i.e., step S201 includes substeps S301-S302. S301, acquiring uniform resource locator parameters of the webpage user interface in the calculation task. Wherein, the web page user interface refers to a page opened by a web UI service of the distributed computing system; uniform resource locator refers to a URL, such as http:// www.xxxx.com: 80/yyy; the uniform resource locator parameter refers to a URL parameter. The URL parameters comprise static character parameters and dynamic character parameters. The static character parameter may be in the form of an IP plus port, such as http:// IP; or in the form of a domain name plus port, such as http:// www.xxx.com: port. The port may be a default port 80 (the default port is not shown), or may be another available port that is customized, such as 8080, 8088, etc. Such as http:// hdp. app. pic. com. cn: 8088. The dynamic character parameters include the name of the computing task, etc., such as jdbc _ hdiser 0102_ 400. S302, according to the uniform resource locator parameters, the data of the page where the running task execution unit of the resident service in the calculation task is located is obtained by utilizing the web crawler technology. When using web crawler technology, it is often implemented with a web crawler tool, such as the web crawler tool pythonoulli, in which many methods are provided that need to be used in a web crawler. The use of the pythonoulli tool allows us to read the data on the world wide web www and ftp just like reading a local file, which can download the html file where the URL is located to a local hard disk or store it as a temporary file. And inputting the name of the page where the running task execution unit of the resident service in the computing task is located by using a python url tool according to the acquired uniform resource locator parameter, wherein the name of the page where the running task execution unit of the resident service in the computing task is located is stages (which can be other names), such as http:// www.xxx.com: 8088/yy/xyxy/stages/page, so as to acquire the data of the page. The method for acquiring data of a stages page where a running task execution unit of a resident service in a computing task is located by using a pythonoulli tool specifically includes: introducing into a urllib library; calling a method in the url lib library to obtain data of the pages of the books, such as http:// www.xxx.com: 8088/yy/xyxy/books/pages; and saving the acquired data of the pages of the stages. If the obtained data of the pages of the locations is desired to be further viewed, the obtained data of the pages of the locations can be output.
And S202, analyzing the number of the running task execution units of the resident service according to the data of the page.
Specifically, as shown in fig. 4, the number of running task execution units of the resident service is parsed from the data of the page, i.e., step S202 includes sub-steps S401 to S402. S401, acquiring the tags of all the running task execution units of the resident service from the data of the page where the running task execution unit of the resident service is acquired. I.e. get the tag in which runningtask is located in the page. The label comprises a label name and a value corresponding to the label name. S402, counting corresponding values in the labels of all the running task execution units as the number of the running task execution units. For example, the statistical id is the corresponding value in the label of "runningtask", and 0+30 is 30. And taking the calculated 30 as the number of the running task execution units. In this way, the number of running task execution units of the services resident in the computing task is obtained.
And S203, calculating and storing the resource usage amount of the resident service according to the linear relation between the number of the running task execution units and the resource usage amount. Preferably, the resource refers to a CPU core, and if one CPU core corresponds to one device, the CPU core used by the resident service can be understood as the number of occupied devices. The number of running task execution units of the resident service is a linear relationship with the number of resource usages, which has been predicted in advance. For example, a running task execution unit corresponds to a CPU core, and it can be understood that a running task execution unit corresponds to a thread, and a thread runs on a CPU core. In other embodiments, there are multiple CPU cores corresponding to one device, and if there are two CPU cores, one running task execution unit corresponds to two CPU cores. In a computing task, usually a plurality of running task execution units execute simultaneously.
The resource usage amount of the resident application for a long time can be acquired through steps S201 to S203.
And S204, counting the resource usage number in a preset time period.
Specifically, as shown in fig. 5, the statistics of the number of resource usage within the preset time period, i.e., S204, includes sub-steps S501-S502. S501, receiving a query instruction, wherein the query instruction comprises a corresponding preset time period. Wherein the preset time period may be one hour, one day, one week, one month, three months, half a year, etc. The preset time period includes a time period between any one of the start points and any one of the end time points, for example, the preset time period may be a preset time period in which the current time is the end time point, or a preset time period in which the current time is earlier than the end time period. The corresponding preset time period in the query command has a default value, such as one month. The query instruction further includes a corresponding query object, such as a CPU core. And S502, counting the resource usage amount of the resident service according to the preset time period corresponding to the query instruction. It can be understood that the resource usage amount at a plurality of corresponding time points in the preset time period is obtained. In other embodiments, counting the number of resource usages of the resident service further includes calculating a resource usage rate based on the counted number of resource usages. Such as calculating the usage rate of the CPU core according to the counted usage number of the CPU core.
And S205, displaying the counted resource usage amount through the view. The usage amount of the resource to be displayed can be displayed through the distributed system monitoring and network monitoring tool. Wherein the distributed system monitoring and network monitoring tool comprises zabbix. zabbix can monitor various network parameters to ensure the safe operation of the system. Specifically, the resource usage is displayed in a curve form, so that the user can see the corresponding resource usage within the preset time period at a glance. For example, the condition of the resource usage amount in the preset time period is shown on a coordinate system with time as an x axis and the resource usage amount as a y axis. The usage amount of the resource to be displayed can be displayed by other suitable tools or suitable manners. For example, when zabbix is used to show the statistical resource usage amount, the query instruction in step S204 may be obtained through a time bar selected/input by the user in zabbix and a selected query object, where the corresponding time on the time bar is a preset time period, and the selected query object includes a CPU core and the like. If the statistics of the resource usage in step 204 are, it can be understood that the resource usage is also shown in this step.
In other embodiments, step S204 further includes setting a time interval, and displaying the counted resource usage amount (resource usage rate) according to the time interval through the view. It is understood that if the preset time period is 1 year, the resource usage amount (resource usage rate) of 1 year is shown in the view. Because there are many corresponding time points in a year, the condition of resource usage quantity (resource usage rate) only needs to be shown in the view; on the other hand, the values of the continuously acquired resource usage amount (resource usage rate) are not particularly different in theory; and counting the resource usage amount per hour in 1 year also needs to occupy some resources. In terms of practicality, a time interval may be set in the view to show the amount of resource usage (resource usage rate) in 1 year. Such as 1 minute apart, etc.
In the embodiment, the data of the page where the running task execution unit of the resident service in the computing task is located is acquired by utilizing a webpage crawler technology; analyzing the number of the running task execution units according to the data of the page; calculating and storing the resource usage amount of the resident service according to the linear relation between the amount of the running task execution units and the resource usage amount; counting the resource usage number in a preset time period; and showing the counted resource usage amount through the view. According to the embodiment of the invention, the resource monitoring of the resident service in the computing task is realized according to the linear relation between the number of the running task execution units of the resident service in the computing task and the resource use number, and the resource use state of the resident service in the computing task is displayed in a visual mode; by displaying the resource use state in the preset time period, the resource use saturation of the resident service in the computing task is displayed, a reliable basis is provided for resource allocation of the resident service in the computing task, resources used by the resident service in the computing task can be allocated more reasonably, and resource waste caused by excessive allocation of resources is avoided; and through the displayed resource use state, a basis can be provided for performance optimization of the calculation resident service, and the analysis of problems of the resident service in the calculation task is facilitated, wherein problems may exist if the displayed resource use state is suddenly and greatly increased.
Fig. 6 is a flowchart illustrating a resource monitoring method for a computing task according to another embodiment of the present invention. This embodiment of the method includes S601-S606. This embodiment differs from the embodiment shown in fig. 1 in that: s601 is added. For details of other steps, please refer to the description of corresponding steps in the embodiment of fig. 1, which is not repeated herein.
S601, setting a time interval. The time intervals are, for example, 1 minute, 5 minutes, etc.
And S602, acquiring data of a page where a running task execution unit of the resident service in the computing task is located by utilizing a webpage crawler technology according to the time interval.
Since a computation task generally does not have a sudden and large increase or sudden and large decrease in resource usage, the number of continuously acquired multiple resource usages may not be numerically different. If the resource usage amount is acquired every moment, on one hand, acquiring the resource usage amount needs to occupy some resources, such as CPU resources, memory resources and the like; on the other hand, the use quantity of a plurality of resources which are continuously obtained has limited reference significance due to small difference in numerical value. Therefore, the time interval is set so as to obtain the number of the running task execution units of the resident service in the calculation task according to the time interval, and the efficiency of resource monitoring is further improved. It is understood that the number of running task execution units of the resident service in the computing task is obtained according to the time interval, and then the statistical resource usage number shown in the view is also shown according to the time interval.
FIG. 7 is a schematic block diagram of a resource monitoring apparatus for a computing task according to an embodiment of the present invention. The device executes on the premise that: when the distributed computing system is started, the web UI service corresponding to the distributed computing system is also started, and the running task execution unit of the task scheduling is displayed on a page (web page user interface) provided by the web UI service. The device 70 includes a page obtaining unit 701, an analyzing unit 702, a calculating unit 703, a counting unit 704, and a displaying unit 705.
The page obtaining unit 701 is configured to obtain data of a page where a running task execution unit of a service resident in a computing task is located by using a web crawler technology. The web crawler technology is a technology for automatically capturing a program or script of web information according to a certain rule. The web crawler technology is widely used for an internet search engine or other similar websites, and can automatically collect all page contents which can be accessed by the websites so as to acquire or update the contents and retrieval modes of the websites.
Specifically, as shown in fig. 8, the page acquisition unit includes a parameter acquisition unit 801 and a page data acquisition unit 802. The parameter obtaining unit 801 is configured to obtain a uniform resource locator parameter of a web page user interface in a computing task. Wherein, the web page user interface refers to a page opened by a web UI service of the distributed computing system; uniform resource locator refers to a URL, such as http:// www.xxxx.com: 80/yyy; the uniform resource locator parameter refers to a URL parameter. The URL parameters comprise static character parameters and dynamic character parameters. The static character parameter may be in the form of an IP plus port, such as http:// IP; or in the form of a domain name plus port, such as http:// www.xxx.com: port. The port may be a default port 80 (the default port is not shown), or may be another available port that is customized, such as 8080, 8088, etc. Such as http:// hdp. app. pic. com. cn: 8088. The dynamic character parameters include the name of the computing task, etc., such as jdbc _ hdiser 0102_ 400. The page data obtaining unit 802 is configured to obtain data of a page where a running task execution unit of a service resident in a computing task is located by using a web crawler technology according to the uniform resource locator parameter. When using web crawler technology, it is often implemented with a web crawler tool, such as the web crawler tool python urllib, in which many methods are provided that need to be used in a web crawler. The use of the pythonoulli tool allows us to read the data on the world wide web www and ftp just like reading a local file, which can download the html file where the URL is located to a local hard disk or store it as a temporary file. In the pythonoulli tool, the name of the page where the running task execution unit of the resident service in the computing task is located is input according to the obtained uniform resource locator parameter, wherein the name of the page where the running task execution unit of the resident service in the computing task is located is stages (other names can be used), such as http:// www.xxx.com: 8088/yy/xyxy/stages/page, so as to obtain the data of the page. The method for acquiring data of a stages page where a running task execution unit of a resident service in a computing task is located by using a python url tool specifically includes: introducing into a urllib library; calling a method in the url lib library to obtain data of the pages of the books, such as http:// www.xxx.com: 8088/yy/xyxy/books/pages; and saving the acquired data of the pages of the stages. If the obtained data of the pages of the locations is desired to be further viewed, the obtained data of the pages of the locations can be output.
The parsing unit 702 is configured to parse the number of the running task execution units of the resident service according to the data of the page.
Specifically, as shown in fig. 9, the parsing unit includes a tag obtaining unit 901 and a tag value counting unit 902. The tag obtaining unit 901 is configured to obtain tags of all running task execution units of the resident service from data of a page where the running task execution unit of the resident service is obtained. Namely, the label where the running task is located in the page is obtained. The label comprises a label name and a value corresponding to the label name. As in the following example, id ═ running task "; value is "0"; id ═ running task; value is "30". The tag value counting unit 902 is configured to count corresponding values in tags of all running task execution units as the number of the running task execution units. For example, the statistical id is the corresponding value in the label of "running task", and 0+30 is 30. And taking the calculated 30 as the number of the running task execution units. In this way, the number of running task execution units of the services resident in the computing task is obtained.
The computing unit 703 is configured to compute and store the resource usage amount of the resident service according to a linear relationship between the number of the running task execution units and the resource usage amount. Preferably, the resource refers to a CPU core, and if one CPU core corresponds to one device, the CPU core used by the resident service can be understood as the number of occupied devices. The number of running task execution units of the resident service is a linear relationship with the number of resource usages, which has been predicted in advance. For example, a running task execution unit corresponds to a CPU core, and it can be understood that a running task execution unit corresponds to a thread, and a thread runs on a CPU core. In other embodiments, there are multiple CPU cores corresponding to one device, and if there are two CPU cores, one running task execution unit corresponds to two CPU cores. In a computing task, usually a plurality of running task execution units execute simultaneously.
The counting unit 704 is configured to count the resource usage amount within a preset time period.
Specifically, as shown in fig. 10, the statistic unit includes a receiving unit 101 and a resource statistic unit 102. The receiving unit 101 is configured to receive an inquiry command, where the inquiry command includes a corresponding preset time period. Wherein the preset time period may be one hour, one day, one week, one month, three months, half a year, etc. The preset time period includes a time period between any one of the start points and any one of the end time points, for example, the preset time period may be a preset time period in which the current time is the end time point, or a preset time period in which the current time is earlier than the end time period. The corresponding preset time period in the query command has a default value, such as one month. The query instruction further includes a corresponding query object, such as a CPU core. The resource counting unit 102 is configured to count the resource usage amount of the resident service according to the preset time period corresponding to the query instruction. It can be understood that the resource usage amount at a plurality of corresponding time points in the preset time period is obtained. In other embodiments, counting the number of resource usages of the resident service further includes calculating a utilization, such as CPU core utilization, based on the counted number of resource usages.
The presentation unit 705 is configured to present the counted resource usage amount through a view. The usage amount of the resource to be displayed can be displayed through the distributed system monitoring and network monitoring tool. Wherein the distributed system monitoring and network monitoring tool comprises zabbix. zabbix can monitor various network parameters to ensure the safe operation of the system. Specifically, the resource usage is displayed in a curve form, so that the user can see the corresponding resource usage within the preset time period at a glance. For example, the condition of the resource usage amount in the preset time period is shown on a coordinate system with time as an x axis and the resource usage amount as a y axis. The usage amount of the resource to be displayed can be displayed by other suitable tools or suitable manners. If zabbix is used to show the counted resource usage amount, the query instruction in the counting unit 704 may be obtained through a time bar selected/input by the user in zabbix and a selected query object, where the corresponding time on the time bar is a preset time period, and the selected query object includes a CPU core and the like. If the statistics in the statistics unit is the resource utilization, it can be understood that the statistics in the presentation unit is also the resource utilization.
In other embodiments, the statistical unit further sets a time interval, and the presentation unit presents the statistical resource usage amount (resource usage rate) according to the time interval through the view. It is understood that if the preset time period is 1 year, the resource usage amount (resource usage rate) of 1 year is shown in the view. Because there are many corresponding time points in a year, the condition of resource usage quantity (resource usage rate) only needs to be shown in the view; on the other hand, the values of the continuously acquired resource usage amount (resource usage rate) are not particularly different in theory; and counting the resource usage amount per hour in 1 year also needs to occupy some resources. In terms of practicality, a time interval may be set in the view to show the amount of resource usage (resource usage rate) in 1 year. Such as 1 minute apart, etc.
In the embodiment, the data of the page where the running task execution unit of the resident service in the computing task is located is acquired by utilizing a webpage crawler technology; analyzing the number of the running task execution units according to the data of the page; calculating and storing the resource usage amount of the resident service according to the linear relation between the amount of the running task execution units and the resource usage amount; counting the resource usage number in a preset time period; and showing the counted resource usage amount through the view. According to the embodiment of the invention, the resource monitoring of the resident service in the computing task is realized according to the linear relation between the number of the running task execution units of the resident service in the computing task and the resource use number, and the resource use state of the resident service in the computing task is displayed in a visual mode; by displaying the resource use state in the query time period, the resource use saturation of the resident service in the computing task is displayed, a reliable basis is provided for resource allocation of the resident service in the computing task, resources used by the resident service in the computing task can be allocated more reasonably, and resource waste caused by excessive allocation of resources is avoided; and through the displayed resource use state, a basis can be provided for performance optimization of the calculation resident service, and the analysis of problems of the resident service in the calculation task is facilitated, wherein problems may exist if the displayed resource use state is suddenly and greatly increased.
Fig. 11 is a schematic block diagram of a resource monitoring apparatus for a computing task according to another embodiment of the present invention. The device 110 includes a setting unit 111, a page obtaining unit 112, an analyzing unit 113, a calculating unit 114, a counting unit 115, and a presenting unit 116. This embodiment differs from the embodiment of fig. 7 in that: a setting unit 111 is added. For details of other units, please refer to descriptions of corresponding units in the embodiment of fig. 6, which are not repeated herein.
A setting unit 111 for setting the time interval. The time intervals are, for example, 1 minute, 5 minutes, etc.
The page obtaining unit 112 is further configured to obtain, according to the time interval, data of a page where the running task execution unit of the service resident in the computing task is located by using a web crawler technology.
Since a computation task generally does not have a sudden and large increase or sudden and large decrease in resource usage, the number of continuously acquired multiple resource usages may not be numerically different. If the resource usage amount is acquired every moment, on one hand, acquiring the resource usage amount needs to occupy some resources, such as CPU resources, memory resources and the like; on the other hand, the use quantity of a plurality of resources which are continuously obtained has limited reference significance due to small difference in numerical value. Therefore, the time interval is set so as to obtain the number of the running task execution units of the resident service in the calculation task according to the time interval, and the efficiency of resource monitoring is further improved. It is understood that the number of running task execution units of the resident service in the computing task is obtained according to the time interval, and then the statistical resource usage number shown in the view is also shown according to the time interval.
Fig. 12 is a schematic block diagram of a resource monitoring device for a computing task according to an embodiment of the present invention. The device 120 may be a terminal, such as a server or the like. The device 120 includes a processor 122, memory, and a network interface 123 connected by a system bus 121, where the memory may include a non-volatile storage medium 124 and an internal memory 125.
The non-volatile storage medium 124 may store an operating system 1241 and program data 1242. The program data 1242, when executed, may cause the processor 122 to perform a method for resource monitoring of computing tasks.
The processor 122 is used to provide computing and control capabilities to support the operation of the overall device 120.
The memory 125 stores program data that, when executed by the processor 122, causes the processor 122 to perform a resource monitoring method for computing tasks.
The network interface 123 is used for performing network communication, such as receiving instructions and the like. Those skilled in the art will appreciate that the configuration shown in fig. 12 is a block diagram of only a portion of the configuration associated with the present application and does not constitute a limitation on the device 120 to which the present application is applied, and that a particular device 120 may include more or less components than those shown, or combine certain components, or have a different arrangement of components.
Wherein the processor 122 is configured to execute program data stored in the memory to perform the following operations:
acquiring data of a page where a running task execution unit of a resident service in a computing task is located by utilizing a webpage crawler technology; analyzing the number of the running task execution units according to the data of the page; calculating and storing the resource usage amount of the resident service according to the linear relation between the amount of the running task execution units and the resource usage amount; counting the resource usage number in a preset time period; and showing the counted resource usage amount through the view.
In an embodiment, when the processor 122 acquires data of a page where a task execution unit of a service resident in a computing task is running by using a web crawler technology, the following operations are specifically performed:
acquiring uniform resource locator parameters of a webpage user interface in a computing task; and acquiring data of a page where the running task execution unit of the resident service in the computing task is located by utilizing a webpage crawler technology according to the uniform resource locator parameter.
In an embodiment, when the processor 122 analyzes the number of the running task execution units according to the data of the page, the following operations are specifically performed:
acquiring tags of all running task execution units of the resident service from the data of the page; and counting corresponding values in the labels of all the running task execution units as the number of the running task execution units.
In an embodiment, when the processor 122 performs statistics on the number of used resources in the preset time period, the following operations are specifically performed:
receiving a query instruction, wherein the query instruction comprises a corresponding preset time period; and counting the resource usage amount of the resident service according to the preset time period corresponding to the query instruction.
In one embodiment, before the processor 122 executes the web crawler technology to obtain data of a page where a running task execution unit of a service resident in a computing task is located, the following operations are further executed:
setting a time interval;
the method for acquiring the data of the page where the running task execution unit of the resident service in the computing task is located by utilizing the webpage crawler technology comprises the following steps: and acquiring the number of the running task execution units of the resident service in the calculation task by utilizing a webpage crawler technology according to the time interval.
It should be understood that, in the embodiment of the present invention, the Processor 122 may be a Central Processing Unit (CPU), and the Processor 122 may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable gate arrays (FPGAs) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. Wherein a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Those skilled in the art will appreciate that the service-integrated device 120 configuration shown in fig. 12 does not constitute a limitation of device 120, and may include more or fewer components than those shown, or some components in combination, or a different arrangement of components. For example, in some embodiments, the service merging device may only include a memory and a processor, and in such embodiments, the structures and functions of the memory and the processor are consistent with those of the embodiment shown in fig. 12, and are not described herein again.
Embodiments of the present invention also provide a computer-readable storage medium, which stores one or more programs, where the one or more programs are executable by one or more processors to implement the following steps:
acquiring data of a page where a running task execution unit of a resident service in a computing task is located by utilizing a webpage crawler technology; analyzing the number of the running task execution units according to the data of the page; calculating and storing the resource usage amount of the resident service according to the linear relation between the amount of the running task execution units and the resource usage amount; counting the resource usage number in a preset time period; and showing the counted resource usage amount through the view.
In an embodiment, when the program data is executed by the processor and the data of the page where the running task execution unit of the resident service in the computing task is located is acquired by using the web crawler technology, the following specific implementation is performed:
acquiring uniform resource locator parameters of a webpage user interface in a computing task; and acquiring data of a page where the running task execution unit of the resident service in the computing task is located by utilizing a webpage crawler technology according to the uniform resource locator parameter.
In an embodiment, when the processor analyzes the number of the running task execution units according to the data of the page, the following operations are specifically performed:
acquiring tags of all running task execution units of the resident service from the data of the page; and counting corresponding values in the labels of all the running task execution units as the number of the running task execution units.
In an embodiment, when the processor counts the resource usage amount in a preset time period, the following operations are specifically performed:
receiving a query instruction, wherein the query instruction comprises a corresponding preset time period; and counting the resource usage amount of the resident service according to the preset time period corresponding to the query instruction.
In one embodiment, before the program data is executed by the processor to obtain data of a page where a running task execution unit of a service resident in a computing task is located by using a web crawler technology, the following operations are further performed:
setting a time interval;
the method for acquiring the data of the page where the running task execution unit of the resident service in the computing task is located by utilizing the webpage crawler technology comprises the following steps: and acquiring the number of the running task execution units of the resident service in the calculation task by utilizing a webpage crawler technology according to the time interval.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described apparatuses, devices and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus, device and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may also be an electric, mechanical or other form of connection.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment of the present invention.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention essentially or partially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-only memory (ROM), a magnetic disk, or an optical disk.
While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (8)
1. A method for resource monitoring of a computing task, the method comprising:
acquiring data of a page where running task execution units of resident services in a computing task are located by utilizing a webpage crawler technology, wherein one task execution unit runs corresponding to one thread, and a plurality of task execution units run corresponding to a plurality of threads concurrently;
analyzing the number of the running task execution units according to the data of the page;
calculating and storing the resource usage amount of the resident service according to the linear relation between the amount of the running task execution units and the resource usage amount;
counting the resource usage number in a preset time period;
displaying the counted resource usage quantity through the view;
analyzing the number of the running task execution units according to the data of the page, wherein the analyzing comprises the following steps:
acquiring tags of all running task execution units of the resident service from the data of the page;
and counting corresponding values in the labels of all the running task execution units as the number of the running task execution units.
2. The method of claim 1, wherein the obtaining data of a page where a running task execution unit of a service resident in a computing task is located by using web crawler technology comprises:
acquiring uniform resource locator parameters of a webpage user interface in a computing task;
and acquiring data of a page where the running task execution unit of the resident service in the computing task is located by utilizing a webpage crawler technology according to the uniform resource locator parameter.
3. The method of claim 1, wherein the counting the number of resource usages within a preset time period comprises:
receiving a query instruction, wherein the query instruction comprises a preset time period;
and counting the resource usage amount of the resident service according to the preset time period corresponding to the query instruction.
4. The method of claim 1, wherein prior to obtaining data for a page in which a running task execution unit of a service resident in a computing task resides using web crawler technology, the method further comprises:
setting a time interval;
the method for acquiring the data of the page where the running task execution unit of the resident service in the computing task is located by utilizing the webpage crawler technology comprises the following steps: and acquiring data of a page where a running task execution unit of the resident service in the computing task is located by utilizing a webpage crawler technology according to the time interval.
5. An apparatus for resource monitoring of computing tasks, the apparatus comprising:
the system comprises a page acquisition unit, a page processing unit and a page processing unit, wherein the page acquisition unit is used for acquiring data of a page where a running task execution unit of a resident service in a computing task is located by utilizing a webpage crawler technology, one task execution unit runs corresponding to one thread, and a plurality of task execution units run corresponding to a plurality of threads concurrently;
the analysis unit is used for analyzing the number of the running task execution units according to the data of the page;
the computing unit is used for computing and storing the resource usage amount of the resident service according to the linear relation between the amount of the running task execution units and the resource usage amount;
the counting unit is used for counting the resource usage number in a preset time period;
the display unit is used for displaying the counted resource usage quantity through the view;
wherein the parsing unit includes:
the tag acquisition unit is used for acquiring tags of all running task execution units of the resident service from the data of the page;
and the tag value counting unit is used for counting corresponding values in tags of all the running task execution units as the number of the running task execution units.
6. The apparatus of claim 5, wherein the page obtaining unit comprises:
the parameter acquisition unit is used for acquiring uniform resource locator parameters of the webpage user interface in the calculation task;
and the page data acquisition unit is used for acquiring data of a page where the running task execution unit of the resident service is located in the calculation task by utilizing a webpage crawler technology according to the uniform resource locator parameter.
7. A resource monitoring device for computing tasks, the device comprising a memory, and a processor coupled to the memory;
the memory is used for storing program data for realizing resource monitoring of the computing task; the processor is configured to execute program data stored in the memory to perform the method of any of claims 1-4.
8. A computer-readable storage medium, storing one or more program data, the one or more program data being executable by one or more processors to implement the method of any one of claims 1-4.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710900536.2A CN108255665B (en) | 2017-09-28 | 2017-09-28 | Resource monitoring method, device and equipment for computing task and readable storage medium |
| PCT/CN2018/083016 WO2019062077A1 (en) | 2017-09-28 | 2018-04-13 | Computing task resource monitoring method, apparatus and device, and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710900536.2A CN108255665B (en) | 2017-09-28 | 2017-09-28 | Resource monitoring method, device and equipment for computing task and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108255665A CN108255665A (en) | 2018-07-06 |
| CN108255665B true CN108255665B (en) | 2020-09-01 |
Family
ID=62722306
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710900536.2A Active CN108255665B (en) | 2017-09-28 | 2017-09-28 | Resource monitoring method, device and equipment for computing task and readable storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN108255665B (en) |
| WO (1) | WO2019062077A1 (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111078537B (en) * | 2019-11-29 | 2023-09-22 | 珠海金山数字网络科技有限公司 | Evaluation method for Unity game bundle package resource division |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106919449A (en) * | 2017-03-21 | 2017-07-04 | 联想(北京)有限公司 | The dispatch control method and electronic equipment of a kind of calculating task |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102855248A (en) * | 2011-06-29 | 2013-01-02 | 中国移动通信集团广西有限公司 | Determination method, apparatus and system for user characteristic information |
| CN103559083B (en) * | 2013-10-11 | 2017-05-10 | 北京奇虎科技有限公司 | Web crawl task scheduling method and task scheduler |
| US9967351B2 (en) * | 2015-01-31 | 2018-05-08 | Splunk Inc. | Automated service discovery in I.T. environments |
-
2017
- 2017-09-28 CN CN201710900536.2A patent/CN108255665B/en active Active
-
2018
- 2018-04-13 WO PCT/CN2018/083016 patent/WO2019062077A1/en active Application Filing
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106919449A (en) * | 2017-03-21 | 2017-07-04 | 联想(北京)有限公司 | The dispatch control method and electronic equipment of a kind of calculating task |
Non-Patent Citations (1)
| Title |
|---|
| Spark Streaming应用与实战全攻略(I)(II);小小默;《CSDN博客》;20170707;第(I)部分第1-2页，第(II)部分第1-5页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2019062077A9 (en) | 2019-06-06 |
| WO2019062077A1 (en) | 2019-04-04 |
| CN108255665A (en) | 2018-07-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| AU2023201395B2 (en) | Data stream processing language for analyzing instrumented software | |
| CN111190888A (en) | Method and device for managing graph database cluster | |
| CN113505302A (en) | Method, device and system for supporting dynamic acquisition of buried point data and electronic equipment | |
| CN108768790A (en) | Distributed search cluster monitoring method and device, computing device, storage medium | |
| CN110704097A (en) | Data collection method and device, electronic equipment and storage medium | |
| CN113934913A (en) | Data capture method and device, storage medium and electronic equipment | |
| CN112486789A (en) | Log analysis system, method and device | |
| CN111274512A (en) | Page loading method, device and medium | |
| CN112711512A (en) | Prometheus monitoring method, device and equipment | |
| CN113283216A (en) | Webpage content display method, device, equipment and storage medium | |
| CN108023905B (en) | Internet of things application system and method | |
| CN108255665B (en) | Resource monitoring method, device and equipment for computing task and readable storage medium | |
| CN111258819A (en) | Data acquisition method, device and system for MySQL database backup file | |
| CN110619006A (en) | Statistical data management method, device, platform and storage medium based on Internet of things | |
| CN112052144B (en) | Information management method, device, electronic equipment and storage medium | |
| US10970055B2 (en) | Identifying software and hardware bottlenecks | |
| CN113220705A (en) | Slow query identification method and device | |
| CN111651314A (en) | Cloud-based Promethus rule file management method and device | |
| CN112905931B (en) | Page information display method and device, electronic equipment and storage medium | |
| CN115202973A (en) | Application running state determining method and device, electronic equipment and medium | |
| CN115114332A (en) | Database query method, device and equipment and readable storage medium | |
| CN109828885B (en) | RocketMQ memory monitoring method and device, electronic equipment and storage medium | |
| CN113835953A (en) | Statistical method and device of job information, computer equipment and storage medium | |
| CN113326237A (en) | Log data processing method and device, terminal device and storage medium | |
| CN112860844A (en) | Case clue processing system, method and device and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |