CN110515795B - Big data component monitoring method and device and electronic equipment - Google Patents
Big data component monitoring method and device and electronic equipment Download PDFInfo
- Publication number
- CN110515795B CN110515795B CN201910687478.9A CN201910687478A CN110515795B CN 110515795 B CN110515795 B CN 110515795B CN 201910687478 A CN201910687478 A CN 201910687478A CN 110515795 B CN110515795 B CN 110515795B
- Authority
- CN
- China
- Prior art keywords
- monitoring
- target component
- component
- monitoring scheme
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2474—Sequence data queries, e.g. querying versioned data
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Fuzzy Systems (AREA)
- Quality & Reliability (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application discloses a monitoring method of a big data assembly, which aims to solve the problem that the automation degree of a monitoring process is low due to the fact that monitoring modules are required to be loaded and deleted manually in the prior art. The method comprises the following steps: monitoring whether a target component to be monitored exists on a server; the server is used for operating the target assembly to be monitored; if the server is monitored to have a target component to be monitored, judging whether a monitoring scheme matched with the target component exists in an available monitoring scheme list, wherein the available monitoring scheme list is used for storing a large data component monitoring scheme configured in advance; and if so, acquiring the monitoring scheme matched with the target assembly and loading the monitoring scheme so as to monitor the target assembly. The application also discloses a monitoring device of the big data assembly, electronic equipment and a computer readable storage medium.
Description
Technical Field
The present application relates to the field of big data component monitoring technologies, and in particular, to a big data component monitoring method and apparatus, an electronic device, and a computer-readable storage medium.
Background
With the rapid development of big data technology, various industries generate a large amount of data every day, and in the face of operation of massive data, various big data service components are generated, such as Hadoop, elastic search, clickHouse, kafka and the like.
In the operation process of the big data assembly, in order to perform tuning and fault diagnosis on the big data assembly, generally, the operated big data assembly needs to be monitored. However, in the related big data component monitoring technology, because the monitoring scheme is usually required to be loaded and deleted manually, the automation degree of the monitoring process is low, and in addition, under the condition that the number of the big data components is large, the processing efficiency of the monitoring process can be reduced.
Disclosure of Invention
The embodiment of the application provides a monitoring method for a big data component, which is used for solving the problem that the automation degree of a monitoring process is low due to the fact that the monitoring scheme needs to be loaded and deleted manually in the prior art.
The embodiment of the application also provides a monitoring device of the big data assembly, electronic equipment and a computer readable storage medium.
The embodiment of the application adopts the following technical scheme:
a big data component monitoring method comprises the following steps:
monitoring whether a target component to be monitored exists on a server; the server is used for operating the target assembly to be monitored;
if the server is monitored to have a target component to be monitored, judging whether a monitoring scheme matched with the target component exists in an available monitoring scheme list, wherein the available monitoring scheme list is used for storing a pre-configured monitoring scheme;
and if so, acquiring the monitoring scheme matched with the target component and loading the monitoring scheme so as to monitor the target component.
The utility model provides a monitoring device of big data assembly, includes monitoring module, judgement module and acquisition module, wherein:
the monitoring module is used for monitoring whether a target component to be monitored exists on the server; the server is used for operating the target assembly to be monitored;
the judging module is used for judging whether a monitoring scheme matched with the target component exists in an available monitoring scheme list or not if the target component to be monitored exists on the server, and the available monitoring scheme list is used for storing a pre-configured monitoring scheme;
and the acquisition module is used for acquiring and loading the monitoring scheme matched with the target component to monitor the target component if the monitoring scheme matched with the target component exists in the available monitoring scheme list.
An electronic device, comprising: a memory, a processor and a computer program stored on the memory and executable on the processor, which computer program, when executed by the processor, carries out the steps of the method for monitoring a big data component as described above.
A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the big data component monitoring method as described above.
The embodiment of the application adopts at least one technical scheme which can achieve the following beneficial effects:
the method provided by the embodiment of the application comprises the steps of firstly, pre-configuring a big data assembly monitoring scheme and storing the big data assembly monitoring scheme in an available monitoring scheme list; secondly, the target assembly to be monitored is determined by monitoring the operation condition of the server, and after the target assembly to be monitored is determined, the server is triggered to judge whether a monitoring scheme matched with the target assembly exists in the available monitoring scheme list, and finally, if the monitoring scheme matched with the target assembly exists in the available monitoring scheme list, the corresponding monitoring scheme can be automatically obtained and loaded to monitor the target assembly.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
fig. 1 is a schematic implementation flow diagram of a monitoring method for a big data component according to an embodiment of the present application;
fig. 2 is a schematic flow chart of an implementation of a monitoring method for a big data component according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of a monitoring apparatus for big data components according to an embodiment of the present disclosure;
fig. 4 is a schematic structural diagram of a monitoring electronic device of a big data component according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be described in detail and completely with reference to the following specific embodiments of the present application and the accompanying drawings. It should be apparent that the described embodiments are only a few embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.
Example 1
In order to solve the problems that in the related big data component monitoring technology, the automation degree of the monitoring process is low due to the fact that manual loading and deletion of the monitoring scheme are generally required, and the processing efficiency of the monitoring process can be reduced under the condition that the number of the big data components is large, embodiment 1 of the application provides a monitoring method for the big data components. The execution subject of the method may be, but is not limited to, a server or a server cluster composed of servers.
For convenience of description, the present application takes an execution subject as an example, and introduces an embodiment of the present application. It is understood that the implementation of the method by the server is merely an exemplary illustration and should not be construed as a limitation of the method.
Referring to fig. 1, a method for monitoring a big data component according to an embodiment of the present application includes the following steps:
the server is used for operating a target assembly to be monitored; in the embodiment of the application, the server can acquire the configuration file corresponding to the client version when the target component to be monitored is on-line/running so as to determine each target component to be monitored in the client of the client version; or, configuration files corresponding to the online client versions may be acquired together to determine each target component to be monitored in the client of each online client version.
In addition, since the client updates are very frequent nowadays, it is often necessary to deploy, for each function, many versions of large data components corresponding to the function on the server, which may lead to an increasingly bulky server. As such, technicians are often required to monitor each of the running big data components on the server to perform tuning and troubleshooting of the big data components and the server. Based on this, in the embodiment of the present application, the target component to be detected may include various big data components that can run on the server, for example, hadoop, elasticSearch, clickwause, kafka, and the like.
In the embodiment of the application, before monitoring whether the target component to be monitored exists on the server, a monitoring interface can be set for the target component to be monitored in advance, so that when the target component to be monitored exists on the server, the preset monitoring interface can be called to monitor the target component.
and the available monitoring scheme list is used for storing a large data component monitoring scheme configured in advance.
The big data component monitoring scheme may include a data acquisition program to be deployed and a program start command when the big data component is monitored. In order to facilitate subsequent query, in the embodiment of the present application, the preconfigured big data component monitoring solutions may be uniformly stored to a certain preset path, for example, may be stored to an available monitoring solution list,
in an embodiment, since a new big data component monitoring scheme is generated in a subsequent operation process, the subsequent big data component monitoring scheme configured in advance may need to be laterally extended, in this application, the big data component monitoring scheme configured in advance may also be uploaded to a server or an available monitoring scheme list may be written into a configuration file, if a new big data component monitoring scheme is generated subsequently, the monitoring scheme of the newly generated big data component may be directly uploaded to the server, or a name of a service of the monitoring scheme of the newly generated big data component may be added to the configuration file, so as to implement lateral extension.
In this embodiment of the present application, if it is monitored that the target component to be monitored exists on the server in step 11, the following method may be adopted to determine whether a monitoring scheme matching the target component exists in the available monitoring scheme list:
step 121, acquiring related information of the target assembly;
wherein the related information at least comprises one of the following: the service name of the target assembly, the service address of the server corresponding to the target assembly, the host name corresponding to the target assembly and the port information corresponding to the target assembly;
and step 122, judging whether a monitoring scheme matched with the target component exists in the available monitoring scheme list or not according to the relevant information.
In step 121 of the embodiment of the present application, the relevant information of the target component is obtained, which may specifically be configuration information of each big data component in the client that determines the version of the client.
The configuration information of the big data component may specifically include information of two aspects: 1. function information corresponding to the big data component; 2. the version information of the big data component includes, for example, a service name of the big data component, a service address of a server corresponding to the big data component, a host name corresponding to the big data component, port information corresponding to the big data component, and supported serialization protocols (Json, protobuf, etc.), service logic of the code, required parameters or formats, and the like.
Optionally, in the present application, in order to improve the speed of obtaining the relevant information, only version information of the target component to be monitored may be obtained, for example, a service name of the target component to be monitored, a service address of a server corresponding to the target component to be monitored, a host name corresponding to the target component to be monitored, and port information corresponding to the target component to be monitored.
The service name of the target component to be monitored is used for determining a monitoring scheme corresponding to the target component; the host name corresponding to the target component to be monitored is used for determining the host running the target component; the service address of the server corresponding to the target component to be monitored is used for determining the service address of the server corresponding to the target component; and the port information corresponding to the target component to be monitored is used for acquiring the related information of the target component.
Specifically, when the related information of the target component is acquired, the related information may be acquired through a Java program, for example, first, a Linux command is executed through Runtime to obtain a first execution result; secondly, calling runtime, getrun (). Exec (command) to execute a first execution result, and waiting for the command execution to be completed by using a waitFor () method; the Runtime.exec method can generate a local Process and return an instance of a Process subclass; then, respectively executing operations on the input and the output of the instance of the Process subclass through a Process. Finally, performing text analysis according to the second execution result, and obtaining the service name of the target component to be monitored running on the server, the service address of the server corresponding to the target component, the host name corresponding to the target component and the port information corresponding to the target component.
In one embodiment, in order to implement the universality and extensibility of the related information, the related information may be generally subjected to normalized management, for example, the related information may be uniformly formatted and written into a metadata table of the Mysql database.
After the relevant information of the target component is obtained in step 121, whether a monitoring scheme matched with the target component exists in the available monitoring scheme list or not can be further judged according to the relevant information through the following steps:
and traversing the available monitoring scheme list, and judging whether a big data component monitoring scheme matched with the component list exists in the available monitoring scheme list.
Specifically, if a big data component monitoring scheme consistent with the service name of the target component in the component list exists in the available monitoring scheme list, it is determined that a big data component monitoring scheme matched with the component list exists in the available monitoring scheme list.
For example, after a service process is started, firstly, a configuration file is loaded to obtain a list a of a big data component monitoring scheme configured in advance; secondly, reading a metadata table in the Mysql database, acquiring that a target component to be monitored is running on a server, and writing a service name of the target component into a list B; and finally, traversing the list A, and judging whether a big data component monitoring scheme consistent with the service name of the target component in the list B exists in the list A.
And step 13, if so, acquiring a monitoring scheme matched with the target assembly and loading the monitoring scheme to monitor the target assembly.
Step 131, acquiring a starting command of a monitoring scheme corresponding to the target component;
step 132, loading the start command of the monitoring scheme to the server corresponding to the target component to monitor the target component.
Continuing with the above example, if a big data component monitoring scheme consistent with the service name of the target component in B exists in a, the monitoring scheme is obtained, and a corresponding start command is obtained, so that the start command of the monitoring scheme is loaded to the server corresponding to the target component to monitor the target component.
Optionally, in the embodiment of the present application, after the start command is started, the monitoring scheme may be written into the configuration file of the time sequence database by using the service address of the server corresponding to the target component and the port information corresponding to the target component as keywords, so as to query the monitoring scheme subsequently. The time sequence database is used for storing the collected data.
Optionally, in this embodiment of the present application, when monitoring the target component, the server may feed back, through the monitoring interface, the monitored state information of each target component to the monitoring report page. The information that the monitoring report displays includes the service name of the target component, the monitoring time, and monitoring status information, such as the operating status of the target component.
In one embodiment, the monitoring report may be stored in a time sequence database of the server, so as to be able to monitor the status of each component in real time, timely find the problem of the process going down, and even be able to choose to reinstall a certain target component.
Specifically, since the service name of the target component to be monitored is unique, if the information of the target component needs to be queried on the monitoring report page subsequently, the query can be performed only by inputting the service name of the target component in the time sequence database.
It should be noted that, before the presentation, corresponding processing needs to be performed on some data according to its own properties, for example, to present the total amount of data of the database, an accumulation is performed on the amount of data obtained on each server, for example, a sum () function, the processing here can be implemented by a built-in function of prometheus, what to do is determined according to business logic, and when the presentation page development completes the deployment, a key is imported.
By adopting the method provided by the embodiment of the application, firstly, a big data component monitoring scheme is configured in advance and stored in an available monitoring scheme list; secondly, the target assembly to be monitored is determined by monitoring the operation condition of the server, and after the target assembly to be monitored is determined, the server is triggered to judge whether a monitoring scheme matched with the target assembly exists in the available monitoring scheme list, and finally, if the monitoring scheme matched with the target assembly exists in the available monitoring scheme list, the corresponding monitoring scheme can be automatically obtained and loaded to monitor the target assembly.
In the embodiment of the present application, because the current client is updated very frequently, and often a large data component of multiple versions corresponding to each function needs to be deployed on the server for each function, for the large data component of multiple versions, there are also multiple corresponding monitoring schemes on the server, which may lead to more and more bloated servers, and further affect the service performance of the server, based on which the following method for deleting the monitoring schemes is further provided in the embodiment of the present application, so as to improve the service performance of the server:
and when monitoring of the target assembly is finished, deleting the monitoring scheme according to a preset script so as to stop monitoring of the target assembly.
Specifically, after monitoring that a target component is monitored, if a monitoring scheme of a certain target component to be monitored needs to be deleted, a delete script can be directly called, and a name of the monitoring scheme corresponding to the target component to be monitored is transferred to the delete script, wherein after the delete script receives the name of the monitoring scheme, an uninstaller can be automatically called to read relevant information of the monitoring scheme from a metadata table, so that servers on which the monitoring scheme runs can be obtained, and then the monitoring scheme running by each server is deleted according to the sequence.
By adopting the method, the problems that the automation degree of the monitoring process is lower because the monitoring scheme is generally required to be deleted in the related technology and the processing efficiency of the monitoring process can be reduced under the condition that the large number of large data components is huge can be avoided.
The following describes the monitoring method for big data components provided in the embodiment of the present application by describing embodiment 2.
Example 2
In order to solve the problems that in a related big data assembly monitoring technology, because a monitoring scheme generally needs to be loaded and deleted manually, the automation degree of a monitoring process is low, and the processing efficiency of the monitoring process can be reduced under the condition that the number of big data assemblies is huge, the embodiment of the application provides a big data assembly monitoring method.
For convenience of description, the following describes embodiments of the present application by taking an execution subject as a server as an example. As shown in fig. 2, an embodiment of the present application provides a method for monitoring a big data component, where the process includes the following steps:
in order to implement various service functions, a large data component of many versions corresponding to the function generally needs to be deployed on a server, which may cause the server to be increasingly bulkier and unable to perform tuning and fault diagnosis on the large data component. Based on this, in the present application, a big data component currently running on the server, for example, hadoop, elastic search, clickHouse, kafka, etc., may be determined first, so as to determine a target component to be monitored.
In addition, in the embodiment of the present application, it is further required to obtain relevant information of a target component to be monitored, where the relevant information at least includes one of the following information: the service name of the target assembly, the service address of the server corresponding to the target assembly, the host name corresponding to the target assembly and the port information corresponding to the target assembly.
When the relevant information of the target component to be monitored is obtained, the method for obtaining the relevant information provided in the above embodiment 1 may be adopted, and for avoiding redundancy, the description is omitted here.
After determining the big data component/target component to be monitored currently running by the server, the following step 22 may be executed to determine a monitoring scheme corresponding to the big data component/target component to be monitored, and further monitor the big data component/target component to be monitored currently running.
The monitoring scheme may include a data acquisition program to be deployed and a program start command when the big data component is monitored. In the embodiment of the application, the big data component monitoring schemes configured in advance can be uniformly stored in the available monitoring scheme list so as to be convenient for subsequent query.
And step 22, determining a monitoring scheme corresponding to the currently running big data assembly, and automatically loading the monitoring scheme to realize the monitoring of the big data assembly.
In the embodiment of the application, firstly, a configuration file can be loaded, and a list A of a large data component monitoring scheme configured in advance is obtained; secondly, reading a metadata table in the Mysql database, acquiring that a target component to be monitored is running on a server, and writing a service name of the target component into a list B; and finally, traversing the list A, and judging whether a big data component monitoring scheme consistent with the service name of the target component in the list B exists in the list A.
And if the big data component monitoring scheme consistent with the service name of the target component in the B exists in the A, acquiring and loading the monitoring scheme matched with the target component so as to monitor the target component.
It should be noted that, in order to avoid the problems that the automation degree of the monitoring process is low and the processing efficiency of the monitoring process may be reduced under the condition that the number of the large data components is large because the monitoring scheme is generally required to be deleted in the related art, in the embodiment of the present application, after the target component is monitored, the monitoring scheme is also deleted according to the preset script to stop monitoring the target component.
Since the embodiment of the present invention adopts the same application concept as that of embodiment 1, the problems in the prior art can also be solved, and details are not described herein.
Example 3
In view of the same inventive concept as the method described above, the embodiment of the present application further provides a monitoring apparatus for big data assemblies, which is used to solve the problems that in the related big data assembly monitoring technology, because the monitoring scheme generally needs to be loaded and deleted manually, the automation degree of the monitoring process is low, and the processing efficiency of the monitoring process may be reduced under the condition that the number of big data assemblies is large.
The specific structural schematic diagram of the monitoring device 30 for big data components is shown in fig. 3, and includes a monitoring module 31, a determining module 32, and an obtaining module 33, where the functions of the modules are as follows:
the monitoring module 31 is used for monitoring whether a target component to be monitored exists on the server; the server is used for operating the target assembly to be monitored;
a determining module 32, configured to determine whether a monitoring scheme matching the target component exists in an available monitoring scheme list if it is monitored that the target component to be monitored exists on the server, where the available monitoring scheme list is used to store a preconfigured big data component monitoring scheme;
an obtaining module 33, configured to, when a monitoring scheme matching the target component exists in the available monitoring scheme list, obtain and load the monitoring scheme corresponding to the target component, so as to monitor the target component.
The determining module 32 specifically includes an acquiring first unit and a first determining unit, and the functions of each unit are as follows:
an acquisition unit configured to acquire information related to the target component; wherein the related information comprises at least one of: the service name of the target assembly, the service address of a server corresponding to the target assembly, the host name corresponding to the target assembly and the port information corresponding to the target assembly;
and the judging unit is used for judging whether the monitoring scheme matched with the target component exists in the available monitoring scheme list or not according to the relevant information.
Optionally, in this embodiment of the application, the obtaining unit may be specifically configured to:
acquiring related information of the target assembly, and writing the service name of the target assembly in the related information into an assembly list; wherein the related information at least comprises one of the following: the service name of the target assembly, the service address of a server corresponding to the target assembly, the host name corresponding to the target assembly and the port information corresponding to the target assembly;
optionally, in this embodiment of the application, the determining unit may be specifically configured to:
and traversing the available monitoring scheme list, and judging whether the big data component monitoring scheme matched with the component list exists in the available monitoring scheme list.
Specifically, if the name of the big data component monitoring scheme consistent with the service name of the target component in the component list exists in the available monitoring scheme list, it is determined that the big data component monitoring scheme matched with the component list exists in the available monitoring scheme list.
The acquiring module 33 specifically includes a second acquiring unit and a monitoring unit, and the functions of each unit are as follows:
the second acquisition unit is used for acquiring a starting command of the monitoring scheme corresponding to the target component;
and the monitoring unit is used for loading the starting command of the monitoring scheme to a server corresponding to the target component so as to monitor the target component.
By adopting the device provided by the application, firstly, a big data component monitoring scheme is configured in advance and stored in an available monitoring scheme list; secondly, the monitoring module determines a target component to be monitored by monitoring the operation condition of the server, and the judging module triggers the server to judge whether a monitoring scheme matched with the target component exists in the available monitoring scheme list after determining the target component to be monitored, and finally, if the monitoring scheme matched with the target component exists in the available monitoring scheme list, the acquisition module can automatically acquire and load the corresponding monitoring scheme to monitor the target component.
Optionally, the apparatus provided in this embodiment of the present application may further include a stop monitoring module, configured to:
and when monitoring of the target assembly is finished, deleting the monitoring scheme according to a preset script so as to stop monitoring of the target assembly.
Adopt the device that this application provided, can avoid among the correlation technique because generally need rely on and delete the monitoring scheme to lead to monitoring process automation degree lower, and under the huge condition of big data assembly quantity, still can reduce the problem of the treatment effeciency of monitor process.
Example 4
An embodiment of the present application further provides an intelligent device for device control, where a hardware structure diagram of the intelligent device is shown in fig. 4, and in a hardware level, the electronic device includes a processor, and optionally further includes an internal bus, a network interface, and a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory, such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.
The processor, the network interface, and the memory may be connected to each other via an internal bus, which may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 4, but that does not indicate only one bus or one type of bus.
And the memory is used for storing programs. In particular, the program may include program code comprising computer operating instructions. The memory may include both memory and non-volatile storage and provides instructions and data to the processor.
The processor reads the corresponding computer program from the nonvolatile memory into the memory and then runs the computer program to form the data synchronization device on the logic level. The processor is used for executing the program stored in the memory and is specifically used for executing the following operations:
monitoring whether a target component to be monitored exists on a server; the server is used for operating the target assembly to be monitored;
if the server is monitored to have a target component to be monitored, judging whether a monitoring scheme matched with the target component exists in an available monitoring scheme list, wherein the available monitoring scheme list is used for storing a large data component monitoring scheme configured in advance;
and if so, acquiring the monitoring scheme matched with the target assembly and loading the monitoring scheme so as to monitor the target assembly.
In one case, the processor may further perform the following: acquiring related information of the target assembly; wherein the related information comprises at least one of: the service name of the target assembly, the service address of a server corresponding to the target assembly, the host name corresponding to the target assembly and the port information corresponding to the target assembly;
and judging whether a monitoring scheme matched with the target component exists in the available monitoring scheme list or not according to the relevant information.
Specifically, if a big data component monitoring scheme consistent with the service name of the target component in the component list exists in the available monitoring scheme list, it is determined that a big data component monitoring scheme matched with the component list exists in the available monitoring scheme list.
Optionally, the processor obtains a monitoring scheme corresponding to the target component and loads the monitoring scheme to monitor the target component, and the method specifically includes:
acquiring a starting command of a monitoring scheme corresponding to a target component;
and loading the starting command of the monitoring scheme to a server corresponding to the target component so as to monitor the target component.
The monitoring method for big data components disclosed in the embodiment of fig. 4 of the present application can be applied to or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and combines hardware thereof to complete the steps of the method.
Of course, besides the software implementation, the electronic device of the present application does not exclude other implementations, such as logic devices or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, and may also be hardware or logic devices.
In addition, fig. 4 also includes some functional modules that are not shown, and are not described herein again.
The embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements each process of the foregoing embodiment of the monitoring method for a big data component, and can achieve the same technical effect, and in order to avoid repetition, details are not repeated here. The computer-readable storage medium may be a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or apparatus that comprises the element.
The above description is only an example of the present invention and is not intended to limit the present invention. Various modifications and alterations to this invention will become apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.
Claims (6)
1. A big data component monitoring method is characterized by comprising the following steps:
monitoring whether a target component to be monitored exists on a server; the server is used for operating the target assembly to be monitored; the method comprises the steps that when a target component to be monitored is on-line/operated, a server acquires a configuration file corresponding to a client version to determine each target component to be monitored in a client of the client version, or acquires configuration files corresponding to a plurality of client versions which are on-line together to determine each target component to be monitored in a client of each on-line client version; setting a monitoring interface for a target component to be monitored in advance before monitoring whether the target component to be monitored exists on a server, so that the preset monitoring interface is called to monitor the target component when the target component to be monitored exists on the server;
if the server is monitored to have a target component to be monitored, judging whether a monitoring scheme matched with the target component exists in an available monitoring scheme list, wherein the available monitoring scheme list is used for storing a large data component monitoring scheme configured in advance;
if so, acquiring the monitoring scheme matched with the target component and loading the monitoring scheme so as to monitor the target component;
judging whether a monitoring scheme matched with the target component exists in the available monitoring scheme list specifically comprises the following steps:
acquiring related information of the target assembly; wherein the related information comprises at least one of: the service name of the target assembly, the service address of a server corresponding to the target assembly, the host name corresponding to the target assembly and the port information corresponding to the target assembly; the service name of the target component to be monitored is used for determining a monitoring scheme corresponding to the target component; the host name corresponding to the target component to be monitored is used for determining the host running the target component; the service address of the server corresponding to the target component to be monitored is used for determining the service address of the server corresponding to the target component to be operated;
judging whether a monitoring scheme matched with the target component exists in the available monitoring scheme list or not according to the related information;
acquiring the relevant information of the target assembly, specifically comprising:
acquiring relevant information of the target component, and writing the service name of the target component in the relevant information into a component list;
the determining, according to the relevant information, whether a monitoring scheme matching the target component exists in the available monitoring scheme list specifically includes:
traversing the available monitoring scheme list, and judging whether the big data component monitoring scheme matched with the component list exists in the available monitoring scheme list or not;
judging whether the big data component monitoring scheme matched with the component list exists in the available monitoring scheme list, specifically comprising:
if the big data component monitoring scheme which is consistent with the service name of the target component in the component list exists in the available monitoring scheme list, judging that the big data component monitoring scheme which is matched with the component list exists in the available monitoring scheme list.
2. The method according to claim 1, wherein the obtaining and loading of the monitoring scheme corresponding to the target component to monitor the target component specifically comprises:
acquiring a starting command of the monitoring scheme corresponding to the target assembly;
and loading the starting command of the monitoring scheme to a server corresponding to the target component so as to monitor the target component.
3. The method of claim 1, further comprising:
and when monitoring of the target assembly is finished, deleting the monitoring scheme according to a preset script so as to stop monitoring of the target assembly.
4. A big data component monitoring device, comprising:
the monitoring module is used for monitoring whether a target component to be monitored exists on the server; the server is used for operating the target assembly to be monitored; the method comprises the steps that when a target component to be monitored is on-line/operated, a server acquires a configuration file corresponding to a client version to determine each target component to be monitored in a client of the client version, or acquires configuration files corresponding to a plurality of client versions which are on-line together to determine each target component to be monitored in a client of each on-line client version; setting a monitoring interface for a target component to be monitored in advance before monitoring whether the target component to be monitored exists on a server, so that the preset monitoring interface is called to monitor the target component when the target component to be monitored exists on the server;
the system comprises a judging module, a monitoring module and a monitoring module, wherein the judging module is used for judging whether a monitoring scheme matched with a target component exists in an available monitoring scheme list if the target component to be monitored exists on the server, and the available monitoring scheme list is used for storing a big data component monitoring scheme configured in advance;
an obtaining module, configured to, when a monitoring scheme matching the target component exists in the available monitoring scheme list, obtain and load the monitoring scheme corresponding to the target component, so as to monitor the target component;
the judging module comprises an acquiring unit and a judging unit:
the acquisition unit acquires the related information of the target component; wherein the related information comprises at least one of: the service name of the target assembly, the service address of a server corresponding to the target assembly, the host name corresponding to the target assembly and the port information corresponding to the target assembly; the service name of the target component to be monitored is used for determining a monitoring scheme corresponding to the target component; the host name corresponding to the target component to be monitored is used for determining the host running the target component; the service address of the server corresponding to the target component to be monitored is used for determining the service address of the server corresponding to the target component to operate;
the judging unit judges whether a monitoring scheme matched with the target component exists in the available monitoring scheme list or not according to the relevant information;
the obtaining unit is specifically configured to:
acquiring related information of the target assembly, and writing the service name of the target assembly in the related information into an assembly list;
the judgment unit is specifically configured to:
traversing the available monitoring scheme list, and judging whether the big data component monitoring scheme matched with the component list exists in the available monitoring scheme list or not;
if the name of the big data component monitoring scheme consistent with the service name of the target component in the component list exists in the available monitoring scheme list, judging that the big data component monitoring scheme matched with the component list exists in the available monitoring scheme list.
5. An electronic device, comprising: memory, processor and computer program stored on the memory and executable on the processor, which when executed by the processor implements the steps of the method of monitoring a big data component as claimed in any one of claims 1~3.
6. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the method for monitoring a big data component as claimed in any one of claims 1~3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910687478.9A CN110515795B (en) | 2019-07-29 | 2019-07-29 | Big data component monitoring method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910687478.9A CN110515795B (en) | 2019-07-29 | 2019-07-29 | Big data component monitoring method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110515795A CN110515795A (en) | 2019-11-29 |
| CN110515795B true CN110515795B (en) | 2023-03-28 |
Family
ID=68624797
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910687478.9A Active CN110515795B (en) | 2019-07-29 | 2019-07-29 | Big data component monitoring method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110515795B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111563019B (en) * | 2020-04-29 | 2023-01-24 | 厦门市美亚柏科信息股份有限公司 | Service assembly monitoring method, system and computer storage medium |
| CN114168431B (en) * | 2022-02-10 | 2022-04-15 | 北京金堤科技有限公司 | Method and apparatus for automatically monitoring service, electronic device and storage medium |
| CN114900444A (en) * | 2022-04-25 | 2022-08-12 | 北京奇艺世纪科技有限公司 | Data monitoring method and device, electronic equipment and storage medium |
| CN115134270B (en) * | 2022-06-28 | 2023-09-08 | 北京奇艺世纪科技有限公司 | Code monitoring method, monitoring system, electronic equipment and storage medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105005527A (en) * | 2015-05-26 | 2015-10-28 | 北京中亦安图科技股份有限公司 | Server-side product monitoring method and device |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107704362A (en) * | 2017-09-25 | 2018-02-16 | 浪潮软件股份有限公司 | A kind of method and device based on Ambari monitoring big data components |
| CN108845916B (en) * | 2018-07-03 | 2022-02-22 | 中国联合网络通信集团有限公司 | Platform monitoring and alarming method, device, equipment and computer readable storage medium |
| CN109766241A (en) * | 2018-12-29 | 2019-05-17 | 中国银行股份有限公司 | System monitoring method, apparatus, computer equipment and computer readable storage medium |
-
2019
- 2019-07-29 CN CN201910687478.9A patent/CN110515795B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105005527A (en) * | 2015-05-26 | 2015-10-28 | 北京中亦安图科技股份有限公司 | Server-side product monitoring method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110515795A (en) | 2019-11-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110515795B (en) | Big data component monitoring method and device and electronic equipment | |
| CN107193750B (en) | Script recording method and device | |
| US9575871B2 (en) | System and method for dynamically debugging data in a multi-tenant database environment | |
| CN108459962B (en) | Code normalization detection method and device, terminal equipment and storage medium | |
| CN109271359B (en) | Log information processing method and device, electronic equipment and readable storage medium | |
| CN109344066B (en) | Method, system and terminal for testing browser page | |
| CN112346818A (en) | Container application deployment method and device, electronic equipment and storage medium | |
| WO2019148657A1 (en) | Method for testing associated environments, electronic device and computer readable storage medium | |
| CN110063042A (en) | A kind of response method and its terminal of database failure | |
| CN111813377A (en) | Method and device for automatically generating application program | |
| CN110647463B (en) | Method and device for restoring test breakpoint and electronic equipment | |
| CN114691445A (en) | Cluster fault processing method and device, electronic equipment and readable storage medium | |
| CN112527459A (en) | Log analysis method and device based on Kubernetes cluster | |
| CN116841653A (en) | Execution method and device of operation and maintenance job, processor and electronic equipment | |
| CN114416442A (en) | Hardware change detection method and device, electronic equipment and readable storage medium | |
| CN114064503A (en) | UI automation test method and device, electronic equipment and storage medium | |
| CN114817010A (en) | Python-based Redfish automatic testing method and device | |
| CN112667463A (en) | Application system baseline state monitoring method and device | |
| CN110688430B (en) | Method and device for obtaining data bypass and electronic equipment | |
| CN110908876B (en) | Method and device for acquiring hardware performance data | |
| CN114968753A (en) | Equipment upgrading test method, medium, electronic equipment and test system | |
| CN112558982A (en) | Code detection method and device and computer equipment | |
| CN109728929B (en) | Test method, test device and storage medium | |
| CN109783756B (en) | Website creation method, server and computer readable storage medium | |
| CN110555090B (en) | Method, system, terminal and storage medium for automatically acquiring CPU BUS Number |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 100081 No.101, 1st floor, building 14, 27 Jiancai Chengzhong Road, Haidian District, Beijing Applicant after: Beijing PERCENT Technology Group Co.,Ltd. Address before: 100081 16 / F, block a, Beichen Century Center, building 2, courtyard 8, Beichen West Road, Chaoyang District, Beijing Applicant before: BEIJING BAIFENDIAN INFORMATION SCIENCE & TECHNOLOGY Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |