CN117743350A

CN117743350A - Data acquisition method and device and nonvolatile storage medium

Info

Publication number: CN117743350A
Application number: CN202311685795.XA
Authority: CN
Inventors: 张梦; 刘小东; 朱义稳
Original assignee: Yunhe Enmo Beijing Information Technology Co ltd
Current assignee: Yunhe Enmo Beijing Information Technology Co ltd
Priority date: 2023-12-08
Filing date: 2023-12-08
Publication date: 2024-03-22

Abstract

The application discloses a data acquisition method and device and a nonvolatile storage medium. Wherein the method comprises the following steps: receiving an acquisition instruction and acquiring data in response to the acquisition instruction, wherein the acquisition instruction comprises: collecting objects and collecting frequencies; determining the type of the acquisition object, and determining a target database corresponding to the type of the acquisition object from a plurality of databases; acquiring first configuration information stored in a cache, establishing connection with a target database under the condition that the first configuration information contains second configuration information of the target database, and acquiring data from the target database under the condition that the connection with the target database is successfully established; and updating the first configuration information under the condition that the first configuration information does not contain the second configuration information of the target database. The method and the device solve the technical problem that the operation state of the database cannot be accurately reflected because the indexes of the database can only be monitored due to the fact that the monitoring indexes cannot be increased according to actual demands.

Description

Data acquisition method and device and nonvolatile storage medium

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a method and an apparatus for data acquisition, and a nonvolatile storage medium.

Background

Database monitoring is critical to maintaining database performance and availability. The potential problems of the database can be identified and solved by monitoring key indexes of the database; and visualizing the running state of the database, an administrator can ensure the efficient running of the database system, and better support is provided for the application program infrastructure of the organization. In the related art, the monitoring frame (such as promethaus) is used for monitoring the database, but the acquisition index of the monitoring frame in the prior art is not abundant enough, and the database index to be monitored of the database is increased, so that the technical problem that the monitoring index cannot be increased according to the actual service requirement, and the accurate running state of the database cannot be obtained exists.

In view of the above problems, no effective solution has been proposed at present.

Disclosure of Invention

The embodiment of the application provides a data acquisition method and device and a nonvolatile storage medium, which at least solve the technical problem that only partial database indexes can be monitored and the running state of a database can not be accurately reflected because the monitoring indexes can not be increased according to actual demands.

According to one aspect of the embodiments of the present application, there is provided a method for data acquisition, including: receiving an acquisition instruction and acquiring data in response to the acquisition instruction, wherein the acquisition instruction comprises: the system comprises an acquisition object and an acquisition frequency, wherein the acquisition object is used for indicating data to be acquired; determining the type of the acquisition object, and determining a target database corresponding to the type of the acquisition object from a plurality of databases, wherein the target database is used for storing the acquisition object; acquiring first configuration information stored in a cache, establishing connection with a target database under the condition that the first configuration information contains second configuration information of the target database, and acquiring data from the target database under the condition that the connection with the target database is successfully established; and updating the first configuration information under the condition that the first configuration information does not contain the second configuration information of the target database.

Optionally, updating the first configuration information includes: obtaining target configuration information, wherein the target configuration information is used for updating first configuration information, and the target configuration information comprises: the first database configuration information and the first monitoring index configuration information, the first database configuration information comprises: a plurality of first configuration files of a plurality of databases, each first configuration file recording at least a database identification, the database identification being used to uniquely identify one database, the first monitoring index configuration information comprising a plurality of second configuration files of a plurality of database indexes, the database indexes comprising: the connection state of each database, the number of active sessions of each database and the space utilization rate of each database, and each second configuration file at least records the name of a database index and a database index identifier, wherein the database index identifier is used for uniquely identifying one database index; comparing the target configuration information with the first configuration information to obtain a comparison result; and updating the first configuration information according to a method corresponding to the comparison result.

Optionally, comparing the target configuration information with the first configuration information includes: determining the database identifier as a first main key of the first database configuration information, wherein the first main key is a reference condition for judging whether to update the second database configuration information in the first configuration information; when comparing the first database configuration information and the second database configuration information, only comparing the database identifier contained in the first database configuration information and the database identifier contained in the second database configuration information; determining the monitoring index attribute information as a second main key for comparing the first monitoring index configuration information, wherein the second main key is a reference condition for judging whether to update the second monitoring index configuration information in the first configuration information, and the monitoring index attribute information comprises: the name of the database index and the identification of the database index; when the first monitoring index configuration information and the second monitoring index configuration information are compared, only the monitoring index attribute information contained in the first monitoring index configuration information and the monitoring index attribute information contained in the second monitoring index configuration information are compared.

Optionally, comparing the results includes: the first comparison result is used for indicating that the database identifier contained in the first database configuration information is the same as the database identifier contained in the second database configuration information, or that the attribute information of the monitoring index contained in the first monitoring index configuration information is the same as the attribute information of the monitoring index contained in the second monitoring index configuration information; the second comparison result is used for indicating that a first target database identifier exists in the first database configuration information or first target attribute information exists in the first monitoring index configuration information, the first target database identifier is a database identifier which is not contained in the second database configuration information, and the first target attribute information is monitoring index attribute information which is not contained in the second monitoring index configuration information; and a third comparison result, wherein the third comparison result is used for indicating that a second target database identifier exists in the second database configuration information, or the second monitoring index configuration information contains second target attribute information, the second target database identifier is a database identifier which is not contained in the first database configuration information, and the second target attribute information is monitoring index attribute information which is not contained in the first database configuration information.

Optionally, updating the first configuration information according to a method corresponding to the comparison result includes: under the condition that the comparison result is a first comparison result, replacing the first target configuration information with target configuration information; if the comparison result is the second comparison result, adding the configuration information of the database corresponding to the first target database identifier to the first configuration information, or adding the configuration information of the monitoring index corresponding to the first target attribute information to the first configuration information; and deleting the configuration information of the database corresponding to the second target database identifier from the first configuration information or deleting the configuration information of the monitoring index corresponding to the second target attribute information from the first configuration information under the condition that the comparison result is the third comparison result.

Optionally, in case of successful connection establishment with the target database, the method for data acquisition further comprises: maintaining a connection state with a target database in a preset time interval; and (3) destroying the connection with the target database without receiving an instruction for collecting data from the target database in a preset time interval.

Optionally, the method for data acquisition further comprises: and under the condition that the alarm information is received when the data is collected and the number of times of receiving the alarm information is larger than the preset number of times, stopping collecting the data, and deleting the configuration information of the target database from the cache, wherein the alarm information is used for indicating the connection error between the alarm information and the target database.

According to another aspect of the embodiments of the present application, there is also provided a data acquisition device, including: the receiving module is used for receiving the acquisition instruction and responding to the acquisition instruction to acquire data, wherein the acquisition instruction comprises: the system comprises an acquisition object and an acquisition frequency, wherein the acquisition object is used for indicating data to be acquired; the system comprises a determining module, a storage module and a storage module, wherein the determining module is used for determining the type of an acquisition object and determining a target database corresponding to the type of the acquisition object from a plurality of databases, wherein the target database is used for storing the acquisition object; the acquisition module is used for acquiring the first configuration information stored in the cache, establishing connection with the target database under the condition that the first configuration information contains the second configuration information of the target database, and acquiring data from the target database under the condition that the connection with the target database is successfully established; and the updating module is used for updating the first configuration information under the condition that the second configuration information of the target database is not contained in the first configuration information.

According to another aspect of the embodiments of the present application, there is further provided a nonvolatile storage medium, in which a computer program is stored, where the method for collecting data described above is performed by running the computer program on a device in which the nonvolatile storage medium is located.

According to another aspect of the embodiments of the present application, there is also provided an electronic device comprising a memory, in which a computer program is stored, and a processor, which is arranged to perform the method of data acquisition described above by means of the computer program.

In the embodiment of the application, the method comprises the steps of receiving an acquisition instruction and responding to the acquisition instruction to acquire data, wherein the acquisition instruction comprises: the system comprises an acquisition object and an acquisition frequency, wherein the acquisition object is used for indicating data to be acquired; determining the type of the acquisition object, and determining a target database corresponding to the type of the acquisition object from a plurality of databases, wherein the target database is used for storing the acquisition object; acquiring first configuration information stored in a cache, establishing connection with a target database under the condition that the first configuration information contains second configuration information of the target database, and acquiring data from the target database under the condition that the connection with the target database is successfully established; under the condition that the first configuration information does not contain the second configuration information of the target database, the first configuration information is updated, a method for collecting data and monitoring indexes in the database is provided, and data collection aiming at various databases is supported; the database connection information and the quality change information are configurable; the aim of adding monitoring indexes according to the actual use condition of the database is fulfilled, so that the technical effects of collecting various monitoring indexes of the database and accurately reflecting the running state of the database are realized; and moreover, a sound database acquisition mechanism is provided, so that the purposes that the database connection fails and is not acquired any more, and a database connection pool can be reused, and the database is prevented from being frequently connected are achieved, so that the technical effects of reducing the number of times of scrubbing and the performance pressure of the database are realized, and the technical problem that only partial database indexes can be monitored and the operation state of the database cannot be accurately reflected due to the fact that the monitoring indexes cannot be increased according to actual requirements is solved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute an undue limitation to the application. In the drawings:

FIG. 1 is a block diagram of the hardware architecture of a computer terminal for implementing a method of data acquisition according to an embodiment of the present application;

FIG. 2 is a flow chart of steps of a method of data acquisition according to an embodiment of the present application;

FIG. 3 is a schematic diagram of pseudocode for an acquisition workflow in accordance with an embodiment of the present application;

FIG. 4 is a flow chart of a custom database and monitoring metrics according to an embodiment of the present application;

FIG. 5 is a block diagram of a data acquisition device according to an embodiment of the present application;

fig. 6 is a workflow diagram of an apparatus for data acquisition according to an embodiment of the present application.

Detailed Description

In order to make the present application solution better understood by those skilled in the art, the following description will be made in detail and with reference to the accompanying drawings in the embodiments of the present application, it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, shall fall within the scope of the present application.

It should be noted that the terms "first," "second," and the like in the description and claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that embodiments of the present application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

For better understanding of the embodiments of the present application, technical terms related in the embodiments of the present application are explained below:

prometheus: an open source service monitoring system and a time sequence database.

Exporter: an assembly for collecting data collects data from a database and provides the data externally through Prometaus monitoring specifications.

Prometheus Server: the core component is responsible for realizing acquisition, storage and query of monitoring data; also a time sequence database which stores the monitoring data in a local disk and externally provides a custom function query language (PromQL) to realize the query and analysis of the data.

Cache (Cache): a technique for temporarily storing data for speeding up data access and reducing server load; in computer systems, caches may exist in multiple levels, including browser caches, operating system caches, server caches, and the like.

(Structured Query Language, SQL)/SQL statement: a language for interacting with the database; for querying, inserting, updating, and deleting data in a database, and for creating, modifying, and deleting database objects (e.g., tables, views, indexes, etc.).

In the related art, the type of a database used for monitoring a component of the database is single, and each Exporter can only acquire one database index; in addition, with frequent use of the database, the data in the database is continuously increased, the indexes to be monitored are also continuously increased, the monitoring indexes provided by the components for monitoring the database are fewer, the monitoring indexes cannot be flexibly configured, and the newly added monitoring indexes are required to be developed based on the secondary development of the source codes, so that the problem that the running state of the database cannot be accurately monitored is solved. In order to solve this problem, related solutions are provided in the embodiments of the present application, and are described in detail below.

In accordance with the embodiments of the present application, a method embodiment of data acquisition is provided, it being noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system, such as a set of computer executable instructions, and, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order other than that illustrated herein.

The method embodiments provided by the embodiments of the present application may be performed in a mobile terminal, a computer terminal, or similar computing device. Fig. 1 shows a block diagram of the hardware architecture of a computer terminal for implementing a method of data acquisition. As shown in fig. 1, the computer terminal 10 may include one or more processors 102 (shown as 102a, 102b, … …,102 n) 102 (the processor 102 may include, but is not limited to, a microprocessor MCU or a processing device such as a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, the method may further include: a display, an input/output interface (I/O interface), a Universal Serial BUS (USB) port (which may be included as one of the ports of the BUS), a network interface, a power supply, and/or a camera. It will be appreciated by those of ordinary skill in the art that the configuration shown in fig. 1 is merely illustrative and is not intended to limit the configuration of the electronic device described above. For example, the computer terminal 10 may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.

It should be noted that the one or more processors 102 and/or other data processing circuits described above may be referred to generally herein as "data processing circuits. The data processing circuit may be embodied in whole or in part in software, hardware, firmware, or any other combination. Furthermore, the data processing circuitry may be a single stand-alone processing module or incorporated, in whole or in part, into any of the other elements in the computer terminal 10. As referred to in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of the path of the variable resistor termination to interface).

The memory 104 may be used to store software programs and modules of application software, such as program instructions/data storage devices corresponding to the data acquisition method in the embodiments of the present application, and the processor 102 executes the software programs and modules stored in the memory 104, thereby performing various functional applications and data processing, that is, implementing the data acquisition method described above. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory located remotely from the processor 102, which may be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The transmission means 106 is arranged to receive or transmit data via a network. The specific examples of the network described above may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC) that can connect to other network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module for communicating with the internet wirelessly.

The display may be, for example, a touch screen type Liquid Crystal Display (LCD) that may enable a user to interact with a user interface of the computer terminal 10.

An embodiment of the present application provides a method for data collection running in the above running environment, and fig. 2 is a flowchart of steps of the method for data collection provided according to the embodiment of the present application, as shown in fig. 2, where the method includes the following steps:

step S202, receiving an acquisition instruction and responding to the acquisition instruction to acquire data, wherein the acquisition instruction comprises: the system comprises an acquisition object and an acquisition frequency, wherein the acquisition object is used for indicating data to be acquired.

The data acquisition method provided by the embodiment of the application can be applied to prometaus and other components/software/platforms for data acquisition; in step S202, when the component/software/platform applying the method for data acquisition provided in the embodiments of the present application receives an acquisition instruction for indicating to acquire data, in response to the acquisition instruction, a corresponding acquisition job is executed to acquire data according to an acquisition frequency (e.g., low frequency, medium frequency, and high frequency) defined in the acquisition instruction, where the data to be acquired in response to the acquisition instruction is also defined by the acquisition instruction; the acquisition frequency is distinguished according to the time interval of each acquisition, for example, the acquisition frequency is low frequency and represents that data is acquired every 10 seconds, the acquisition frequency is intermediate frequency and represents that data is acquired every 5 seconds, and the acquisition frequency is high frequency and represents that data is acquired every 1 second; the acquisition object comprises: data representing a user name, data representing a password, data representing a port for connecting to a database, a network address of a database, etc., which are stored in different databases.

In step S204, the type of the acquisition object is determined, and a target database corresponding to the type of the acquisition object is determined from a plurality of databases, wherein the target database is used for storing the acquisition object.

In step S204, the collection object included in the response instruction is obtained, where the collection object is used to indicate the data to be collected, determine the data type of the data to be collected (i.e. the type of the collection object), and after determining the type of the data to be collected, read the data to be collected from the (target) database storing the collection object according to the collection operation flow corresponding to the target database, fig. 3 is a schematic diagram of the pseudo code of the collection operation flow, and as shown in fig. 3, the target database is an Oracle database, and the collection operation flow of the Oracle database defines the time interval for capturing the data, the path for capturing the data and the corresponding timeout time under different operation flows when the data is collected from the Oracle database. The above-mentioned collection object may indicate only one type of data, or may indicate a plurality of types of data, and when the collection object indicates a plurality of types of data, since different types of data are stored in different databases, the determined target database is also a plurality of.

Step S206, obtaining the first configuration information stored in the cache, establishing connection with the target database under the condition that the first configuration information contains the second configuration information of the target database, and collecting data from the target database under the condition that the connection with the target database is successfully established.

After determining the (target) database storing the data to be acquired (i.e., the acquisition object) in step S204, in step S206, (first) configuration information stored in a cache of a component/software/platform for performing the method of data acquisition provided in the embodiment of the present application is acquired, if (second) configuration information of all (target) databases determined in step S204 is present in the (first) configuration information contained in the cache, a connection is established with the target database, and after successfully establishing a connection with the target database, data is acquired therefrom.

Step S208, in the case that the second configuration information of the target database is not included in the first configuration information, updating the first configuration information.

In step S208, if (second) configuration information of all (target) databases is not included in (first) configuration information stored in a cache of a component/software/platform for executing the method of data acquisition provided in the embodiment of the present application, or if (second) configuration information of one of the target databases is not included in the case where a plurality of target databases are determined in step S204, the (first) configuration information stored in the cache is updated.

The (first) configuration information in the cache is updated whenever the configuration information of one (target) database cannot be queried in the (first) configuration information in the cache, in this embodiment, the (first) configuration information in the cache is updated according to the received target configuration information, where the target configuration information is sent by another device (such as a hub collector) connected to a component/software/platform that performs the data collection method provided by the embodiment of the present application, and after determining that the configuration information of the target database is absent in the cache, a trigger condition is generated, and the other device (such as the hub collector) is triggered to send the target configuration information, where the target configuration information is all the configuration information stored in the other device (such as the hub collector) and is predefined. Fig. 4 is a flowchart of a custom database and monitoring index, as shown in fig. 4, the target configuration information includes monitoring database information (i.e., first database configuration information) and monitoring index information (i.e., first monitoring index configuration information), where the monitoring database information (i.e., first database configuration information) is a file in a data sequence format (yaml format) and is defined by a plurality of monitoring object types. The monitoring database information (i.e., first database configuration information) records global configuration information of a database (i.e., a monitored object), including: the time interval for pushing the configuration information to the promethaus and exporter, the monitored object list, the monitored object type and the local path of the monitored object configuration file, for example, the database types supported by other devices (such as hub collectors) include: the following information is recorded in the form of codes in monitoring database information (namely, first database configuration information) by various databases such as an Oracle database, a MySQL database, a PostgreSQL database, a Dameng database, a KingbaseES database, a GBase 8s database, a TiDB database, a Shenzhou general database, a MogDB database, an openGauss database, a MongoDB database and the like:

Each type of database (i.e. the monitoring object) corresponds to a (first) configuration file of a lightweight data exchange format (JSON), and connection information of the database (i.e. the monitoring object) is recorded in the (first) configuration file, which comprises: general connection information and connection information specific to a database (i.e., a monitoring object), wherein the general connection information includes: a monitoring object globally unique identifier (i.e., database identifier, db_id), a monitoring object name (i.e., database name, db_name), a monitoring object network address (Internet Protocol Address, IP) (i.e., database IP, db_ip), a monitoring object port (i.e., a connection port of a database to a component/software/platform executing the method of data collection provided by the embodiments of the present application, db_port), a monitoring object user name (i.e., a user name of a user using the database), and a monitoring object user name password (i.e., a user password corresponding to a user using the database). The connection information specific to the database (i.e., the monitoring object), for example, the connection information specific to the Oracle database includes: authentication information (sid: system Identifierauthentication) for indicating whether authentication is required when a connection is established with the Oracle database, and if System Identifierauthentication is "true", the configuration file of the Oracle database must be configured with a user name and a password; if System Identifierauthentication has a value of "false" indicating that authentication is not required, the user name and password may not be configured in the configuration file of the Oracle database. For example, when the target database is an Oracle database, the (first) configuration file of the Oracle database is Oracle.

{"db_id":"252393bc-1d07-939b-7b0a-219476b1abc7",

Database ID #: "252393bc-1d07-939b-7b0a-219476b1abc7" # "

"db_name": "business_name1", # database name: "business_name1" # "

"db_ip": "192.168.X.x", # database IP: "192.168.X.x" #)

"db_port": "1521", # database port: "1521" #

"user": "dummy_id2", # user: "dummy_id2" #)

"password": "dummy_password", # password: "dummy_password" #)

"sid":"AA",#SID＝"AA"#

"authentication": "true" # authentication: "true" # ".

The monitoring index information (i.e., the first monitoring index configuration information) is also a file in a data-based serial format (yaml format), and is formed by defining information of a plurality of index items (i.e., database indexes), wherein a target database corresponds to the plurality of index items (i.e., database indexes), such as data for indicating a connection state of the database, data for indicating the number of active sessions of the database, and data for indicating a space usage rate of the database; the index item further comprises: an index name (name), a unique index identifier of each index, an acquisition sql statement, an acquisition frequency (interval), an additional attribute (label) of the index, status information (enable) indicating whether the acquired index is enabled, and a global attribute (global_label) of the global index, wherein the global attribute (global_label) defaults to contain the additional attribute (label) of the index, the additional attribute (label) of the index is a column name returned in the acquisition sql statement, and if the column name is not contained in the information returned by the acquisition sql statement, the additional attribute (label) of the index is discarded; therefore, to avoid that no column name exists in the information returned by the gather sql statement, in this embodiment, the defined gather sql statement only supports a select format capable of specifying a specific column to be retrieved, returning the column name of the specific column to be retrieved, and does not support a select from format for retrieving all columns in the specification table, and it is defined that the information returned by the gather sql statement must be of a numeric type. In addition, in the embodiment of the present application, the acquisition frequency (interval) may be divided into a low-frequency acquisition (lr), an intermediate-frequency acquisition (mr) and a high-frequency acquisition (hr) according to different acquisition time intervals, where the acquisition time interval of the low-frequency acquisition is longest, the acquisition time interval of the intermediate-frequency acquisition is smaller than the acquisition time interval of the low-frequency acquisition and greater than the acquisition time interval of the high-frequency acquisition, and the acquisition time interval of the high-frequency acquisition is shortest. Each type of database corresponds to a (second) profile of a monitoring index, for example, the (second) profile of the monitoring index of the Oracle database is Oracle.

As shown in fig. 4, to update the (first) configuration information in the cache using the target configuration information including the (first) database configuration information and the (first) monitor index configuration information, the registration of the target configuration information at the monitor configuration registry needs to be completed first, the database (i.e., monitor object) is registered to the promethaus and the exporter, and when the database (i.e., monitor object) is registered, the global configuration (db_config. Yaml) of the local monitor object, the configuration (e.g., targets/oracle. Json) of the monitor object type, and the configuration (e.g., oracle_metric. Yaml) of the monitor index are read first, and the monitor object is registered to the promethaus and the exporter according to the push interval (push_interval). For example, when registering the monitoring object to the precursor, the monitoring object information is written into the precursor/targets directory, and stored in the local cache as a target file (i.e., a second configuration file), the contents of the target file (i.e., the second configuration file) are the database index identifier (db_id) and the database index name (db_name), where db_id and db_name are the general attributes of the global index, and represent the label that each monitoring index (i.e., the database index) default to include, e.g., the label that default to include is

[{"db_id":"252393bc-1d07-939b-7b0a-219476b1abc7",

Database index #: "252393bc-1d07-939b-7b0a-219476b1abc7"

"db_name": "business_name1" # database name: "db_name" } ", the (second) configuration file of each database index has recorded therein the database index identifier" 252393bc-1d07-939b-7b0a-219476b1abc7 "and the database index name" business_name1". When registering the monitoring object to the exporter, accessing a monitoring index registration interface of the exporter, and pushing the (first) monitoring index configuration information to the exporter in a hypertext transfer protocol submitting (Hypertext Transfer Protocol POST, HTTP POST) request mode. After the target configuration information containing the information is obtained, comparing the (first) configuration information in the cache with the received target configuration information, judging whether to update the cache according to a comparison result, and updating the (first) configuration information in the cache according to a method corresponding to the comparison result.

According to an alternative embodiment of the present application, comparing the target configuration information with the first configuration information includes: determining the database identifier as a first main key of the first database configuration information, wherein the first main key is a reference condition for judging whether to update the second database configuration information in the first configuration information; when comparing the first database configuration information and the second database configuration information, only comparing the database identifier contained in the first database configuration information and the database identifier contained in the second database configuration information; determining the monitoring index attribute information as a second main key for comparing the first monitoring index configuration information, wherein the second main key is a reference condition for judging whether to update the second monitoring index configuration information in the first configuration information, and the monitoring index attribute information comprises: the name of the database index and the identification of the database index; when the first monitoring index configuration information and the second monitoring index configuration information are compared, only the monitoring index attribute information contained in the first monitoring index configuration information and the monitoring index attribute information contained in the second monitoring index configuration information are compared.

In this embodiment, when updating the (first) configuration information in the cache with the received target configuration information, it is necessary to compare the (first) database configuration information contained in the target configuration information with the (second) database configuration information contained in the cache, respectively, and compare the (first) monitor indicator configuration information contained in the target configuration information with the (second) monitor indicator configuration information contained in the cache, and determine whether there is configuration information not in the cache in the target configuration information according to the comparison result. Wherein when comparing the (first) database configuration information of the target configuration information with the (second) database configuration information contained in the cache, the database identification (dbId) is taken as the (first) primary key for comparison, i.e. when comparing the (first) database configuration information in the target configuration information with the (second) database configuration information in the cache, only the database identification in the (first) database configuration information and the database identification in the (second) database configuration information are compared. And when comparing the (first) monitoring index configuration information in the target information and the (second) monitoring index configuration information in the cache, the name (db_name) of the database index and the database index identification (db_id) in the monitoring index attribute information are used as (second) primary keys at the time of comparison, namely when the (first) monitoring index configuration information in the target configuration information and the (second) monitoring index configuration information in the cache are compared, only the name (db_name) of the database index in the (first) monitoring index configuration information and the name (db_id) of the database index and the database index identification (db_id) in the (second) monitoring index configuration information are compared.

In this embodiment, the obtained target configuration information is compared with the (first) configuration information in the cache, so that the following three comparison results can be obtained: indicating the same (first) comparison result of the (first/second) primary key, indicating the (second) comparison result of the (first/second) primary key existing in the target information but not in the cache, and when the comparison result is the (first) comparison result, indicating that the database identifier (dbId) or the monitoring index attribute information (i.e. the name (db_name) and the database index identifier (db_id) of the database index in the target configuration information) can be queried in the cache, i.e. the database in the target configuration information coincides with the database in the cache. When the comparison result is the (second) comparison result, the first target database identifier (dbId) which is not contained in the (first) configuration information exists in the target configuration information, or the monitoring index attribute information (i.e., the first target attribute information) which is not contained in the (first) configuration information exists in the target configuration information. The comparison result further includes a (third) comparison result indicating that the (first/second) primary key is present in the cache but not present in the target information, and when the comparison result is the (third) comparison result, the second target database configuration identifier (dbId) indicating that the target configuration information is not present in the cache, or the monitoring index attribute information (i.e., the second target attribute information) not present in the target configuration information is present in the cache.

In this embodiment, the method for updating the cache is determined according to the comparison result of the target configuration information and the (first) configuration information in the cache, where when the comparison result is the (first) comparison result, the configuration information is not required to be newly added, only the existing configuration information in the cache is updated, and the configuration information in the cache is updated by replacing the (first) configuration information with the target configuration information. When the comparison result is the first) comparison result, related information of a new database needs to be added in the cache, and the configuration message in the cache is updated by adding the first target database identifier or the first target attribute information which exist in the target information but do not exist in the cache into the cache. When the comparison result is the (third) comparison result, the configuration information in the cache is updated by deleting the configuration information of the database indicated by the second target database configuration identifier (dbId) or the second target attribute information, wherein the configuration information indicates that the second target database configuration identifier (dbId) or the database corresponding to the second target attribute information has not been used for a period of time, for example, has not been used for a preset period of time.

According to some optional embodiments of the present application, in case of successful connection establishment with the target database, the method for data acquisition further comprises: maintaining a connection state with a target database in a preset time interval; and (3) destroying the connection with the target database without receiving an instruction for collecting data from the target database in a preset time interval.

In some embodiments, a database acquisition mechanism is further provided, after connection is successfully established with the target database, a connection state with the target database is maintained, a unique identifier of the target database is generated according to configuration of the target database, and the unique identifier of the target database is stored in a cache; if the acquisition instruction is received again in a preset connection period (namely a preset time interval), and the acquisition object indicated by the acquisition instruction is stored in the target database, directly returning to the connection channel established with the target database before, and acquiring data from the target database through the connection channel, so that multiplexing of database connection is realized. If the acquisition instruction for acquiring the data from the target database is not received in a period (i.e. a preset time interval) after the connection is successfully established with the target database, i.e. the connection channel between the target database is not called in a period (i.e. a preset time interval) after the connection is successfully established with the target database, the connection channel between the target database and the target database is destroyed, and resource consumption caused by frequent connection of the database is prevented.

According to further alternative embodiments of the present application, the method of data acquisition further comprises: and under the condition that the alarm information is received when the data is collected and the number of times of receiving the alarm information is larger than the preset number of times, stopping collecting the data, and deleting the configuration information of the target database from the cache, wherein the alarm information is used for indicating the connection error between the alarm information and the target database.

In other embodiments the database acquisition mechanism further comprises: if the alarm information of the database connection error is received and the number of times of receiving the alarm information exceeds the preset number of times when the data is collected, the database is not connected continuously, the data is not collected from the database, and the performance pressure of the database is prevented from being increased due to frequent access to the database. For example, when index data of the target database is collected from the target database, the target database connection error is continuously reported 5 times, and the preset times are 3 times, at this time, the connection channel between the target database and the target database is destroyed/deleted, data is not collected from the connection channel, and frequent access to the target database is prevented.

By the steps, the configuration information in the cache can be updated through the database automatic updating mechanism, the collectable monitoring indexes are increased, and the running state of the database can be monitored more comprehensively due to the increase of the monitoring indexes; meanwhile, a sound database mechanism is provided, data are not collected any more when the database connection fails, a database connection pool can be reused in a short time, and the connection is destroyed when the set multiplexing time is exceeded, so that the aim of preventing the database from being frequently connected is fulfilled, and the operation pressure of the database is reduced.

Fig. 5 is a block diagram of a data acquisition device according to an embodiment of the present application, and as shown in fig. 5, the data acquisition device includes: the receiving module 50 is configured to receive an acquisition instruction and acquire data in response to the acquisition instruction, where the acquisition instruction includes: the system comprises an acquisition object and an acquisition frequency, wherein the acquisition object is used for indicating data to be acquired; a determining module 52, configured to determine a type of the acquisition object, and determine a target database corresponding to the type of the acquisition object from a plurality of databases, where the target database is used to store the acquisition object; the acquisition module 54 is configured to acquire first configuration information stored in the cache, establish a connection with the target database if the first configuration information includes second configuration information of the target database, and acquire data from the target database if the connection with the target database is successfully established; and an updating module 56, configured to update the first configuration information if the first configuration information does not include the second configuration information of the target database.

FIG. 6 is a workflow diagram of a data acquisition device, as shown in FIG. 6, the data acquisition device receives an acquisition instruction through a receiving module, wherein the acquisition instruction defines an acquisition frequency, an acquisition operation flow corresponding to different acquisition frequencies, and an acquisition object; the acquisition object indicates data to be acquired, and the acquisition frequency is divided into a low frequency, an intermediate frequency and a high frequency according to the time interval of the acquired data. After receiving the acquisition instruction, determining, by the determining module 52, a target database storing acquisition objects, wherein different types of acquisition objects are stored in different databases; for example, oracle data is stored in an Oracle database, mySQL data is stored in a MySQL database, and DM data is stored in a Dream (DM) database. After determining the target database for collecting data, querying (first) configuration information in the cache through the collecting module 54, wherein the first configuration information stores database connection information (i.e. second database configuration information) and index collection information (i.e. second monitoring index configuration information); the database connection information includes: a name of the database as a monitoring object (i.e., a monitoring object name), a user who uses/logs in the database as a monitoring object (i.e., a monitoring object user), a password of the user who uses/logs in the database as a monitoring object (i.e., a monitoring object password), a port at which the database as a monitoring object transmits data (i.e., a monitoring object port), and a network address of the database as a monitoring object (i.e., a monitoring object IP); the index acquisition information comprises: the name of the monitoring index, the additional attribute label of the monitoring index, the acquisition sql statement of the monitoring index, the acquisition frequency corresponding to the monitoring index and the like. If the (first) configuration information in the cache contains the (second) configuration information of the target database, calling an adapter of the target database to establish connection with the target database, and collecting data from the database; for example, the target database includes: and respectively establishing connection with the Oracle database, the MySQL database and the dream database, and collecting data from the Oracle database, the MySQL database and the dream database after successful connection is determined. In addition, if the connection is successfully established with the target database, the established connection relation/connection channel is stored in the database connection pool cache as a connection pool of the target database, and the unique database Chi Biaoshi of the connection relation/connection channel in the database connection pool cache is marked for the connection relation/connection channel; when data is acquired from the target database again in a preset time interval, connection is not required to be repeatedly established, and the connection relation/connection channel established before is directly invoked. If data is not collected from the target database any more in the preset time interval, deleting the connection relation/connection channel of the target database from the database Chi Huancun, and achieving the technical effect of cleaning the invalid database connection pool regularly. If there is no (second) configuration information of the target database in the cache, the update module 56 updates the (first) configuration information in the cache with the target configuration information, where the target configuration information is transmitted by other devices (such as hub) collectors connected to the data acquisition device, and as shown in fig. 6, the target configuration information transmitted by the hub collectors is composed of a database connection configuration and an index item acquisition configuration, and before the (first) configuration information in the cache is updated with the target configuration information, the configuration information transmitted by the hub collectors is first parsed by a configuration rule parser into the same format as the configuration information stored in the cache, and then stored in the cache. As shown in fig. 6, if the acquisition module 54 successfully acquires data from the target database, and further needs to parse the acquired data into a format defined by the data acquisition device by using a result parser, for example, using promethaus to acquire the data, the first two fields of the data in the format meeting the requirements of promethaus acquisition are in turn a target_id (unique identifier of the acquisition object/data) and a target_name (unique identifier of the acquisition object/data), and the last two fields are in turn a user (user name) and a value (value).

It should be noted that, the preferred implementation manner of the embodiment shown in fig. 5 may refer to the related description of the embodiment shown in fig. 2, which is not repeated herein.

The embodiment of the application also provides a nonvolatile storage medium, wherein the nonvolatile storage medium stores a computer program, and the device in which the nonvolatile storage medium is arranged executes the data acquisition method through running the computer program.

The above-described nonvolatile storage medium is used to store a program that performs the following functions: receiving an acquisition instruction and acquiring data in response to the acquisition instruction, wherein the acquisition instruction comprises: the system comprises an acquisition object and an acquisition frequency, wherein the acquisition object is used for indicating data to be acquired; determining the type of the acquisition object, and determining a target database corresponding to the type of the acquisition object from a plurality of databases, wherein the target database is used for storing the acquisition object; acquiring first configuration information stored in a cache, establishing connection with a target database under the condition that the first configuration information contains second configuration information of the target database, and acquiring data from the target database under the condition that the connection with the target database is successfully established; and updating the first configuration information under the condition that the first configuration information does not contain the second configuration information of the target database.

In another aspect of the embodiments of the present application, there is also provided an electronic device comprising a memory in which a computer program is stored, and a processor arranged to perform the above method of data acquisition by the computer program.

The processor in the electronic device is configured to execute a program that performs the following functions: receiving an acquisition instruction and acquiring data in response to the acquisition instruction, wherein the acquisition instruction comprises: the system comprises an acquisition object and an acquisition frequency, wherein the acquisition object is used for indicating data to be acquired; determining the type of the acquisition object, and determining a target database corresponding to the type of the acquisition object from a plurality of databases, wherein the target database is used for storing the acquisition object; acquiring first configuration information stored in a cache, establishing connection with a target database under the condition that the first configuration information contains second configuration information of the target database, and acquiring data from the target database under the condition that the connection with the target database is successfully established; and updating the first configuration information under the condition that the first configuration information does not contain the second configuration information of the target database.

It should be noted that, each module in the data acquisition device may be a program module (for example, a set of program instructions for implementing a specific function), or may be a hardware module, and for the latter, it may be represented by the following form, but is not limited thereto: the expression forms of the modules are all a processor, or the functions of the modules are realized by one processor.

The foregoing embodiment numbers of the present application are merely for describing, and do not represent advantages or disadvantages of the embodiments.

In the foregoing embodiments of the present application, the descriptions of the embodiments are emphasized, and for a portion of this disclosure that is not described in detail in this embodiment, reference is made to the related descriptions of other embodiments.

In the several embodiments provided in the present application, it should be understood that the disclosed technology content may be implemented in other manners. The above-described embodiments of the apparatus are merely exemplary, and the division of the units, for example, may be a logic function division, and may be implemented in another manner, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interfaces, units or modules, or may be in electrical or other forms.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be essentially or a part contributing to the related art or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing program codes.

The foregoing is merely a preferred embodiment of the present application and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present application and are intended to be comprehended within the scope of the present application.

Claims

1. A method of data acquisition, comprising:

receiving an acquisition instruction and acquiring data in response to the acquisition instruction, wherein the acquisition instruction comprises: the method comprises the steps of collecting objects and collecting frequencies, wherein the collecting objects are used for indicating data to be collected;

determining the type of the acquisition object, and determining a target database corresponding to the type of the acquisition object from a plurality of databases, wherein the target database is used for storing the acquisition object;

acquiring first configuration information stored in a cache, establishing connection with the target database under the condition that the first configuration information contains second configuration information of the target database, and acquiring data from the target database under the condition that the connection with the target database is successfully established;

and updating the first configuration information under the condition that the first configuration information does not contain the second configuration information of the target database.

2. The method of claim 1, wherein updating the first configuration information comprises:

obtaining target configuration information, wherein the target configuration information is used for updating the first configuration information, and the target configuration information comprises: first database configuration information and first monitoring index configuration information, the first database configuration information includes: a plurality of first configuration files of a plurality of databases, each of the first configuration files recording at least a database identification, the database identification being used to uniquely identify one database, the first monitoring index configuration information comprising a plurality of second configuration files of a plurality of database indexes, the database indexes comprising: the connection state of each database, the number of active sessions of each database and the space utilization rate of each database, and each second configuration file records at least the name of a database index and a database index identifier, wherein the database index identifier is used for uniquely identifying one database index;

comparing the target configuration information with the first configuration information to obtain a comparison result;

and updating the first configuration information according to a method corresponding to the comparison result.

3. The method of claim 2, wherein comparing the target configuration information with the first configuration information comprises:

determining the database identifier as a first main key of the first database configuration information, wherein the first main key is a reference condition for judging whether to update the second database configuration information in the first configuration information;

when comparing the first database configuration information and the second database configuration information, only comparing the database identifier contained in the first database configuration information and the database identifier contained in the second database configuration information;

determining the monitoring index attribute information as a second main key for comparing the first monitoring index configuration information, wherein the second main key is a reference condition for judging whether to update the second monitoring index configuration information in the first configuration information, and the monitoring index attribute information comprises: the name of the database index and the database index identification;

and when comparing the first monitoring index configuration information with the second monitoring index configuration information, only comparing the monitoring index attribute information contained in the first monitoring index configuration information with the monitoring index attribute information contained in the second monitoring index configuration information.

4. A method according to claim 3, wherein the comparison comprises:

a first comparison result, where the first comparison result is used to indicate that a database identifier included in the first database configuration information is the same as a database identifier included in the second database configuration information, or that attribute information of a monitoring index included in the first monitoring index configuration information is the same as attribute information of a monitoring index included in the second monitoring index configuration information;

a second comparison result, where the second comparison result is used to indicate that a first target database identifier exists in the first database configuration information, or that first target attribute information exists in the first monitoring index configuration information, the first target database identifier is a database identifier that is not included in the second database configuration information, and the first target attribute information is monitoring index attribute information that is not included in the second monitoring index configuration information; the method comprises the steps of,

and a third comparison result, where the third comparison result is used to indicate that a second target database identifier exists in the second database configuration information, or the second monitoring index configuration information includes second target attribute information, the second target database identifier is a database identifier that is not included in the first database configuration information, and the second target attribute information is monitoring index attribute information that is not included in the first database configuration information.

5. The method of claim 4, wherein updating the first configuration information according to the method corresponding to the comparison result comprises:

under the condition that the comparison result is the first comparison result, replacing the first target configuration information with the target configuration information;

if the comparison result is the second comparison result, adding configuration information of a database corresponding to the first target database identifier to the first configuration information, or adding configuration information of a monitoring index corresponding to the first target attribute information to the first configuration information;

and deleting the configuration information of the database corresponding to the second target database identifier from the first configuration information or deleting the configuration information of the monitoring index corresponding to the second target attribute information from the first configuration information under the condition that the comparison result is the third comparison result.

6. The method of claim 1, wherein in the event of successful connection establishment with the target database, the method further comprises:

maintaining a connection state with the target database in a preset time interval;

And destroying the connection with the target database without receiving an instruction for collecting data from the target database in the preset time interval.

7. The method according to claim 1, wherein the method further comprises:

and when the data is collected, alarm information is received, and the number of times of receiving the alarm information is larger than the preset number of times, stopping collecting the data, and deleting the configuration information of the target database from the cache, wherein the alarm information is used for indicating a connection error between the alarm information and the target database.

8. A data acquisition device, comprising:

the receiving module is used for receiving an acquisition instruction and responding to the acquisition instruction to acquire data, wherein the acquisition instruction comprises: the method comprises the steps of collecting objects and collecting frequencies, wherein the collecting objects are used for indicating data to be collected;

the determining module is used for determining the type of the acquisition object and determining a target database corresponding to the type of the acquisition object from a plurality of databases, wherein the target database is used for storing the acquisition object;

the acquisition module is used for acquiring first configuration information stored in a cache, establishing connection with the target database under the condition that the first configuration information contains second configuration information of the target database, and acquiring data from the target database under the condition that the connection with the target database is successfully established;

And the updating module is used for updating the first configuration information under the condition that the second configuration information of the target database is not contained in the first configuration information.

9. A non-volatile storage medium, wherein a computer program is stored in the non-volatile storage medium, and wherein a method of data acquisition according to any one of claims 1 to 7 is performed by running the computer program on a device in which the non-volatile storage medium is located.

10. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to perform the method of data acquisition according to any of claims 1 to 7 by means of the computer program.