CN112597130A - Data early warning method and device, electronic equipment and storage medium - Google Patents
Data early warning method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112597130A CN112597130A CN202011529721.3A CN202011529721A CN112597130A CN 112597130 A CN112597130 A CN 112597130A CN 202011529721 A CN202011529721 A CN 202011529721A CN 112597130 A CN112597130 A CN 112597130A
- Authority
- CN
- China
- Prior art keywords
- data
- monitoring
- calling
- api
- monitored
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000012544 monitoring process Methods 0.000 claims abstract description 174
- 238000012545 processing Methods 0.000 claims abstract description 65
- 238000004590 computer program Methods 0.000 claims description 6
- 230000003287 optical effect Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 239000008186 active pharmaceutical agent Substances 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 238000007405 data analysis Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention discloses a data early warning method, a data early warning device, electronic equipment and a storage medium. The data early warning method comprises the following steps: acquiring API call data to be monitored of a data source management platform; determining calling monitoring data of API calling data to be monitored; and under the condition that the calling monitoring data meet the data early warning condition, carrying out alarm processing on the API calling data. The technical scheme of the embodiment of the invention can realize real-time monitoring and early warning on the API calling data.
Description
Technical Field
The embodiment of the invention relates to the technical field of data processing, in particular to a data early warning method and device, electronic equipment and a storage medium.
Background
With the development of internet technology, big data high concurrency is more and more common. The monitoring and early warning of external interface call data in a big data high concurrency scene becomes a technical key point in the technical field of the internet.
In the prior art, a database locking mode is mainly adopted to solve the problem of high concurrency of big data. However, frequent locking processing is caused by the database locking processing mode, so that the data processing time cannot meet the processing time limit requirement under the condition of high concurrency of big data, and real-time early warning cannot be performed on the condition of high concurrency of the big data.
Disclosure of Invention
The embodiment of the invention provides a data early warning method, a data early warning device, electronic equipment and a storage medium, and achieves the effect of monitoring and early warning API call data in real time.
In a first aspect, an embodiment of the present invention provides a data early warning method, including:
acquiring calling data of an Application Program Interface (API) to be monitored of a data source management platform;
determining calling monitoring data of API calling data to be monitored;
and under the condition that the calling monitoring data meet the data early warning condition, carrying out alarm processing on the API calling data.
In a second aspect, an embodiment of the present invention further provides a data early warning apparatus, including:
the data acquisition module is used for acquiring API call data to be monitored of the data source management platform;
the calling monitoring data determining module is used for determining calling monitoring data of the API calling data to be monitored;
and the data early warning module is used for carrying out alarm processing on the API calling data under the condition that the calling monitoring data meets the data early warning condition.
In a third aspect, an embodiment of the present invention further provides an electronic device, where the electronic device includes:
one or more processors;
storage means for storing one or more programs;
when the one or more programs are executed by the one or more processors, the one or more processors implement the data pre-warning method provided by any embodiment of the invention.
In a fourth aspect, an embodiment of the present invention further provides a computer storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data early warning method provided in any embodiment of the present invention.
According to the technical scheme, the API call data to be monitored are analyzed from the data source management platform, data processing is further carried out on the API call data to be monitored to obtain the call monitoring data, and finally alarm processing is carried out on the API call data corresponding to the call monitoring data meeting the data early warning condition.
Drawings
Fig. 1 is a flowchart of a data early warning method according to an embodiment of the present invention;
fig. 2 is a flowchart of a data early warning method according to a second embodiment of the present invention;
fig. 3 is a schematic diagram of a data early warning apparatus according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention.
It should be further noted that, for the convenience of description, only some but not all of the relevant aspects of the present invention are shown in the drawings. Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the operations (or steps) as a sequential process, many of the operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations may be re-arranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.
Example one
Fig. 1 is a flowchart of a data early warning method according to an embodiment of the present invention, where the embodiment is applicable to a situation of performing real-time early warning on an API call in a big data high concurrency scenario, and the method may be executed by a data early warning apparatus, and the apparatus may be implemented by software and/or hardware, and may be generally integrated in an electronic device. Accordingly, as shown in fig. 1, the method comprises the following operations:
and S110, obtaining API call data to be monitored of the data source management platform.
The data source management platform may be a platform for managing data sources. The data source may be a plurality of databases that meet business needs. The API call data to be monitored can be all or part of the API call data which needs to be monitored by the data source management platform. The API call data may be data that an external application calls from a data source through an external API. The embodiment of the invention does not limit the API call data and the specific data content of the API call data to be monitored.
At present, each internet application or internet internal program needs to access a large number of APIs, however, development departments or groups of internet enterprises are independent of each other, external APIs accessed by different project groups are different, or different departments access the same external APIs, which causes confusion and resource waste in external API management inside a company, and increases the operation cost of the company. Therefore, a uniform data source management platform is established, and a large amount of calls are subjected to convergence management. However, a large number of API calls are gathered together, and a large-data high-concurrency scene can be formed, so that how to perform real-time early warning on the API with abnormal calls in the data source management platform becomes a problem which needs to be solved urgently.
In the embodiment of the invention, before data early warning is carried out on the API of the data source management platform, each API calling data of the data source management platform can be analyzed, all or part of the analyzed API calling data is used as the API calling data to be monitored, so that real-time early warning is carried out on the API calling condition by using the API calling data to be monitored.
For example, when an external application or a user needs to call an external data resource by using the data source management platform, the external application may send a data call request to the unified interface of the data source management platform. The data source management platform calls the API with the request data according to the data call request, monitors the uniform interface and the called API, and can use the call data of all or part of the interfaces of the analyzed data source management platform as the call data of the API to be monitored. Wherein the external application may be an application capable of making data calls from the data source management platform. The external data resources may be different databases connected by the data source management platform through various APIs. The unified interface can be a unified interface provided by the data source management platform for an external application or a user. The data call request may be a request sent by an external application or user to the data source management platform. According to the data call request, the data source management platform can feed back the data to be called to the sending end of the data call request.
In an optional embodiment of the present invention, acquiring API call data to be monitored of the data source management platform may include: and acquiring mirror copy data of the original API call data from each external API of the data source management platform by a data mirror copy method, wherein the mirror copy data is used as the API call data to be monitored.
The data mirror copy method may be a copy method that retains two or more online data. The raw API call data may be data generated by an external API during the call. The external API may be an interface for the data source management platform to interface with external data resources. The mirror image copy data may be data obtained by mirror image copying API call data of each external API of the data source management platform by using a data mirror image copy method.
Specifically, before API call data to be monitored of the data source management platform are obtained, original API call data are analyzed from each external API, a data mirror image copying method is adopted to copy the original API call data to obtain mirror image copy data, and the mirror image copy data of the original API call data are further used as the API call data to be monitored. It should be noted that the acquisition of the mirror copy data does not affect the main flow of the data call, and therefore, the adoption of the data mirror copy method does not increase the time consumption of the main flow of the data call. The data call main flow can be a flow for a user or an application to make a data call.
In an optional embodiment of the present invention, after obtaining the mirror copy data of the original API call data from each external API of the data source management platform by the data mirror copy method, the method may further include: and sending the API call data to be monitored to a message queue.
Wherein the message queue may be a container that stores data.
In the embodiment of the invention, after the mirror image copy data of the original API call data is obtained from each external API of the data source management platform, the mirror image copy data of the original API call data is sent to the message queue, and the data in the message queue is further subjected to data processing to complete the alarm processing of the API call data.
And S120, determining calling monitoring data of the API calling data to be monitored.
The calling monitoring data can be data obtained by performing data analysis or monitoring processing on the API calling data to be monitored.
Specifically, after the API call data to be monitored is obtained, the API call data to be monitored is subjected to data processing, the API call data to be monitored subjected to data processing is further taken as call monitoring data, and finally, alarm processing is performed according to the call monitoring data.
And S130, under the condition that the calling monitoring data meet the data early warning condition, carrying out alarm processing on the API calling data.
The data early warning condition may be a condition for triggering an alarm process. The alarm processing may be an operation of calling an API corresponding to the monitoring data that satisfies the data early warning condition. For example, the alarm processing may include interrupting external data resource service, switching routes, and sending information related to a failure API to the data source management platform. The embodiment of the invention does not limit the specific operation of alarm processing.
In the embodiment of the invention, once the calling monitoring data meets the data early warning condition, the real-time warning processing is immediately carried out on each API calling data meeting the data early warning condition. And if the calling monitoring data do not meet the data early warning condition, normally monitoring each API in real time, and not carrying out alarm processing on the API calling data.
In the embodiment of the invention, the data source management platform can configure the early warning condition or the API calling authority for each external API, and each link called by the interface of the data source management platform can be controlled through the early warning condition and the calling authority. In addition, if the calling monitoring data meets the early warning condition, the data source management platform can sense the external data resource abnormality and the user abnormal operation, and the data source management platform can issue an operation and maintenance alarm notification so that operation and maintenance personnel can timely perform fault processing or limit the access authority of the abnormal operation user. If the external data resources have problems, the data source management platform can be switched to the similar data sources, and operation and maintenance personnel can perform data cleaning, user permission releasing or broadband increasing and the like on the external data resources with problems, so that the usability of the external data resources is enhanced.
According to the technical scheme, API call data to be monitored are analyzed from the data source management platform, data processing is further carried out on the API call data to be monitored to obtain call monitoring data, and finally alarm processing is carried out on the API call data corresponding to the call monitoring data meeting the data early warning condition.
Example two
Fig. 2 is a flowchart of a data early warning method provided in the second embodiment of the present invention, which is embodied on the basis of the above embodiment, and in this embodiment, a specific optional implementation is given to determine call monitoring data of API call data to be monitored according to a data monitoring dimension, and accordingly, as shown in fig. 2, the method includes the following operations:
s210, API call data to be monitored of the data source management platform are obtained.
And S220, determining data monitoring dimensions.
The data monitoring dimension may be a dimension for monitoring the API call data to be monitored. Different types of API call data to be monitored can correspond to the same or different data monitoring dimensions.
Specifically, when the data source management platform monitors the API call condition in real time, the data monitoring dimension can be determined according to the early warning requirement, and the API call data to be monitored is further analyzed according to the data monitoring dimension.
And S230, determining calling monitoring data of the API calling data to be monitored according to the data monitoring dimension.
Specifically, after the data monitoring dimension is determined according to the early warning requirement, the data dimension called by the API to be monitored is determined according to the data monitoring dimension, and the monitoring data is further determined and called according to the data dimension called by the API to be monitored.
For example, assuming that the data monitoring dimension is a call volume of a certain API, the data calling dimension of the API to be monitored is the number of times that the API is called by each user or external application, the number of times that the API is called by each user or external application is summed, and the summed result is used as call monitoring data. Or, the calling monitoring data that a certain user or a certain external application calls a certain API to be monitored calling data may be obtained by only using the user or the external application as a data monitoring dimension. Or, the monitoring period can also be determined, and the calling monitoring data of a certain API calling data to be monitored in a certain period range is determined.
In an optional embodiment of the invention, the data monitoring dimension may comprise an interface monitoring dimension and/or a user monitoring dimension; determining calling monitoring data of the API calling data to be monitored according to the data monitoring dimension may include: determining user calling monitoring data of a target calling user according to the user monitoring dimension; and/or determining interface calling monitoring data of the target calling interface according to the interface monitoring dimension.
Wherein, the interface monitoring dimension may be a monitoring for part or all of the interface call dimension. The user monitoring dimension may be a monitoring for some or all of the user dimensions. The target invoking user may be a user who needs to be monitored. The user call monitoring data may be data associated with a target call user called interface. For example, user call monitoring data may include, but is not limited to, monitoring data for a certain user call API type, a certain user call volume to a certain API, and a certain user call total to API calls. The target call interface may be an interface that needs to be monitored. The interface call monitoring data may be associated data with the target call interface being called. For example, the interface call monitoring data may include, but is not limited to, monitoring data for the amount of API calls, the type of API calls, the average elapsed time for API calls, and the rate of API call failures. The embodiment of the invention does not limit the specific data types of the monitoring data called by the user and the monitoring data called by the interface.
Specifically, before performing alarm processing on API call data, a data monitoring dimension is first determined. And if the interface calling condition needs to be monitored, determining the data monitoring dimension as the interface monitoring dimension, determining a target calling interface according to the interface monitoring dimension, and further performing data analysis according to the target calling interface to obtain interface calling monitoring data. And if the user calling condition needs to be monitored, determining the data monitoring dimension as a user monitoring dimension, determining a target calling user according to the user monitoring dimension, and further performing data analysis on API calling data of the target calling user to obtain user calling monitoring data. Therefore, the embodiment of the invention can determine the user calling monitoring data of the target calling user according to the user monitoring dimension and/or determine the interface calling monitoring data of the target calling interface according to the interface monitoring dimension.
In an optional embodiment of the present invention, determining call monitoring data of the API call data to be monitored according to the data monitoring dimension may include: subscribing a message queue through a distributed data processing server; acquiring API call data to be monitored according to the data monitoring dimension and the message queue through a distributed data processing server to calculate call monitoring data; the distributed data processing server determines to call the monitoring data in a distributed atom counting mode.
The distributed data processing server may be a server capable of parallel data processing. The distributed atomic counting approach may be an approach to perform batch task counting.
Correspondingly, after the API call data to be monitored is sent to the message queue, the distributed data processing server subscribing to the message queue may obtain the API call data to be monitored stored in the message queue. Specifically, the distributed data processing server may obtain API call data to be monitored in the message queue according to the data monitoring dimension. And further, the distributed data processing server calculates the calling monitoring data in a distributed atomic counting mode according to the data monitoring dimension and the API calling data to be monitored.
For example, assuming that the data monitoring dimension is a user monitoring dimension, the distributed data processing server may analyze user call monitoring data according to the API call data to be monitored in the message queue, and perform data processing on the user call monitoring data in a distributed atomic counting manner to obtain call monitoring data of a target call user. Specifically, the number of times a certain user calls a certain API, the total number of times a certain user calls the API, or the number of times all users call a certain API, calculated by the distributed data processing server, may be used as the call monitoring data of the target call user. In addition, the distributed data processing server can exert the advantages of distributed computing, so that the linear increase of data processing performance becomes possible by adding the data processing server, the data processing efficiency is greatly improved, the effect of quickly consuming the API call data to be monitored is realized, and the early warning feedback speed is improved.
And S240, determining to call a monitoring data threshold.
The calling monitoring data threshold may be a fixed value or a data range.
Specifically, the monitoring data threshold is determined and invoked according to the interface monitoring dimension and/or the user monitoring dimension. The monitoring data threshold value is called and can be set according to the actual early warning requirement.
For example, the number of times that a user calls a certain interface within a certain time range can be determined according to the user monitoring dimension, if the number of times that the user calls the certain interface falls within a certain data interval, it is characterized that the user calls the interface to belong to a normal access behavior, and further the data interval can be used as a monitoring data threshold. Similarly, the data interval may be a fixed value.
And S250, under the condition that the data statistic of the calling monitoring data is larger than or equal to the calling monitoring data threshold, determining that the calling monitoring data meets the data early warning condition, and performing alarm processing on the API calling data.
The data statistics may be the result of calling the monitoring data for data processing.
Specifically, before the data early warning condition is determined, the calling monitoring data can be determined, data processing is further performed on the calling monitoring data to obtain data statistics, if the data statistics is larger than or equal to the calling monitoring data threshold, the calling monitoring data meets the data early warning condition, the API calling data corresponding to the calling monitoring data meeting the early warning condition is analyzed, and warning processing is performed on the API calling data. Otherwise, the calling monitoring data is indicated to not meet the data early warning condition, and at the moment, the alarm processing is not needed.
In the embodiment of the invention, the monitoring calculation of the API call data to be monitored is separated from the call of the external data resource of the data source management platform, namely the data call main flow, so that the operation of the data call main flow is not influenced, and the external data resource call time delay is not generated.
According to the embodiment of the invention, after the API call data to be monitored of the data source management platform is obtained, the call monitoring data of the API call data to be monitored is determined according to the data monitoring dimension, and the alarm processing is further carried out on the API call data under the condition that the call monitoring data meets the data early warning condition, so that the real-time alarm processing of the API call data from each data dimension is realized, the data dimension of data early warning is improved, and the expandability of data early warning is improved.
It should be noted that any permutation and combination between the technical features in the above embodiments also belong to the scope of the present invention.
EXAMPLE III
Fig. 3 is a schematic diagram of a data early warning apparatus according to a third embodiment of the present invention, and as shown in fig. 3, the apparatus includes: data acquisition module 310, call monitoring data determination module 320, and data early warning module 330, wherein:
the data acquisition module 310 is configured to acquire API call data to be monitored of the data source management platform;
a calling monitoring data determining module 320, configured to determine calling monitoring data of the API calling data to be monitored;
and the data early warning module 330 is configured to perform alarm processing on the API call data when it is determined that the call monitoring data meets the data early warning condition.
Optionally, the data obtaining module 310 is specifically configured to: and acquiring mirror image copy data of the original API call data from each external API of the data source management platform through a data mirror image copy method, wherein the mirror image copy data is used as the API call data to be monitored.
Optionally, the data early warning apparatus may further include: and the to-be-monitored API call data sending module is used for sending the to-be-monitored API call data to the message queue.
Optionally, the monitoring data determining module 320 is called, and is specifically configured to: determining a data monitoring dimension; and determining calling monitoring data of the API calling data to be monitored according to the data monitoring dimension.
Optionally, the data monitoring dimension includes an interface monitoring dimension and/or a user monitoring dimension; the monitoring data determination module 320 is specifically configured to: determining user calling monitoring data of a target calling user according to the user monitoring dimension; and/or determining the interface calling monitoring data of the target calling interface according to the interface monitoring dimension.
Optionally, the monitoring data determining module 320 is called, and is specifically configured to: subscribing a message queue through a distributed data processing server; acquiring the API call data to be monitored through the distributed data processing server according to the data monitoring dimension and the message queue to calculate the call monitoring data; and the distributed data processing server determines the calling monitoring data in a distributed atom counting mode.
Optionally, the data early warning module 330 is specifically configured to: determining a calling monitoring data threshold; and determining that the calling monitoring data meet a data early warning condition under the condition that the data statistic of the calling monitoring data is greater than or equal to the calling monitoring data threshold.
According to the technical scheme, API call data to be monitored are analyzed from the data source management platform, data processing is further carried out on the API call data to be monitored to obtain call monitoring data, and finally alarm processing is carried out on the API call data corresponding to the call monitoring data meeting the data early warning condition.
The data early warning device can execute the data early warning method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method. For details of the data early warning method provided in any embodiment of the present invention, reference may be made to the technical details not described in detail in this embodiment.
Since the data early warning apparatus described above is an apparatus capable of executing the data early warning method in the embodiment of the present invention, based on the data early warning method described in the embodiment of the present invention, a person skilled in the art can understand a specific implementation manner of the data early warning apparatus in the embodiment and various variations thereof, so that a detailed description of how the data early warning apparatus implements the data early warning method in the embodiment of the present invention is omitted here. As long as those skilled in the art implement the device used in the data early warning method in the embodiment of the present invention, the device is within the scope of the present application.
Example four
Fig. 4 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention. FIG. 4 illustrates a block diagram of an electronic device 412 suitable for use in implementing embodiments of the present invention. The electronic device 412 shown in fig. 4 is only an example and should not bring any limitations to the functionality and scope of use of the embodiments of the present invention. The electronic device 412 may be, for example, a computer device or a server device, etc.
As shown in fig. 4, the electronic device 412 is in the form of a general purpose computing device. The components of the electronic device 412 may include, but are not limited to: one or more processors 416, a storage device 428, and a bus 418 that couples the various system components including the storage device 428 and the processors 416.
The electronic device 412 may also communicate with one or more external devices 414 (e.g., keyboard, pointing device, camera, display 424, etc.), with one or more devices that enable a user to interact with the electronic device 412, and/or with any devices (e.g., network card, modem, etc.) that enable the electronic device 412 to communicate with one or more other computing devices. Such communication may be through an Input/Output (I/O) interface 422. Also, the electronic device 412 may communicate with one or more networks (e.g., a Local Area Network (LAN), Wide Area Network (WAN), and/or a public Network, such as the internet) via the Network adapter 420. As shown, network adapter 420 communicates with the other modules of electronic device 412 over bus 418. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 412, including but not limited to: microcode, device drivers, Redundant processing units, external disk drive Arrays, disk array (RAID) systems, tape drives, and data backup storage systems, to name a few.
The processor 416 executes various functional applications and data processing by executing programs stored in the storage device 428, for example, implementing the data pre-warning method provided by the above-described embodiment of the present invention: acquiring API call data to be monitored of a data source management platform; determining calling monitoring data of API calling data to be monitored; and under the condition that the calling monitoring data meet the data early warning condition, carrying out alarm processing on the API calling data.
According to the technical scheme, API call data to be monitored are analyzed from the data source management platform, data processing is further carried out on the API call data to be monitored to obtain call monitoring data, and finally alarm processing is carried out on the API call data corresponding to the call monitoring data meeting the data early warning condition.
EXAMPLE five
An embodiment of the present invention further provides a computer storage medium storing a computer program, where the computer program is executed by a computer processor to perform the data early warning method according to any one of the above embodiments of the present invention: acquiring API call data to be monitored of a data source management platform; determining calling monitoring data of API calling data to be monitored; and under the condition that the calling monitoring data meet the data early warning condition, carrying out alarm processing on the API calling data.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM) or flash Memory), an optical fiber, a portable compact disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, Radio Frequency (RF), etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A data early warning method is characterized by comprising the following steps:
acquiring API call data to be monitored of a data source management platform;
determining calling monitoring data of the API calling data to be monitored;
and under the condition that the calling monitoring data are determined to meet the data early warning condition, carrying out alarm processing on the API calling data.
2. The method according to claim 1, wherein the obtaining API call data to be monitored of the data source management platform includes:
and acquiring mirror copy data of the original API call data from each external API of the data source management platform by a data mirror copy method, wherein the mirror copy data is used as the API call data to be monitored.
3. The method of claim 2, further comprising, after obtaining the mirror copy data of the original API call data from each external API of the data source management platform by the data mirror copy method:
and sending the API call data to be monitored to a message queue.
4. The method of claim 1, wherein the determining call monitoring data of the API call data to be monitored comprises:
determining a data monitoring dimension;
and determining calling monitoring data of the API calling data to be monitored according to the data monitoring dimension.
5. The method of claim 4, wherein the data monitoring dimension comprises an interface monitoring dimension and/or a user monitoring dimension;
the determining the calling monitoring data of the API calling data to be monitored according to the data monitoring dimension comprises the following steps:
determining user calling monitoring data of a target calling user according to the user monitoring dimension; and/or
And determining interface calling monitoring data of the target calling interface according to the interface monitoring dimension.
6. The method according to claim 4 or 5, wherein the determining call monitoring data of the API call data to be monitored according to the data monitoring dimension comprises:
subscribing a message queue through a distributed data processing server;
acquiring the API call data to be monitored through the distributed data processing server according to the data monitoring dimension and the message queue to calculate the call monitoring data;
and the distributed data processing server determines the calling monitoring data in a distributed atom counting mode.
7. The method of claim 1, wherein the determining that the call monitoring data satisfies a data pre-alarm condition comprises:
determining a calling monitoring data threshold;
and determining that the calling monitoring data meet a data early warning condition under the condition that the data statistic of the calling monitoring data is greater than or equal to the calling monitoring data threshold.
8. A data early warning device, characterized by comprising:
the data acquisition module is used for acquiring API call data to be monitored of the data source management platform;
the calling monitoring data determining module is used for determining calling monitoring data of the API calling data to be monitored;
and the data early warning module is used for carrying out alarm processing on the API calling data under the condition that the calling monitoring data is determined to meet the data early warning condition.
9. An electronic device, characterized in that the electronic device comprises:
one or more processors;
storage means for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the data alert method of any of claims 1-7.
10. A computer storage medium on which a computer program is stored, which program, when being executed by a processor, carries out the data pre-warning method as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011529721.3A CN112597130A (en) | 2020-12-22 | 2020-12-22 | Data early warning method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011529721.3A CN112597130A (en) | 2020-12-22 | 2020-12-22 | Data early warning method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112597130A true CN112597130A (en) | 2021-04-02 |
Family
ID=75200744
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011529721.3A Pending CN112597130A (en) | 2020-12-22 | 2020-12-22 | Data early warning method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112597130A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109062699A (en) * | 2018-08-15 | 2018-12-21 | 郑州云海信息技术有限公司 | A kind of resource monitoring method, device, server and storage medium |
| CN110278124A (en) * | 2019-06-18 | 2019-09-24 | 努比亚技术有限公司 | The monitoring method of interface, device and computer readable storage medium on line |
| CN110362455A (en) * | 2019-07-15 | 2019-10-22 | 北京奇艺世纪科技有限公司 | A kind of data processing method and data processing equipment |
| CN111427878A (en) * | 2020-03-20 | 2020-07-17 | 深圳乐信软件技术有限公司 | Data monitoring and alarming method, device, server and storage medium |
-
2020
- 2020-12-22 CN CN202011529721.3A patent/CN112597130A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109062699A (en) * | 2018-08-15 | 2018-12-21 | 郑州云海信息技术有限公司 | A kind of resource monitoring method, device, server and storage medium |
| CN110278124A (en) * | 2019-06-18 | 2019-09-24 | 努比亚技术有限公司 | The monitoring method of interface, device and computer readable storage medium on line |
| CN110362455A (en) * | 2019-07-15 | 2019-10-22 | 北京奇艺世纪科技有限公司 | A kind of data processing method and data processing equipment |
| CN111427878A (en) * | 2020-03-20 | 2020-07-17 | 深圳乐信软件技术有限公司 | Data monitoring and alarming method, device, server and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8595564B2 (en) | Artifact-based software failure detection | |
| JP5474177B2 (en) | Distributed application monitoring | |
| US5193178A (en) | Self-testing probe system to reveal software errors | |
| US10489232B1 (en) | Data center diagnostic information | |
| EP3304315A1 (en) | Automatic anomaly detection and resolution system | |
| CN108595316B (en) | Lifecycle management method, manager, device, and medium for distributed application | |
| US8694625B2 (en) | Selective registration for remote event notifications in processing node clusters | |
| US10541892B2 (en) | System and method for monitoring, sensing and analytics of collaboration devices | |
| CN111240940B (en) | Real-time service monitoring method and device, electronic equipment and storage medium | |
| CN111726358A (en) | Attack path analysis method and device, computer equipment and storage medium | |
| CN111522703A (en) | Method, apparatus and computer program product for monitoring access requests | |
| CN115629933A (en) | Business system monitoring method, device, equipment and storage medium | |
| CN112306833A (en) | Application program crash statistical method and device, computer equipment and storage medium | |
| CN113098715B (en) | Information processing method, device, system, medium and computing equipment | |
| US9009735B2 (en) | Method for processing data, computing node, and system | |
| CN113760491A (en) | Task scheduling system, method, equipment and storage medium | |
| US9348721B2 (en) | Diagnosing entities associated with software components | |
| EP2770447B1 (en) | Data processing method, computational node and system | |
| US9594622B2 (en) | Contacting remote support (call home) and reporting a catastrophic event with supporting documentation | |
| CN112597130A (en) | Data early warning method and device, electronic equipment and storage medium | |
| CN115514618A (en) | Alarm event processing method and device, electronic equipment and medium | |
| CN110928940B (en) | Data writing method and device based on kafka cluster, electronic equipment and storage medium | |
| CN113419835A (en) | Job scheduling method, device, equipment and medium | |
| CN114024867B (en) | Network anomaly detection method and device | |
| CN112711487A (en) | Data source management and control method and device, management and control server and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |