CN117149570A - Data processing method, device, equipment and storage medium - Google Patents
Data processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN117149570A CN117149570A CN202311129653.5A CN202311129653A CN117149570A CN 117149570 A CN117149570 A CN 117149570A CN 202311129653 A CN202311129653 A CN 202311129653A CN 117149570 A CN117149570 A CN 117149570A
- Authority
- CN
- China
- Prior art keywords
- data
- storage area
- marking
- target
- target storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 24
- 238000012544 monitoring process Methods 0.000 claims abstract description 136
- 238000012545 processing Methods 0.000 claims abstract description 67
- 230000002159 abnormal effect Effects 0.000 claims abstract description 40
- 238000000034 method Methods 0.000 claims abstract description 34
- 238000004590 computer program Methods 0.000 claims description 20
- 238000013500 data storage Methods 0.000 claims description 12
- 238000005516 engineering process Methods 0.000 abstract description 3
- 230000005540 biological transmission Effects 0.000 abstract description 2
- 230000015654 memory Effects 0.000 description 15
- 238000010586 diagram Methods 0.000 description 12
- 238000004891 communication Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 230000005856 abnormality Effects 0.000 description 4
- 238000011835 investigation Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 238000013480 data collection Methods 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000013468 resource allocation Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 238000013138 pruning Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3055—Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0652—Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- Human Computer Interaction (AREA)
- Mathematical Physics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The disclosure provides a data processing method, a device, equipment and a storage medium, which can be applied to the technical fields of network transmission and financial science and technology. The method comprises the following steps: storing monitoring data associated with the application system to a target storage area under the condition that the abnormal running state of the application system is detected; marking the monitoring data in the target storage area to obtain first marking data; processing the first mark data based on the storage resource use state of the target storage area to obtain second mark data matched with the storage resource use state; and uploading the marking data to the target server under the condition that the running state of the application system is detected to be normal, wherein the marking data comprise the first marking data and/or the second marking data.
Description
Technical Field
The present disclosure relates to the field of network transmission and financial technology, and in particular, to a data processing method, apparatus, device, storage medium, and program product.
Background
With the development of network technology, monitoring data becomes an indispensable part for monitoring the running states of various application systems, and meanwhile, the influence degree of network stability on the quality of the monitoring data is higher and higher.
In the process of implementing the disclosed concept, the inventor finds that the following defects exist in the related art in terms of the collection of monitoring data: in an abnormal network state, the efficiency of monitoring data acquisition is low, the quality of the acquired data is low, and the actual business production requirements are difficult to meet.
Disclosure of Invention
In view of the foregoing, the present disclosure provides a data processing method, apparatus, device, storage medium, and program product.
According to a first aspect of the present disclosure, there is provided a data processing method comprising: storing monitoring data associated with the application system into a target storage area under the condition that the abnormal running state of the application system is detected;
marking the monitoring data in the target storage area to obtain first marking data;
processing the first marking data based on the storage resource use state of the target storage area to obtain second marking data matched with the storage resource use state;
and uploading the marking data to a target server under the condition that the running state of the application system is detected to be recovered to be normal, wherein the marking data comprises the first marking data and/or the second marking data.
According to an embodiment of the present disclosure, the processing the first tag data based on the storage resource usage status of the target storage area to obtain second tag data matched with the storage resource usage status includes:
monitoring storage resource usage status information of the target storage area based on the data amount of the first flag data stored in the target storage area; and
and processing the first marking data under the condition that the storage resource use state information is larger than or equal to a preset threshold value, and generating second marking data matched with the storage resource use state.
According to an embodiment of the present disclosure, when the storage resource usage status information is greater than or equal to a preset threshold, processing the first flag data to generate second flag data that matches the storage resource usage status includes:
under the condition that the storage resource use state information is larger than or equal to a preset threshold value, carrying out downsampling on the first marking data to generate downsampled first marking data; and
and under the condition that the data volume of the first mark data after the downsampling is larger than or equal to the upper limit of the storage resource allocation, deleting and updating the first mark data after the downsampling according to a relevant preset rule to generate the second mark data.
According to an embodiment of the present disclosure, before the uploading of the tag data to the target server side under the condition that the restoration of the operation state of the application system is detected, the method further includes:
under the condition that the abnormal running state of the application system is detected, the uploading of the monitoring data to the target server is suspended; and
and sending a network connection request to the target server at regular time based on a preset rule related to the abnormal running state of the application system.
According to an embodiment of the present disclosure, before the above-mentioned monitoring data associated with the above-mentioned application system is stored in the target storage area in the case where an abnormal running state of the application system is detected, the method further includes:
storing monitoring data to the target storage area under the condition that the running state of the application system is detected to be normal;
uploading the monitoring data in the target storage area to the target server, and
and processing the monitoring data in the target storage area based on the uploading result of the monitoring data in the target storage area.
According to an embodiment of the present disclosure, the processing the monitoring data in the target storage area based on the uploading result of the monitoring data in the target storage area includes:
Deleting the monitoring data in the target storage area under the condition that the uploading result of the monitoring data in the target storage area is normal; and
and when the uploading result of the monitoring data in the target storage area is abnormal, the monitoring data in the target storage area is reserved.
According to an embodiment of the present disclosure, after the uploading of the tag data to the target server side under the condition that the restoration of the operation state of the application system is detected, the method further includes:
and deleting the mark data in the target storage area when the uploading result of uploading the mark data to the target server is normal.
A second aspect of the present disclosure provides a data processing apparatus comprising:
the data storage module is used for storing the monitoring data associated with the application system to the target storage area under the condition that the abnormal running state of the application system is detected;
the data marking module is used for marking the monitoring data in the target storage area to obtain first marking data;
the data processing module is used for processing the first marking data based on the storage resource use state of the target storage area to obtain second marking data matched with the storage resource use state; and
And the data uploading module is used for uploading the marking data to the target server under the condition that the running state of the application system is detected to be recovered to be normal, wherein the marking data comprises the first marking data and/or the second marking data.
A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method.
A fourth aspect of the present disclosure also provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the above-described method.
A fifth aspect of the present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the above method.
According to the data processing method, the device, the equipment, the storage medium and the program product, monitoring data are automatically stored in the target storage area through abnormal conditions of monitoring the running state of the application system, the stored data in the target storage area are automatically marked, first marked data are generated, the first marked data are processed, second marked data are obtained, the marked data are uploaded to the target server based on the condition that the running state of the application system is recovered to be normal, the integrity of monitoring data resources is guaranteed under the condition that the network is abnormal, the efficiency of monitoring data acquisition is improved, and objective data basis is provided for the investigation of system abnormal problems.
Drawings
The foregoing and other objects, features and advantages of the disclosure will be more apparent from the following description of embodiments of the disclosure with reference to the accompanying drawings, in which:
FIG. 1 schematically illustrates an application scenario diagram of a data processing method, apparatus, device, medium and program product according to an embodiment of the present disclosure;
FIG. 2 schematically illustrates a flow chart of a data processing method according to an embodiment of the disclosure;
FIG. 3 schematically illustrates a flow chart of another data processing method according to an embodiment of the disclosure;
FIG. 4 schematically illustrates a system architecture diagram of a data processing method according to an embodiment of the present disclosure;
FIG. 5 schematically illustrates a block diagram of a data processing apparatus according to an embodiment of the present disclosure; and
fig. 6 schematically illustrates a block diagram of an electronic device adapted to implement a data processing method according to an embodiment of the disclosure.
Detailed Description
Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and/or the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.
Where expressions like at least one of "A, B and C, etc. are used, the expressions should generally be interpreted in accordance with the meaning as commonly understood by those skilled in the art (e.g.," a system having at least one of A, B and C "shall include, but not be limited to, a system having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).
In the technical solution of the present disclosure, the related user information (including, but not limited to, user personal information, user image information, user equipment information, such as location information, etc.) and data (including, but not limited to, data for analysis, stored data, displayed data, etc.) are information and data authorized by the user or sufficiently authorized by each party, and the related data is collected, stored, used, processed, transmitted, provided, disclosed, applied, etc. and processed, all in compliance with the related laws and regulations and standards of the related country and region, necessary security measures are taken, no prejudice to the public order, and corresponding operation entries are provided for the user to select authorization or rejection.
The inventor finds that in the related art, in an abnormal network state, the efficiency of monitoring data acquisition is low, the quality of the acquired monitoring data is low, and the actual service production requirement is difficult to meet, and meanwhile, in the abnormal network state, the system frequently sends network requests and error codes due to the fact that middleware cannot be normally connected, and finally, the conditions of high system occupation memory rate and CPU utilization rate are caused. In view of this, the present disclosure provides a method for processing data, by automatically storing monitoring data in a target storage area through an abnormal condition of monitoring an operation state of an application system, and automatically marking the stored data in the target storage area, generating first marked data, processing the first marked data, thereby obtaining second marked data, and uploading the marked data to a target server based on a condition that the operation state of the application system is recovered to be normal, thereby ensuring the integrity of monitoring data resources under a network abnormal condition, improving the efficiency of monitoring data acquisition, and providing objective data basis for the investigation of abnormal problems of the system.
The embodiment of the disclosure provides a data processing method, a device, equipment and a storage medium, wherein the method comprises the following steps: storing monitoring data associated with the application system to a target storage area under the condition that the abnormal running state of the application system is detected; marking the monitoring data in the target storage area to obtain first marking data; processing the first mark data based on the storage resource use state of the target storage area to obtain second mark data matched with the storage resource use state; and uploading the marking data to the target server under the condition that the running state of the application system is detected to be normal, wherein the marking data comprise the first marking data and/or the second marking data.
Fig. 1 schematically illustrates an application scenario diagram of a data processing method, apparatus, device and storage medium according to an embodiment of the present disclosure.
As shown in fig. 1, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, a server 105, and a target server 106. The network 104 is a medium used to provide a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others. The target server 106 may be a remote monitoring server, where the target server 106 is connected to the server 105 through a network, and the server 105 uploads the monitoring data to the target server 106 through the network.
The user may interact with the server 105 through the network 104 using at least one of the first terminal device 101, the second terminal device 102, the third terminal device 103, to receive or send messages, etc. Various communication client applications, such as a shopping class application, a web browser application, a search class application, an instant messaging tool, a mailbox client, social platform software, etc. (by way of example only) may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103.
The first terminal device 101, the second terminal device 102, the third terminal device 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 105 may be a server providing various services, such as a background management server (by way of example only) providing support for websites browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as the user request, and feed back the processing result (e.g., a web page, information, or data acquired or generated according to the user's data processing request) to the terminal device.
It should be noted that the data processing method provided in the embodiments of the present disclosure may be generally performed by the server 105. Accordingly, the data processing apparatus provided by the embodiments of the present disclosure may be generally provided in the server 105. The data processing method provided by the embodiments of the present disclosure may also be performed by a server or a server cluster that is different from the server 105 and is capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, the target server 106, and/or the server 105. Accordingly, the data processing apparatus provided by the embodiments of the present disclosure may also be provided in a server or a server cluster that is different from the server 105 and is capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, the target server 106, and/or the server 105.
It should be understood that the number of terminal devices, networks, target servers and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, target servers, and servers, as desired for implementation.
Fig. 2 schematically illustrates a flow chart of a data processing method according to an embodiment of the present disclosure.
As shown in fig. 2, the data processing method of this embodiment includes operations S210 to S240.
In operation S210, in case an abnormality of the operation state of the application system is detected, monitoring data associated with the application system is stored to a target storage area.
According to embodiments of the present disclosure, the application system may include an application monitoring system, a data monitoring system, a service resource monitoring system, and the like. The operating state anomaly may be an application system network state anomaly, such as a network disconnection, a network delay, a network drop, etc. The monitoring data can be data obtained by collecting performance data, resource data or availability data of the application system, and specifically, the performance data can be data processing response processing speed of various application systems or related monitoring data of processing stability of various application systems in a certain time; the resource data can be monitoring data of the occupation condition of resources such as CPU utilization rate, memory utilization rate and the like of various application systems; the availability data may be monitoring data such as whether various application systems are available (whether the server is down, responds normally), whether the application systems are available (whether an application process exists), and the like. The target storage area may be a storage space in the application system for storing the monitoring data, such as cloud storage, a mobile hard disk, etc., which is not limited herein, and meanwhile, the type of the monitoring data, the size of the data, the format of the data, etc. in the target storage area are not limited herein.
In operation S220, the monitoring data in the target storage area is marked to obtain first marking data.
According to the embodiment of the present disclosure, the marking data may be adding a preset identifier to the monitoring data in the target storage area, for example, "to be uploaded, xxxxxx" is added before various types of monitoring data, and it should be noted that a form and an adding manner of the preset identifier may be set according to actual needs of a user, which is not limited specifically herein.
In operation S230, the first tag data is processed based on the storage resource usage status of the target storage area, resulting in second tag data that matches the storage resource usage status.
According to embodiments of the present disclosure, the storage resource usage state may include a used state and a remaining storage state of the storage resource. The used state may be a state of periodically monitoring the use state of the storage resource through the system, including CPU usage, memory usage, disk space, and the like. These may be implemented by monitoring tools, agents, or other means. By monitoring, the current use condition of the storage resource can be known. The remaining storage information may be calculated according to the basic information of the storage resource and the currently stored monitoring data, and these may be calculated by comprehensively considering the indexes such as the CPU utilization rate, the memory utilization condition, and the like. The specific computing method may be determined based on system requirements and storage resource configuration.
In operation S240, if it is detected that the operating state of the application system is restored to normal, the tag data is uploaded to the target server, where the tag data includes the first tag data and/or the second tag data.
According to an embodiment of the present disclosure, the operation state restoration may be a restoration of a network state of the application system. The target server may be a remote monitoring server, which may provide capabilities of real-time monitoring, timely alarming, data grouping, etc., for example, a monitoring platform for cloud resources.
According to the embodiment of the disclosure, the monitoring data is automatically stored in the target storage area through the abnormal condition of the monitoring of the running state of the application system, the stored data in the target storage area is automatically marked to generate the first marked data, the first marked data is processed to obtain the second marked data, and the marked data is uploaded to the target server based on the condition that the running state of the application system is recovered to be normal, so that the integrity of the monitoring data resource is ensured under the abnormal condition of the network, the efficiency of monitoring data acquisition is improved, and objective data basis is provided for the investigation of the abnormal problem of the system.
According to an embodiment of the present disclosure, processing first tag data based on a storage resource usage state of a target storage area to obtain second tag data matching the storage resource usage state includes: monitoring storage resource usage status information of the target storage area based on a data amount of the first flag data stored in the target storage area; and processing the first marking data under the condition that the storage resource use state information is greater than or equal to a preset threshold value, and generating second marking data matched with the storage resource use state.
According to an embodiment of the present disclosure, the storage resource usage status may include a used status and a remaining storage status of the storage resource, and the preset threshold may be a threshold set according to used status information and/or remaining storage information of the storage resource, for example, 90% of the used status information of the storage resource reaches an upper limit of the storage resource configuration, that is, 90% is the preset threshold, and it should be noted that the preset threshold may be set progressively according to a user requirement or an actual situation, which is not specifically limited herein. The processing of the first flag data may be downsampling, pruning or updating the first flag data according to whether the used state information of the storage resource has reached the upper limit of the storage resource configuration or the remaining storage information of the storage resource reaches the threshold of the storage resource.
In another possible embodiment, the processing of the first tag data may further be a resource expansion of a storage resource storing the first tag data, for example, expanding a disk space.
According to the embodiment of the disclosure, the first marked data is processed, so that the storage requirement of the target storage area for the monitoring data can be met under the condition of network failure, and the integrity of the monitoring data resource is ensured.
According to an embodiment of the present disclosure, in a case where storage resource usage state information is greater than or equal to a preset threshold, processing first tag data to generate second tag data that matches the storage resource usage state includes: under the condition that the storage resource use state information is greater than or equal to a preset threshold value, carrying out downsampling on the first marking data to generate downsampled first marking data; and under the condition that the data volume of the first mark data after the downsampling is larger than or equal to the upper limit of the storage resource configuration, deleting and updating the first mark data after the downsampling according to the related preset rule to generate second mark data.
According to the embodiment of the disclosure, the downsampling process may be to generalize granularity of the monitored data collection, reduce fineness of the monitored data collection to achieve the purpose of reducing storage resources occupied by the data, for example, the granularity of the data collected originally may be 1 second for one data page, and downsampling process may be performed for 3 seconds for one data page. After the first flag data is downsampled, if the downsampled first flag data amount in the target storage area is greater than or equal to the storage resource allocation upper limit after a period of time, the downsampled first flag data is subjected to deletion and updating, for example, the 1 month monitoring data amount in the original target storage area is deleted and updated, and the monitoring data amount of 20 days in the recent period is reserved.
According to an embodiment of the present disclosure, before the uploading of the tag data to the target server side is detected to be restored to normal operation state of the application system, the method further includes: under the condition that the abnormal running state of the application system is detected, uploading of the monitoring data to the target server side is suspended; and sending a network connection request to the target server at regular time based on a preset rule associated with the abnormal running state of the application system.
According to the embodiment of the disclosure, when the abnormal running state of the application system network is detected, uploading of the monitoring data to the target server is stopped, and the network connection request is only sent to the target server at regular time.
According to the embodiment of the disclosure, under the abnormal network state, the timing is set to send the network connection request to the target server, so that the situation that the system frequently sends the network request and the error code due to the fact that the network cannot be normally connected and finally the system occupies memory rate and CPU utilization rate is high is avoided.
Fig. 3 schematically illustrates a flow chart of another data processing method according to an embodiment of the disclosure.
According to an embodiment of the present disclosure, as shown in fig. 3, before storing the monitoring data associated with the application system to the target storage area in the case where the operation state abnormality of the application system is detected, operations S310 to S330 are further included.
In operation S310, the monitoring data is stored to the target storage area in case that it is detected that the operation state of the application system is restored to normal.
In operation S320, the monitoring data in the target storage area is uploaded to the target server.
In operation S330, the monitoring data in the target storage area is processed based on the uploading result of the monitoring data in the target storage area.
According to the embodiment of the disclosure, according to the uploading result of the monitoring data in the target storage area, the monitoring data in the target storage area is deleted or reserved.
According to an embodiment of the present disclosure, based on an upload result of monitoring data in a target storage area, processing the monitoring data in the target storage area includes: deleting the monitoring data in the target storage area under the condition that the uploading result of the monitoring data in the target storage area is normal; and under the condition that the uploading result of the monitoring data in the target storage area is abnormal, the monitoring data in the target storage area is reserved.
According to the embodiment of the disclosure, when uploading of the monitoring data in the target storage area is successful, the monitoring data in the target storage area is deleted to provide a storage space for the monitoring data to be stored later, and when uploading of the monitoring data in the target storage area fails, the monitoring data in the target storage area is reserved and the reserved monitoring data is processed later to meet actual service requirements.
According to an embodiment of the present disclosure, after uploading the tag data to the target server under the condition that it is detected that the operation state of the application system is restored to normal, the method further includes: and deleting the marked data in the target storage area under the condition that the uploading result of uploading the marked data to the target server is normal.
Fig. 4 schematically illustrates a system architecture diagram of a data processing method according to an embodiment of the present disclosure.
According to the embodiment of the disclosure, as shown in fig. 4, performance data, resource data or availability data of an application system are acquired by a data acquisition module to obtain monitoring data; the collected monitoring data are stored into a target storage area through a data storage module; under the condition that the running state of the application system is abnormal (network abnormality) is detected, marking the monitoring data in the target storage area through a data marking module to obtain first marking data, and processing the first marking data according to the storage resource use state of the target storage area to obtain second marking data matched with the storage resource use state; and uploading the marked data to the target server through the data uploading module under the condition that the running state of the application system is detected to be normal (network is normal).
Based on the data processing method, the disclosure also provides a data processing device. The device will be described in detail below in connection with fig. 5.
Fig. 5 schematically shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure.
As shown in fig. 5, the data processing apparatus 500 of this embodiment includes a data storage module 510, a data marking module 520, a data processing module 530, and a data uploading module 540.
The data storage module 510 is configured to store, in a case where an abnormal operation state of the application system is detected, monitoring data associated with the application system to the target storage area. In an embodiment, the data storage module 510 may be used to perform the operation S210 described above, which is not described herein.
The data marking module 520 is configured to mark the monitored data in the target storage area to obtain first marked data. In an embodiment, the data marking module 520 may be used to perform the operation S220 described above, which is not described herein.
The data processing module 530 is configured to process the first tag data based on a storage resource usage state of the target storage area, to obtain second tag data that matches the storage resource usage state. In an embodiment, the data processing module 530 may be configured to perform the operation S230 described above, which is not described herein.
And the data uploading module 540 is configured to upload the tag data to the target server under the condition that the operating state of the application system is detected to be recovered to be normal, where the tag data includes the first tag data and/or the second tag data. In an embodiment, the data uploading module 540 may be used to perform the operation S240 described above, which is not described herein.
According to the embodiment of the disclosure, through the data storage module 510, the data marking module 520, the data processing module 530 and the data uploading module 540, by monitoring the abnormal condition of the operation state of the application system, the monitoring data is automatically stored in the target storage area, the stored data in the target storage area is automatically marked, the first marking data is generated, the first marking data is processed, the second marking data is obtained, and the marking data is uploaded to the target server based on the condition that the operation state of the application system is recovered to be normal, so that the integrity of the monitoring data resource is ensured under the condition of network abnormality, the efficiency of monitoring data acquisition is improved, and meanwhile, objective data basis is provided for the investigation of the abnormal problem of the system.
According to an embodiment of the present disclosure, a data processing module includes an information monitoring sub-module and a data processing sub-module.
And the information monitoring sub-module is used for monitoring the storage resource use state information of the target storage area based on the data quantity of the first marking data stored in the target storage area.
And the data processing sub-module is used for processing the first marking data under the condition that the storage resource use state information is greater than or equal to a preset threshold value, and generating second marking data matched with the storage resource use state.
According to an embodiment of the present disclosure, a data processing sub-module includes a downsampling processing unit and an updating processing unit.
And the downsampling processing unit is used for downsampling the first mark data under the condition that the storage resource use state information is greater than or equal to a preset threshold value, so as to generate the downsampled first mark data.
And the updating processing unit is used for deleting and updating the first mark data after the downsampling according to the related preset rule under the condition that the data volume of the first mark data after the downsampling is larger than or equal to the upper limit of the storage resource configuration, so as to generate second mark data.
According to an embodiment of the present disclosure, the data processing apparatus further includes a suspend module and a request transmitting module.
And the pause module is used for pausing uploading of the monitoring data to the target server side under the condition that the abnormal running state of the application system is detected before uploading of the marking data to the target server side under the condition that the normal running state of the application system is detected.
The request sending module is used for sending the network connection request to the target server at regular time based on a preset rule related to the abnormal running state of the application system before uploading the marking data to the target server under the condition that the running state of the application system is detected to be normal.
According to an embodiment of the present disclosure, the data processing apparatus further includes a monitoring data storage module, a monitoring data uploading module, and a monitoring data processing module.
And the monitoring data storage module is used for storing the monitoring data associated with the application system to the target storage area before the monitoring data are stored to the target storage area under the condition that the abnormal running state of the application system is detected, and storing the monitoring data to the target storage area under the condition that the normal running state of the application system is detected.
And the monitoring data uploading module is used for uploading the monitoring data in the target storage area to the target server before the monitoring data associated with the application system is stored in the target storage area under the condition that the running state of the application system is abnormal.
And the monitoring data processing module is used for processing the monitoring data in the target storage area based on the uploading result of the monitoring data in the target storage area before the monitoring data associated with the application system is stored in the target storage area under the condition that the abnormal running state of the application system is detected.
According to an embodiment of the present disclosure, a monitoring data processing module includes a delete sub-module and a reserve sub-module.
And the deleting sub-module is used for deleting the monitoring data in the target storage area under the condition that the uploading result of the monitoring data in the target storage area is normal.
And the reservation sub-module is used for reserving the monitoring data in the target storage area under the condition that the uploading result of the monitoring data in the target storage area is abnormal.
According to an embodiment of the present disclosure, the data processing apparatus further includes a data deleting module configured to delete the tag data in the target storage area after uploading the tag data to the target server if the operation state of the application system is detected to be restored to be normal, and if an uploading result of uploading the tag data to the target server is normal.
Any of the data storage module 510, the data tagging module 520, the data processing module 530, and the data uploading module 540 may be combined in one module or any of the modules may be split into multiple modules according to embodiments of the present disclosure. Alternatively, at least some of the functionality of one or more of the modules may be combined with at least some of the functionality of other modules and implemented in one module. According to embodiments of the present disclosure, at least one of the data storage module 510, the data tagging module 520, the data processing module 530, and the data uploading module 540 may be implemented at least in part as hardware circuitry, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware in any other reasonable way of integrating or packaging the circuitry, or in any one of or a suitable combination of three of software, hardware, and firmware. Alternatively, at least one of the data storage module 510, the data tagging module 520, the data processing module 530, and the data uploading module 540 may be at least partially implemented as a computer program module which, when executed, performs the corresponding functions.
Fig. 6 schematically illustrates a block diagram of an electronic device adapted to implement a data processing method according to an embodiment of the disclosure.
As shown in fig. 6, an electronic device 600 according to an embodiment of the present disclosure includes a processor 601 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. The processor 601 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or an associated chipset and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), or the like. Processor 601 may also include on-board memory for caching purposes. The processor 601 may comprise a single processing unit or a plurality of processing units for performing different actions of the method flows according to embodiments of the disclosure.
In the RAM 603, various programs and data necessary for the operation of the electronic apparatus 600 are stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. The processor 601 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM 602 and/or the RAM 603. Note that the program may be stored in one or more memories other than the ROM 602 and the RAM 603. The processor 601 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in the one or more memories.
According to an embodiment of the present disclosure, the electronic device 600 may also include an input/output (I/O) interface 605, the input/output (I/O) interface 605 also being connected to the bus 604. The electronic device 600 may also include one or more of the following components connected to an input/output (I/O) interface 605: an input portion 606 including a keyboard, mouse, etc.; an output portion 607 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The drive 610 is also connected to an input/output (I/O) interface 605 as needed. Removable media 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on drive 610 so that a computer program read therefrom is installed as needed into storage section 608.
The present disclosure also provides a computer-readable storage medium that may be embodied in the apparatus/device/system described in the above embodiments; or may exist alone without being assembled into the apparatus/device/system. The computer-readable storage medium carries one or more programs which, when executed, implement methods in accordance with embodiments of the present disclosure.
According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, the computer-readable storage medium may include ROM 602 and/or RAM 603 and/or one or more memories other than ROM 602 and RAM 603 described above.
Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowcharts. The program code means for causing a computer system to carry out the data processing methods provided by the embodiments of the present disclosure when the computer program product is run on the computer system.
The above-described functions defined in the system/apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 601. The systems, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.
In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted, distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and/or installed from the removable medium 611. The computer program may include program code that may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.
In such an embodiment, the computer program may be downloaded and installed from a network through the communication portion 609, and/or installed from the removable medium 611. The above-described functions defined in the system of the embodiments of the present disclosure are performed when the computer program is executed by the processor 601. The systems, devices, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.
According to embodiments of the present disclosure, program code for performing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, such computer programs may be implemented in high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. Programming languages include, but are not limited to, such as Java, c++, python, "C" or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
Those skilled in the art will appreciate that the features recited in the various embodiments of the disclosure and/or in the claims may be provided in a variety of combinations and/or combinations, even if such combinations or combinations are not explicitly recited in the disclosure. In particular, the features recited in the various embodiments of the present disclosure and/or the claims may be variously combined and/or combined without departing from the spirit and teachings of the present disclosure. All such combinations and/or combinations fall within the scope of the present disclosure.
The embodiments of the present disclosure are described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the embodiments cannot be used advantageously in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be made by those skilled in the art without departing from the scope of the disclosure, and such alternatives and modifications are intended to fall within the scope of the disclosure.
Claims (11)
1. A data processing method, comprising:
storing monitoring data associated with an application system to a target storage area under the condition that the abnormal running state of the application system is detected;
Marking the monitoring data in the target storage area to obtain first marking data;
processing the first marking data based on the storage resource use state of the target storage area to obtain second marking data matched with the storage resource use state;
and uploading the marking data to a target server under the condition that the running state of the application system is detected to be normal, wherein the marking data comprise the first marking data and/or the second marking data.
2. The method of claim 1, wherein the processing the first tag data based on the storage resource usage status of the target storage area to obtain second tag data that matches the storage resource usage status comprises:
monitoring storage resource usage state information of the target storage area based on the data amount of the first flag data stored in the target storage area; and
and processing the first marking data under the condition that the storage resource use state information is larger than or equal to a preset threshold value, and generating second marking data matched with the storage resource use state.
3. The method according to claim 2, wherein the processing the first tag data to generate second tag data matching the storage resource usage status if the storage resource usage status information is greater than or equal to a preset threshold value includes:
performing downsampling processing on the first mark data under the condition that the storage resource use state information is greater than or equal to a preset threshold value, and generating downsampled first mark data; and
and under the condition that the data quantity of the first mark data after the downsampling is larger than or equal to the upper limit of the storage resource configuration, deleting and updating the first mark data after the downsampling according to a related preset rule to generate the second mark data.
4. The method of claim 1, wherein the step of uploading the tag data to the target server before detecting that the operating state of the application system is restored to normal further comprises:
under the condition that the abnormal running state of the application system is detected, the uploading of the monitoring data to the target server side is suspended; and
and sending a network connection request to the target server at regular time based on a preset rule associated with the abnormal running state of the application system.
5. The method of claim 1, wherein, in the event that an operational state anomaly of an application system is detected, prior to storing monitoring data associated with the application system to a target storage area, further comprising:
storing monitoring data to the target storage area under the condition that the running state of the application system is detected to be normal;
uploading the monitoring data in the target storage area to the target server, and
and processing the monitoring data in the target storage area based on the uploading result of the monitoring data in the target storage area.
6. The method of claim 5, wherein the processing the monitoring data in the target storage area based on the uploading result of the monitoring data in the target storage area comprises:
deleting the monitoring data in the target storage area under the condition that the uploading result of the monitoring data in the target storage area is normal; and
and under the condition that the uploading result of the monitoring data in the target storage area is abnormal, reserving the monitoring data in the target storage area.
7. The method of claim 1, wherein after the uploading the tag data to the target server side under the condition that the operating state of the application system is detected to be recovered to be normal, further comprising:
and deleting the marked data in the target storage area under the condition that the uploading result of uploading the marked data to the target server is normal.
8. A data processing apparatus comprising:
the data storage module is used for storing monitoring data associated with the application system to a target storage area under the condition that the abnormal running state of the application system is detected;
the data marking module is used for marking the monitoring data in the target storage area to obtain first marking data;
the data processing module is used for processing the first marking data based on the storage resource use state of the target storage area to obtain second marking data matched with the storage resource use state; and
and the data uploading module is used for uploading the marking data to the target server under the condition that the running state of the application system is detected to be recovered to be normal, wherein the marking data comprises the first marking data and/or the second marking data.
9. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs,
wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-7.
10. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method according to any of claims 1-7.
11. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311129653.5A CN117149570A (en) | 2023-09-04 | 2023-09-04 | Data processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311129653.5A CN117149570A (en) | 2023-09-04 | 2023-09-04 | Data processing method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117149570A true CN117149570A (en) | 2023-12-01 |
Family
ID=88883878
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311129653.5A Pending CN117149570A (en) | 2023-09-04 | 2023-09-04 | Data processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117149570A (en) |
-
2023
- 2023-09-04 CN CN202311129653.5A patent/CN117149570A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10523580B2 (en) | Automatic cloud provisioning based on related internet news and social network trends | |
| CN111597065B (en) | Method and device for collecting equipment information | |
| CN110727560A (en) | Cloud service alarm method and device | |
| CN113900834B (en) | Data processing method, device, equipment and storage medium based on Internet of things technology | |
| CN107644075B (en) | Method and device for collecting page information | |
| CN111427701A (en) | Workflow engine system and business processing method | |
| CN115357761A (en) | Link tracking method and device, electronic equipment and storage medium | |
| CN113495825A (en) | Line alarm processing method and device, electronic equipment and readable storage medium | |
| CN116360937A (en) | Task scheduling method, device, equipment and storage medium | |
| CN116302561A (en) | State control method, device, equipment and storage medium for application instance | |
| CN117149570A (en) | Data processing method, device, equipment and storage medium | |
| CN115202973A (en) | Application running state determining method and device, electronic equipment and medium | |
| CN112818204B (en) | Service processing method, device, equipment and storage medium | |
| CN115080434A (en) | Case execution method, device, equipment and medium | |
| CN114168607A (en) | Global serial number generation method, device, equipment, medium and product | |
| CN113781154A (en) | Information rollback method, system, electronic equipment and storage medium | |
| CN114721882B (en) | Data backup method and device, electronic equipment and storage medium | |
| CN114844810B (en) | Heartbeat data processing method, device, equipment and medium | |
| CN113760587B (en) | Method and device for processing data | |
| CN110262756B (en) | Method and device for caching data | |
| CN117349565A (en) | Page rendering method, device, equipment and medium | |
| CN116795543A (en) | Data processing method, device, equipment and storage medium | |
| CN117271277A (en) | Alarming method, device, equipment and storage medium for abnormal service | |
| CN116821251A (en) | Data processing method, data processing device, electronic equipment and storage medium | |
| CN114861054A (en) | Information acquisition method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |