CN116185635A - Data processing, elastic expansion method, computing device and computer storage medium - Google Patents

Data processing, elastic expansion method, computing device and computer storage medium Download PDF

Info

Publication number
CN116185635A
CN116185635A CN202310202536.0A CN202310202536A CN116185635A CN 116185635 A CN116185635 A CN 116185635A CN 202310202536 A CN202310202536 A CN 202310202536A CN 116185635 A CN116185635 A CN 116185635A
Authority
CN
China
Prior art keywords
index data
data
index
memory
time
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310202536.0A
Other languages
Chinese (zh)
Inventor
周弘懿
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba China Co Ltd
Original Assignee
Alibaba China Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba China Co Ltd filed Critical Alibaba China Co Ltd
Priority to CN202310202536.0A priority Critical patent/CN116185635A/en
Publication of CN116185635A publication Critical patent/CN116185635A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5061—Partitioning or combining of resources
    • G06F9/5072—Grid computing
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The embodiment of the application provides a data processing method, an elastic stretching method, computing equipment and a computer storage medium. The data processing method is applied to an index acquisition end and comprises the following steps: collecting first index data generated by a data source in a first time window; detecting that the first index data is stored in the local storage medium, and storing the first index data into the local storage medium so as to process the first index data in real time; acquiring the first index data from the local storage medium; and processing the first index data to obtain second index data, wherein the second index data is used for triggering an execution end to execute preset operation, and the first index data and the second index data are index values representing the running state of the data source. The technical scheme provided by the embodiment of the invention shortens the link of index data processing and reduces the complexity of index data processing.

Description

Data processing, elastic expansion method, computing device and computer storage medium
Technical Field
The embodiment of the invention relates to the technical field of data processing, in particular to a data processing method, an elastic telescoping method, computing equipment and a computer storage medium.
Background
The data index may include data formed by quantifying a certain event of the data source, and may be used to measure an operational state of the data source. Thus, in order to grasp the operation state of the data source, it is generally necessary to acquire index data from the data source by using the index acquisition terminal.
In general, index data obtained by directly collecting the data from a data source at an index collection end is referred to as raw index data. With the development of big data, the original index data cannot meet the requirements of accuracy and real-time performance of data source running state detection, so that the original index data is further processed to obtain target index data to accurately and timely detect the data source after being collected.
In the prior art, the processing link of the original index data is generally that the index collection end collects the original index data from the data source, then the index collection end gathers the collected original index data into a message queue, such as kafka, rocketMQ, and then uses a computing engine, such as spark, flink, etc., to consume the original index data from the message queue for processing.
The inventor finds that the processing mode of the original index data in the related technology has the technical problems of high overall complexity and low stability in the process of realizing the inventive concept.
Disclosure of Invention
The embodiment of the invention provides a data processing method, an elastic expansion method, a device, a computing device and a computer storage medium.
In a first aspect, an embodiment of the present invention provides a data processing method, which is applied to an index collection end, including:
collecting first index data generated by a data source in a first time window;
storing the first index data in a local storage medium;
detecting that the first index data is stored in the local storage medium, and acquiring the first index data from the local storage medium so as to process the first index data in real time;
and processing the first index data to obtain second index data, wherein the second index data is used for triggering an execution end to execute preset operation, and the first index data and the second index data are index values representing the running state of the data source.
In a second aspect, an embodiment of the present invention provides an elastic extension method, which is applied to an index collection end, including:
collecting third index data generated by a first cloud server in a cloud service cluster in a first time window;
storing the third index data in a local storage medium;
Detecting that the third index data is stored in the local storage medium, and acquiring the third index data from the local storage medium so as to process the third index data in real time;
processing the third index data to obtain fourth index data, wherein the third index data and the fourth index data are index values representing the running state of the first cloud server;
generating an elastic expansion instruction under the condition that the fourth index data meets the data requirement;
and sending the elastic expansion instruction to an execution end, so that the execution end responds to the elastic expansion instruction to execute elastic expansion processing on the cloud service cluster.
In a third aspect, an embodiment of the present invention provides a data processing apparatus, including:
the first acquisition module is used for acquiring first index data generated by the data source in a first time window;
the first storage module is used for storing the first index data into a local storage medium;
the first acquisition module is used for detecting that the first index data is stored in the local storage medium, and acquiring the first index data from the local storage medium so as to process the first index data in real time;
The first processing module is used for processing the first index data to obtain second index data, and the second index data is used for triggering the execution end to execute preset operation, wherein the first index data and the second index data are index values representing the running state of the data source.
In a fourth aspect, an embodiment of the present invention provides an elastic telescopic device, including:
the second acquisition module is used for acquiring third index data generated by the first cloud server in the cloud service cluster in the first time window;
the second storage module is used for storing the third index data into a local storage medium;
the second acquisition module is used for detecting that the third index data is stored in the local storage medium, and acquiring the third index data from the local storage medium so as to process the third index data in real time;
the second processing module is used for processing the third index data to obtain fourth index data, wherein the third index data and the fourth index data are index values representing the running state of the first cloud server;
the first instruction generation module is used for generating an elastic telescopic instruction under the condition that the fourth index data meets the data requirement;
And the first instruction sending module is used for sending the elastic telescopic instruction to an execution end so that the execution end responds to the elastic telescopic instruction to execute elastic telescopic processing on the cloud server cluster.
The embodiment of the invention adopts the first index data generated by the acquisition data source in the first time window; storing the first index data in a local storage medium; detecting that the first index data is stored in the local storage medium, and acquiring the first index data from the local storage medium so as to process the first index data in real time; the method comprises the steps of processing index data to obtain second index data, wherein the second index data are used for triggering an execution end to execute preset operation, the first index data and the second index data are the technical scheme of representing index values of the running state of a data source, so that the processing of the index data can be locally performed at an index acquisition end at the edge side, other middleware is not relied on in the processing process, a link of index data processing is shortened, and the complexity of index data processing is reduced.
These and other aspects of the invention will be more readily apparent from the following description of the embodiments.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 schematically illustrates a flow chart of a data processing method provided by one embodiment of the present invention;
FIG. 2 schematically illustrates a schematic diagram of a data processing method according to an embodiment of the present invention;
FIG. 3 schematically illustrates a flow chart of an elastic telescoping method provided by one embodiment of the present invention;
FIG. 4 schematically illustrates a block diagram of a data processing apparatus provided by one embodiment of the present invention;
FIG. 5 schematically illustrates a block diagram of an elastic telescoping apparatus provided by one embodiment of the present invention;
FIG. 6 schematically illustrates a block diagram of a computing device provided by one embodiment of the invention.
Detailed Description
In order to enable those skilled in the art to better understand the present invention, the following description will make clear and complete descriptions of the technical solutions according to the embodiments of the present invention with reference to the accompanying drawings.
In some of the flows described in the specification and claims of the present invention and in the foregoing figures, a plurality of operations occurring in a particular order are included, but it should be understood that the operations may be performed out of order or performed in parallel, with the order of operations such as 101, 102, etc., being merely used to distinguish between the various operations, the order of the operations themselves not representing any order of execution. In addition, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel. It should be noted that, the descriptions of "first" and "second" herein are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, and are not limited to the "first" and the "second" being different types.
It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present invention are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region, and provide corresponding operation entries for the user to select authorization or rejection.
The data index may include data formed by quantifying a certain event of the data source, and may be used to measure an operational state of the data source. Thus, in order to grasp the operation state of the data source, it is generally necessary to acquire index data from the data source by using the index acquisition terminal.
In general, index data obtained by directly collecting the data from a data source at an index collection end is referred to as raw index data. With the development of big data, the original index data cannot meet the requirements of accuracy and real-time performance of data source running state detection, so that the original index data is further processed to obtain target index data to accurately and timely detect the data source after being collected.
In the prior art, the processing link of the original index data is generally that the index collection end collects the original index data from the data source, then the index collection end gathers the collected original index data into a message queue, such as kafka, rocketMQ, and then uses a computing engine, such as spark, flink, etc., to consume the original index data from the message queue for processing.
The inventor finds that the processing mode of the original index data in the related technology has the technical problems of high overall complexity and low stability in the process of realizing the inventive concept.
In order to solve the technical problems in the related art, the embodiment of the invention provides a data processing method, which adopts first index data generated by a collected data source in a first time window; storing the first index data in a local storage medium; detecting that the first index data is stored in the local storage medium, and acquiring the first index data from the local storage medium so as to process the first index data in real time; the method comprises the steps of processing index data to obtain second index data, wherein the second index data are used for triggering an execution end to execute preset operation, and the first index data and the second index data are index values representing the running state of a data source.
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to fall within the scope of the invention.
Fig. 1 schematically illustrates a flowchart of a data processing method according to an embodiment of the present invention, where the data processing method may be applied to an index collection end, and may include the following steps:
101, collecting first index data generated by a data source in a first time window;
102, storing the first index data into a local storage medium;
103, detecting that the first index data is stored in the local storage medium, and acquiring the first index data from the local storage medium;
104, processing the first index data to obtain second index data, wherein the second index data is used for triggering the execution end to execute a preset operation, and the first index data and the second index data are index values representing the running state of the data source.
According to the embodiment of the invention, the first index data can be used for reflecting the operation state of a data source, such as normal or abnormal operation, high or low operation speed and the like, wherein the data source can be an application, a system (middleware, a database, an operating system, storage), a network, equipment and the like. It should be noted that, the first index data may be a phenomenon index that characterizes a working state of a data source, for example: the first index data may be, but not limited to, a fine index indicating performance of a machine or a system, for example, process utilization, CPU utilization, memory utilization, and the like.
According to the embodiment of the invention, the index data is an index value reflecting the running state of the data source, and can be stored in the form of index name-index value, so that the index data has the characteristic of small data volume, and the data volume of one index data is generally within 150 bytes.
According to the embodiment of the invention, the index collection end can be deployed at the data source side so that the index collection end can timely collect the first index data after the first index data is generated by the data source.
According to the embodiment of the invention, under the condition that the index acquisition end is deployed on the data source side, the acquired first index data can be stored in a local storage medium of the data source. Because the data volume of the index data is small, when the first index data is stored in the local storage medium of the data source, the storage pressure is not caused to the local storage medium of the data source.
Furthermore, in the actual application scenario, the acquisition and processing of the index data have real-time requirements, so that the real-time performance of the triggering preset operation of the execution end is ensured, and therefore, the index acquisition end only acquires the first index data generated by the data source in a shorter first time window, such as the last 3 seconds and the last 5 seconds. Since the data amount of each piece of index data is small and the number of index data is small, the storage pressure of the local storage medium of the data source can be further reduced.
According to one embodiment of the present invention, the data source may notify the index collection terminal after generating the first index data, so that the index collection terminal collects the first index data in response to the notification of the data source.
According to another embodiment of the present invention, the index collection end may actively pull the first index data from the data source.
According to the embodiment of the invention, after the first index data is collected by the index collection end, the first index data can be stored in a local storage medium, wherein the local storage medium can comprise a hard disk, a magnetic disk, a memory, a flash memory and the like which are deployed locally.
According to one embodiment of the invention, the index acquisition device can detect the local storage medium in real time, and when the first index data is detected to be stored in the local storage medium, the stored first index data is directly acquired from the local storage medium and processed. Therefore, the real-time performance of the first index data processing can be improved.
According to another embodiment of the invention, the index collection device may wait for a data processing request, which may be initiated by a user or by another device, and in case the data processing request is obtained, obtain the first index data from the local storage medium and process it.
According to other embodiments of the present invention, the index collection device may be further configured to obtain the first index data from the local storage medium at preset time intervals, where the preset time intervals may be set according to the requirement of instantaneity.
According to an embodiment of the present invention, the processing of the first index data may include, for example, a series of operations performed based on the first index data, so that the second index data may be the result of the operations of the first index data.
According to the embodiment of the invention, when the first index data is subjected to the operation processing, the operation may be performed based on only the first index data, but is not limited thereto, and additional reference data may be acquired and the first index data may be operated based on the reference data to obtain the second index data.
According to an embodiment of the present invention, the preset operation may include, for example, a capacity expansion operation, an upgrade operation, an operation and maintenance operation, and the like.
In the embodiment of the invention, the first index data generated by the acquisition data source in a first time window is adopted; storing the first index data in a local storage medium; detecting that the first index data is stored in the local storage medium, and acquiring the first index data from the local storage medium so as to process the first index data in real time; according to the technical scheme, the first index data and the second index data are index values representing the running state of the data source, and the first index data are only collected by the index collecting device in a first time window and are index values reflecting the state of the data source, so that the occupation of storage resources and calculation resources of the index collecting device is small, the index data can be processed locally at the index collecting end at the edge side, other middleware is not relied on in the processing process, the link of index data processing is shortened, the complexity of index data processing is reduced, the local storage medium is detected in real time after the first index data are stored in the local storage medium, the first index data are processed immediately after the first index data are stored in the local storage medium, and the real-time performance of subsequent processing based on the processing result of the index data is improved.
According to an embodiment of the present invention, the data processing method further includes:
determining whether the first index data meets a release condition or not, wherein the release condition is determined according to the acquisition time of the first index data;
and deleting the first index data from the local storage medium to release the storage space of the local storage medium under the condition that the first index data meets the release condition.
According to an embodiment of the present invention, determining whether the first index data satisfies the release condition may be specifically implemented as:
determining the acquisition time of the first index data;
determining a difference value between the acquisition time and the current time;
comparing the difference value with a preset time threshold value, and determining that the first index data meets the release condition under the condition that the acquisition time is greater than the preset time threshold value, or else, determining that the first index data does not meet the release condition.
According to the embodiment of the invention, the release condition may, for example, indicate that the first index data has no real-time property, for example, the acquisition time of the first index data is far from the current time, and because the processing of the index data has a high requirement on the real-time property of the index data, the first index data having no real-time property belongs to invalid data in this case, the first index data may be deleted from the local storage medium, and the storage space of the local storage medium is released.
According to the embodiment of the invention, when the first index data is acquired, the index acquisition device may also record the acquisition time of the first index data as a timestamp of the first index data.
According to an embodiment of the present invention, the acquisition time of the first index data may be determined by a time stamp of the first index data.
According to the embodiment of the invention, the real-time property of the first index data can be reflected by the difference value obtained by making the difference between the acquisition time of the first index data and the current time, and the larger the difference value is, the worse the real-time property is, the smaller the difference value is, and the closer the acquisition time of the first index data is to the current time, the stronger the real-time property is.
According to the embodiment of the present invention, the preset time threshold may be, for example, 5 seconds, 10 seconds, 1 minute, etc., and the value of the preset time threshold is not limited in the embodiment of the present invention, and the value of the preset time threshold may be flexibly set by a person skilled in the art according to the real-time requirement for the first index data processing.
According to an embodiment of the present invention, the data processing method further includes:
determining whether the second index data meets the preset data requirement;
Generating a trigger instruction under the condition that the second index data meets the preset data requirement;
and sending the trigger instruction to an execution end corresponding to the data source so that the execution end can respond to the trigger instruction to execute the preset operation.
According to an embodiment of the present invention, the preset data requirements may be preset according to the operation state requirements of the data source. The preset data requirement may be implemented, for example, greater than a preset threshold, less than a preset threshold, greater than or equal to a preset threshold, less than or equal to a preset threshold, etc. For example, when the first index data is the CPU utilization, the preset data requirement may be, for example, that the CPU utilization is less than a preset threshold; when the first index data is the transaction response time, the preset data requirement may be, for example, that the transaction response time is greater than a preset threshold.
According to the embodiment of the invention, the index acquisition end can execute the judging process of whether the second index data meets the preset data requirement or not, and correspondingly, the triggering instruction can also be generated at the index acquisition end.
According to the embodiment of the invention, the triggering instruction can be realized as the prompt information, and the prompt information can be sent to related personnel after being generated, so that the related personnel can judge whether the preset operation needs to be executed or not based on the prompt information after acquiring the prompt information. But is not limited thereto, the trigger instruction may also be implemented as an executable instruction, which may be directly sent to the execution end, so that the execution end executes the executable instruction.
According to the embodiment of the invention, the index acquisition end can acquire a plurality of data sources, and correspondingly, each data source can respectively correspond to different execution ends. After the trigger instruction is generated, an execution end corresponding to the data source generating the first index data may be determined, and the trigger instruction may be sent to the execution end.
Fig. 2 schematically illustrates a schematic diagram of a data processing method according to an embodiment of the present invention.
In fig. 2, 201 may represent a data source, 202 may represent an index collection end, 2021 may represent a storage medium of the index collection end, 2022 may represent a calculation module of the index collection end, and 203 may represent an execution end.
As shown in fig. 2, the index collection end 202 may collect first index data from the data source 201, and then the index collection end 202 may store the first index data to a storage medium 2021 local to the index collection end 202.
Then, when the calculation processing needs to be performed on the first index data, the calculation module may acquire the first index data from the storage medium 2021, then perform the calculation processing on the first index data to obtain second index data, and determine whether the second index data meets the data requirement, so as to generate a trigger instruction when the second index data meets the data requirement, and send the trigger instruction to the execution end.
According to an embodiment of the present invention, storing the index data into the storage device may be specifically implemented as:
storing the first index data into a first memory of an index acquisition end;
and under the condition that the storage amount of the first memory exceeds a first storage threshold value, transferring the first index data from the first memory to the second memory.
According to the embodiment of the invention, the memory is fast, so that the first index data can be stored into the first memory of the index acquisition end after the first index data is acquired in order to ensure the real-time performance of the first index data acquisition.
According to the embodiment of the invention, a large amount of first index data is generated by the data source, if all the acquired data are stored in the first memory of the index acquisition end, on one hand, the memory overhead is increased, and on the other hand, if more memory space in the first memory is occupied, the index acquisition end or the data source lacks enough memory space to execute other processes or operations. Therefore, when the storage amount of the first memory exceeds the first storage threshold, the first index data can be transferred from the first memory to the second memory, wherein the storage amount of the second memory is larger than the storage amount of the first memory.
According to an embodiment of the present invention, the second memory may include a shared memory (shared memory) of an operating system of the computing device, but is not limited thereto, and may also include a local disk or an external memory, such as NAS (Network Attached Storage ).
According to the embodiment of the invention, the first index data carries a time stamp, and the time stamp is determined according to the acquisition time of the first index data.
According to an embodiment of the present invention, the first index data may be time-series data, and the time-series data may be a series of time data arranged in chronological order. The time interval of a set of time series is a constant value (e.g., 10 seconds, 1 minute, 5 minutes).
According to an embodiment of the present invention, each time data includes a time point and an index data value, and the time data in the time sequence is illustratively stored in a (value) pair, where the value is used to indicate the index data value, and the time is used to indicate the timestamp.
According to an embodiment of the present invention, the obtaining the first index data from the local storage medium may be specifically implemented as:
determining whether the first index data is stored in the first memory based on the timestamp;
And under the condition that the first index data is determined to be stored in the first memory, acquiring the first index data from the first memory, otherwise, acquiring the first index data from the second memory.
According to the embodiment of the invention, after the first index data is acquired, the first index data is firstly stored in the first memory, and then the first index data can be transferred from the first memory to the second memory, so that before the first index data is acquired, the first index data is required to be firstly stored in the first memory or the second memory.
According to the embodiment of the invention, since the time stamp is determined by sensing the acquisition time of the first index data, when the first index data is acquired, the time stamp can be determined by the acquisition time first, then the time stamp of the existing time data in the first memory can be traversed first, whether the time data existing in the first memory has the same time stamp as the time stamp of the first index data or not is determined, if so, the first index data is determined to be stored in the first memory, so that the first index data can be acquired from the first memory, and if not, the first index data is determined to have been transferred from the first memory to the second memory, and the first index data can be acquired from the second memory.
According to an embodiment of the present invention, the data processing method further includes:
determining whether the storage amount of the second memory exceeds a second storage threshold;
and under the condition that the storage amount of the second memory exceeds a second storage threshold value, executing memory release operation on the second memory so as to reduce the storage amount of the second memory.
According to the embodiment of the invention, the first index data stored in the second memory carries a time stamp, and the time stamp is determined according to the acquisition time of the first index data.
According to an embodiment of the present invention, performing a memory release operation on the second memory may be specifically implemented as:
acquiring a preset reference time stamp;
determining third index data stored in the second memory and corresponding to a timestamp with a preset time span of the reference timestamp;
and deleting the third index data from the second memory.
According to an embodiment of the present invention, the reference time stamp may be a fixed time, but is not limited thereto, and the reference time stamp may be determined based on the current time. In particular, the reference timestamp may characterize a time that is a preset length of time from the current time, e.g., the reference timestamp may characterize a timestamp 10 minutes ago with the current time.
According to an embodiment of the present invention, the preset time span may be implemented as a time difference, which may be obtained by subtracting the reference time stamp from the time stamp of the time data stored in the second memory, for example.
According to the embodiment of the present invention, time data having a time difference smaller than 0 may be used as the third index data.
According to the embodiment of the invention, the time data corresponding to the time difference smaller than 0 is the time data stored in the second memory before the reference time, and the part of the time data is longer than the current time, so that the part of the time data can be considered to be unnecessary to be reused, and the part of the time data can be deleted from the second memory as the third index data.
According to an embodiment of the present invention, after the first index data is transferred from the first memory to the second memory of the computing device where the index collection end is disposed, the data processing method further includes:
and deleting the first index data stored in the first memory.
According to an embodiment of the present invention, the first index data generated by the collected data source may be specifically implemented as:
and acquiring first index data generated by a data source according to a preset acquisition frequency.
According to the embodiment of the invention, the specific value of the preset acquisition frequency can be flexibly specified by a person skilled in the art according to the real-time requirement of index data acquisition, and the embodiment of the invention does not specifically limit the value of the preset acquisition frequency.
According to an embodiment of the present invention, the first index data generated by the collected data source may be specifically implemented as:
acquiring a data acquisition request;
responding to a data acquisition request, and determining at least one first index acquisition end belonging to the same cluster with the index acquisition end;
and taking the at least one first index acquisition end as a data source to acquire survival index data generated by the at least one first index acquisition equipment.
According to the embodiment of the invention, the data acquisition request can be triggered periodically, namely, the index acquisition end can acquire the index of at least the first acquisition end in the same cluster at intervals.
According to the embodiment of the invention, the index acquisition terminals belonging to the same cluster can communicate with each other, and any one index acquisition terminal can acquire indexes by taking other index acquisition terminals as data sources.
According to an embodiment of the invention, the survival index data may be used to characterize whether the index collection end is in a production working state.
According to an embodiment of the present invention, the processing of the first index data may be specifically implemented as:
and determining second index data based on the survival index data of each first index acquisition end, wherein the second index data is used for representing the survival rate of index equipment in the cluster.
According to the embodiment of the invention, the first number of the index collection ends currently in the production working state can be firstly determined according to the survival index data, and then the second index data is obtained by dividing the first number by the total number of the index collection ends.
According to an embodiment of the present invention, the data processing method further includes:
generating an operation and maintenance instruction under the condition that the second index data is lower than a preset survival threshold value;
and sending the operation and maintenance instruction to the instruction execution end so that the execution end can respond to the operation and maintenance instruction to execute operation and maintenance operation on the cluster.
According to the embodiment of the invention, the operation and maintenance operation can comprise operation and maintenance operation of the index collection end or deployment operation of the index collection end, and the survival rate of the index collection end in the cluster is improved by operating and maintaining the index collection end which is not in a survival state or deploying a new index collection end.
According to the embodiment of the invention, the preset survival threshold value can be flexibly set by a person skilled in the art according to practical application requirements, and can be 90%, 95% or the like.
According to the embodiment of the invention, the processing of the first index data to obtain the second index data can be specifically implemented as:
determining a target processing mode corresponding to the data source from a plurality of candidate processing modes;
And processing the first index data by using a target processing mode to obtain second index data.
According to the embodiment of the invention, corresponding processing modes can be configured for different data sources in advance, and after the first index data is acquired from the storage device, the processing mode corresponding to the data source can be determined.
According to an embodiment of the present invention, the processing manner may include, for example:
under the same time stamp, performing four arithmetic operations on the plurality of time data to obtain second index data; or under the same time window, calculating the average value, the median and the like of a plurality of time data to obtain second index data; or comparing the same index data with the same ratio under different time windows to obtain second index data.
According to the embodiment of the invention, since the processing of the index data generally only involves numerical operation of the data value, the requirement on the computing performance of the data source is low, and therefore, when the processing of the index data is performed, only less computing resources of the data source are required to be occupied.
The following description will be made with reference to a specific embodiment of a specific implementation process for obtaining the second index data by performing the first index data, and it should be noted that the following examples are only used to illustrate an exemplary processing manner, and those skilled in the art may flexibly select other processing manners according to actual application requirements.
In one embodiment of the present invention, the first index data may include time data a, time data b and time data c, and the acquisition time stamp is 1674029000, and the second index data may be obtained by adding time data b to time data a and dividing by time data c.
The specific calculation process is as follows:
(metric_a{timestamp＝1674029000}+metric_b{timestamp＝1674029000})/metric_c{timest amp＝1674029000}。
where metric_a represents time data a, metric_b represents time data b, metric_c represents time data c, and timestamp represents a timestamp.
In another embodiment of the present invention, the first index data may include a value of the time data d in the range of the time stamps 1674029000 to 1674039000 and a value of the time data e in the range of the time stamps 1674029000 to 1674039000, in which case the second index data may be obtained by first calculating a first average value of the time data d in the range of the time stamps 1674029000 to 1674039000, then calculating a second average value of the time data e in the range of the time stamps 1674029000 to 1674039000, and then dividing the first average value by the second average value.
The specific calculation process is as follows:
avg(range(metric_d{timestamp＝1674029000},metric_d{timestamp＝1674039000}))/avg(ra nge(metric_e{timestamp＝1674029000},metric_e{timestamp＝1674039000}))。
Where metric_d represents time data d, metric_e represents time data e, and timestamp represents a timestamp.
In another embodiment of the present invention, the first index data may include a value of the time data f in the range of the time stamps 1674029000 to 1674039000 and a value of the time data f in the range of the time stamps 1674019000 to 1674029000, in which case the second index data may be obtained by calculating a third average value of the time data f in the range of the time stamps 1674029000 to 1674039000, then calculating a fourth average value of the time data f in the range of the time stamps 1674019000 to 1674029000, and then calculating a difference value between the third average value and the fourth average value.
The specific calculation process is as follows:
avg(range(metric_f{timestamp＝1674029000},metric_f{timestamp＝1674039000}))―
avg(range(metric_f{timestamp＝1674019000},metric_f{timestamp＝1674029000}))。
where metric_f represents time data f and timestamp represents a timestamp.
Fig. 3 schematically illustrates a flowchart of an elastic stretching method according to an embodiment of the present invention, where the elastic stretching method may be applied to an index collection end, and may include the following steps:
301, collecting third index data generated by a first cloud server in a cloud service cluster in a first time window;
302, storing third index data in a local storage medium;
303, detecting that the third index data is stored in the local storage medium, and acquiring the third index data from the local storage medium so as to process the third index data in real time;
304, processing the third index data to obtain fourth index data, wherein the third index data and the fourth index data are index values representing the running state of the first cloud server;
305, generating an elastic expansion instruction under the condition that the fourth index data meets the data requirement;
306, sending the elastic expansion instruction to the execution end, so that the execution end responds to the elastic expansion instruction to execute elastic expansion processing on the cloud service cluster.
According to an embodiment of the present invention, generating the elastic expansion instruction may be specifically implemented as:
generating a first elastic expansion instruction, wherein the first elastic expansion instruction is used for indicating an execution end to adjust the number of cloud servers in a cloud service cluster;
or alternatively, the process may be performed,
and generating a second elastic expansion instruction, wherein the second elastic expansion instruction is used for indicating the execution end to adjust the configuration of the first cloud server.
According to an embodiment of the present invention, when the fourth index data characterizes that the first cloud server is currently facing a larger pressure, the index collecting end may generate an elastic expansion instruction, and the elastic expansion instruction may instruct the executing end to create at least one second cloud server in the cloud server cluster, so that the pressure facing the first cloud server is shared by using the at least one second cloud server.
According to another embodiment of the present invention, when the fourth index data characterizes that the first cloud server is currently under a larger or smaller pressure, the index collection end may generate an elastic expansion instruction, and the elastic expansion instruction may instruct the execution end to increase or decrease the resource configuration of the first cloud server. For example, when the pressure faced by the first cloud server is high, the CPU of the first cloud server may be adjusted from 2 cores to 4 cores, so as to improve the processing capability of the first cloud server; when the pressure faced by the first cloud server is smaller, the CPU of the first cloud server can be adjusted from 4 cores to 2 cores, so that the running cost of the first cloud server is reduced.
In an embodiment of the present invention, the configuration of the first cloud server may further include, for example, GPU core number, memory capacity, disk capacity, and the like of the first cloud server.
Fig. 4 schematically illustrates a block diagram of a data processing apparatus according to an embodiment of the present invention, as shown in fig. 4, a data processing apparatus 400 may include:
a first acquisition module 401, configured to acquire first index data generated by a data source within a first time window;
a first storage module 402, configured to store first index data in a local storage medium;
The first obtaining module 403 is configured to detect that the first index data is stored in the local storage medium, and obtain the first index data from the local storage medium, so as to perform real-time processing on the first index data;
the first processing module 404 is configured to process the first index data to obtain second index data, where the second index data is used to trigger the execution end to execute a preset operation, and the first index data and the second index data are index values that represent an operation state of the data source.
According to an embodiment of the present invention, the data processing apparatus 400 further includes:
the first determining module is used for determining whether the second index data meets the preset data requirement;
the second instruction generation module is used for generating a trigger instruction under the condition that the second instruction data meets the preset data requirement;
and the second instruction sending module is used for sending the trigger instruction to the execution end corresponding to the data source so that the execution end can respond to the trigger instruction to execute the preset operation.
According to an embodiment of the present invention, the first storage module 402 includes:
the first storage sub-module is used for storing the first index data into a first memory of the index acquisition end;
and the transferring sub-module is used for transferring the first index data from the first memory to the second memory of the computing equipment of the deployment index acquisition end under the condition that the storage amount of the first memory exceeds a first storage threshold value.
According to the embodiment of the invention, the first index data carries a time stamp, and the time stamp is determined according to the acquisition time of the first index data.
According to an embodiment of the present invention, the first acquisition module 403 includes:
a first determination submodule for determining whether the first index data is stored in the first memory or not based on the time stamp;
the first obtaining sub-module is used for obtaining the first index data from the first memory under the condition that the first index data are determined to be stored in the first memory, otherwise, obtaining the first index data from the second memory.
According to an embodiment of the present invention, the data processing apparatus 400 further includes:
the second determining module is used for determining whether the storage amount of the second memory exceeds a second storage threshold value;
and the memory release module is used for executing memory release operation on the second memory under the condition that the storage amount of the second memory exceeds a second storage threshold value so as to reduce the storage amount of the second memory.
According to the embodiment of the invention, the first index data stored in the second memory carries a time stamp, and the time stamp is determined according to the acquisition time of the first index data.
According to an embodiment of the present invention, a memory release module includes:
the reference time stamp obtaining mold module is used for obtaining a preset reference time stamp;
The third index data determining submodule is used for determining third index data stored in the second memory and corresponding to a time stamp with a preset time span of the reference time stamp;
and the first deleting sub-module is used for deleting the third index data from the second memory.
According to an embodiment of the present invention, the data processing apparatus 400 further includes:
the first deleting module is used for deleting the first index data stored in the first memory.
According to an embodiment of the present invention, the first acquisition module 401 includes:
the first acquisition sub-module is used for acquiring first index data generated by a data source according to a preset acquisition frequency.
According to an embodiment of the present invention, the first acquisition module 401 includes:
the request acquisition module is used for acquiring a data acquisition request;
the acquisition end determining module is used for responding to the data acquisition request and determining at least one first index acquisition end belonging to the same cluster with the index acquisition end;
and the second acquisition sub-module is used for taking at least one first index acquisition end as a data source and acquiring survival index data generated by at least one first index acquisition device.
According to an embodiment of the invention, the first processing module 404 comprises:
the first processing submodule is used for determining second index data based on the survival index data of each first index acquisition end, and the second index data are used for representing the survival rate of index equipment in the cluster;
According to an embodiment of the present invention, the data processing apparatus 400 further includes:
the third instruction generation module is used for generating an operation and maintenance instruction under the condition that the second instruction data is lower than a preset survival threshold value;
and the third instruction sending module is used for sending the operation and maintenance instruction to the instruction execution end so that the execution end responds to the operation and maintenance instruction to execute operation and maintenance operation on the cluster.
According to an embodiment of the invention, the first processing module 404 comprises:
the processing mode determining submodule is used for determining a target processing mode corresponding to the data source from a plurality of candidate processing modes;
and the second processing sub-module is used for processing the first index data by utilizing the target processing mode to obtain second index data.
The data processing apparatus of fig. 4 may perform the data processing method of the embodiment shown in fig. 1, and its implementation principle and technical effects are not repeated. The specific manner in which the respective modules and units of the data processing apparatus in the above embodiments perform operations has been described in detail in the embodiments related to the method, and will not be described in detail here.
Fig. 5 schematically illustrates a block diagram of an elastic telescopic device according to an embodiment of the present invention, as shown in fig. 4, an elastic telescopic device 500 may include:
The second collection module 501 is configured to collect third index data generated by a first cloud server in the cloud service cluster within a first time window;
a second storage module 502, configured to store third index data into a local storage medium;
a second obtaining module 503, configured to detect that the third index data is stored in the local storage medium, and obtain the third index data from the local storage medium, so as to perform real-time processing on the third index data;
the second processing module 504 is configured to process the third index data to obtain fourth index data, where the third index data and the fourth index data are index values representing an operation state of the first cloud server;
the first instruction generating module 505 is configured to generate an elastic expansion instruction when the fourth index data meets the data requirement;
the first instruction sending module 506 is configured to send an elastic scaling instruction to the execution end, so that the execution end performs elastic scaling processing on the first cloud server in response to the elastic scaling instruction.
According to an embodiment of the present invention, the first instruction generation module 505 includes:
the first instruction generation unit is used for generating a first elastic telescopic instruction, and the first elastic telescopic instruction is used for indicating the execution end to adjust the number of cloud servers in the cloud service cluster;
The second instruction generating unit is used for generating a second elastic telescopic instruction, and the second elastic telescopic instruction is used for indicating the execution end to adjust the configuration of the first cloud server.
The elastic expansion device shown in fig. 5 may perform the elastic expansion method shown in the embodiment shown in fig. 3, and its implementation principle and technical effects are not repeated. The specific manner in which the individual modules, units, of the elastic expansion device of the above embodiment perform the operations has been described in detail in connection with the embodiments of the method, and will not be described in detail here.
In one possible design, the data processing apparatus and the elastic expansion apparatus provided by the embodiments of the present invention may be implemented as a computing device, as shown in fig. 6, where the computing device may include a storage component 601 and a processing component 602;
the storage component 601 stores one or more computer instructions, where the one or more computer instructions are called by the processing component 602 to execute, so as to implement a data processing method and an elastic scaling method provided by an embodiment of the present invention.
Of course, the computing device may necessarily include other components, such as input/output interfaces, communication components, and the like. The input/output interface provides an interface between the processing component and a peripheral interface module, which may be an output device, an input device, etc. The communication component is configured to facilitate wired or wireless communication between the computing device and other devices, and the like.
The computing device may be a physical device or an elastic computing host provided by the cloud computing platform, and at this time, the computing device may be a cloud server, and the processing component, the storage component, and the like may be a base server resource rented or purchased from the cloud computing platform.
When the computing device is a physical device, the computing device may be implemented as a distributed cluster formed by a plurality of servers or terminal devices, or may be implemented as a single server or a single terminal device.
The embodiment of the invention also provides a computer readable storage medium which stores a computer program, and the computer program can realize the data processing method and the elastic expansion method provided by the embodiment of the invention when being executed by a computer.
The embodiment of the invention also provides a computer program product, which comprises a computer program, wherein the computer program can realize the data processing method and the elastic expansion method provided by the embodiment of the invention when being executed by a computer.
Wherein the processing components of the respective embodiments above may include one or more processors to execute computer instructions to perform all or part of the steps of the methods described above. Of course, the processing component may also be implemented as one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic elements for executing the methods described above.
The storage component is configured to store various types of data to support operation in the device. The memory component may be implemented by any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (14)

1. The data processing method is characterized by being applied to an index acquisition end and comprising the following steps of:
collecting first index data generated by a data source in a first time window;
storing the first index data in a local storage medium;
detecting that the first index data is stored in the local storage medium, and acquiring the first index data from the local storage medium so as to process the first index data in real time;
and processing the first index data to obtain second index data, wherein the second index data is used for triggering an execution end to execute preset operation, and the first index data and the second index data are index values representing the running state of the data source.
2. The method according to claim 1, wherein the method further comprises:
determining whether the first index data meets a release condition or not, wherein the release condition is determined according to the acquisition time of the first index data;
and deleting the first index data from the local storage medium under the condition that the first index data meets a release condition so as to release the storage space of the local storage medium.
3. The method of claim 2, wherein determining whether the first metric data satisfies a release condition comprises:
determining a difference value between the acquisition time and the current time;
comparing the difference value with a preset time threshold value, and determining that the first index data meets a release condition under the condition that the acquisition time is larger than the preset time threshold value, or else, the first index data does not meet the release condition, wherein the preset time threshold value comprises the difference value between the acquisition time and the current time.
4. The method according to claim 1, wherein the method further comprises:
determining whether the second index data meets preset data requirements;
generating a trigger instruction under the condition that the second index data meets the preset data requirement;
and sending the trigger instruction to an execution end corresponding to the data source so that the execution end can respond to the trigger instruction to execute preset operation.
5. The method of claim 1, wherein storing the metric data into a local storage medium comprises:
storing the first index data into a first memory of the index acquisition end;
And under the condition that the storage amount of the first memory exceeds a first storage threshold value, the first index data are transferred from the first memory to a second memory, wherein the storage amount of the first memory is larger than the first memory.
6. The method of claim 5, wherein the first index data carries a time stamp, the time stamp being determined based on a time of acquisition of the first index data;
the obtaining the first index data from the local storage medium includes:
determining whether the first index data is stored in the first memory based on the timestamp;
and under the condition that the first index data is determined to be stored in the first memory, acquiring the first index data from the first memory, otherwise, acquiring the first index data from the second memory.
7. The method of claim 5, wherein the method further comprises:
determining whether the storage capacity of the second memory exceeds a second storage threshold;
and under the condition that the storage amount of the second memory exceeds a second storage threshold value, executing memory release operation on the second memory so as to reduce the storage amount of the second memory.
8. The method of claim 7, wherein the first metric data stored in the second memory carries a timestamp, the timestamp being determined based on a time of acquisition of the first metric data;
the performing the memory release operation on the second memory includes:
acquiring a preset reference time stamp;
determining third index data stored in the second memory and corresponding to a timestamp with a preset time span of the reference timestamp;
and deleting the third index data from the second memory.
9. The method of claim 1, wherein the first index data generated by the acquisition data source comprises:
acquiring a data acquisition request;
responding to the data acquisition request, and determining at least one first index acquisition end belonging to the same cluster with the index acquisition end;
and taking the at least one first index acquisition end as a data source to acquire survival index data generated by the at least one first index acquisition equipment.
10. The method of claim 9, wherein the processing the first metric data comprises:
determining second index data based on survival index data of each first index acquisition end, wherein the second index data is used for representing the survival rate of index equipment in the cluster;
The method further comprises the steps of:
generating an operation and maintenance instruction under the condition that the second index data is lower than a preset survival threshold value;
and sending the operation and maintenance instruction to an instruction execution end so that the execution end responds to the operation and maintenance instruction to execute operation and maintenance operation on the cluster.
11. The elastic telescoping method is characterized by being applied to an index acquisition end and comprising the following steps of:
collecting third index data generated by a first cloud server in a cloud service cluster in a first time window;
storing the third index data in a local storage medium;
detecting that the third index data is stored in the local storage medium, and acquiring the third index data from the local storage medium so as to process the third index data in real time;
processing the third index data to obtain fourth index data, wherein the third index data and the fourth index data are index values representing the running state of the first cloud server;
generating an elastic expansion instruction under the condition that the fourth index data meets the data requirement;
and sending the elastic expansion instruction to an execution end, so that the execution end responds to the elastic expansion instruction to execute elastic expansion processing on the cloud service cluster.
12. The method of claim 11, wherein generating the expansion instruction comprises:
generating a first elastic expansion instruction, wherein the first elastic expansion instruction is used for indicating the execution end to adjust the number of cloud servers in the cloud service cluster;
or alternatively, the process may be performed,
and generating a second elastic expansion instruction, wherein the second elastic expansion instruction is used for indicating the execution end to adjust the configuration of the first cloud server.
13. A computing device comprising a processing component and a storage component;
the storage component stores one or more computer instructions; the one or more computer instructions are configured to be invoked by the processing component to implement a data processing method according to any one of claims 1 to 10, or an elastically scalable method according to claim 11.
14. A computer storage medium, characterized in that a computer program is stored, which, when being executed by a computer, implements the data processing method according to any one of claims 1 to 10 or implements the elastic stretching method according to claim 11.
CN202310202536.0A 2023-02-24 2023-02-24 Data processing, elastic expansion method, computing device and computer storage medium Pending CN116185635A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310202536.0A CN116185635A (en) 2023-02-24 2023-02-24 Data processing, elastic expansion method, computing device and computer storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310202536.0A CN116185635A (en) 2023-02-24 2023-02-24 Data processing, elastic expansion method, computing device and computer storage medium

Publications (1)

Publication Number Publication Date
CN116185635A true CN116185635A (en) 2023-05-30

Family

ID=86446146

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310202536.0A Pending CN116185635A (en) 2023-02-24 2023-02-24 Data processing, elastic expansion method, computing device and computer storage medium

Country Status (1)

Country Link
CN (1) CN116185635A (en)

Similar Documents

Publication Publication Date Title
CN108270618B (en) Alarm determination method, device and alarm system
US10558544B2 (en) Multiple modeling paradigm for predictive analytics
CN111459782B (en) Method and device for monitoring service system, cloud platform system and server
US20110185235A1 (en) Apparatus and method for abnormality detection
WO2016188175A1 (en) Hardware fault analysis system and method
US9176798B2 (en) Computer-readable recording medium, failure prediction device and applicability determination method
US20150106342A1 (en) System and method of detecting cache inconsistencies
CN110727556A (en) BMC health state monitoring method, system, terminal and storage medium
WO2017215557A1 (en) Method and device for collecting performance monitor unit (pmu) events
CN112615742A (en) Method, device, equipment and storage medium for early warning
CN114328102A (en) Equipment state monitoring method, device, equipment and computer readable storage medium
CN112751726A (en) Data processing method and device, electronic equipment and storage medium
CN113190423A (en) Service data monitoring method, device and system
CN113391981A (en) Early warning method for monitoring index and related equipment
CN106951360B (en) Data statistical integrity calculation method and system
CN105471938B (en) Server load management method and device
CN113342608A (en) Method and device for monitoring streaming computing engine task
CN111159009B (en) Pressure testing method and device for log service system
CN114116128B (en) Container instance fault diagnosis method, device, equipment and storage medium
CN111654405A (en) Method, device, equipment and storage medium for fault node of communication link
CN116185635A (en) Data processing, elastic expansion method, computing device and computer storage medium
WO2023159638A1 (en) Battery fault detection system, method and apparatus
CN115525392A (en) Container monitoring method and device, electronic equipment and storage medium
CN106686082B (en) Storage resource adjusting method and management node
CN115543671A (en) Data analysis method, device, equipment, storage medium and program product

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination