CN114385463A - Data acquisition method and device and electronic equipment - Google Patents

Abstract

The embodiments of the present application disclose a data collection method, an apparatus, and an electronic device. Wherein, the method includes: in response to the collection instruction, acquiring the collection data collected in the current collection period as the data to be processed; calculating the similarity between the data to be processed and the historical collection data collected in the historical collection period; if the similarity satisfies the specified Threshold condition, to store the data to be processed. Through the above method, the similarity between the calculated data to be processed and the historical collection data collected in the historical collection period can be compared with the specified threshold condition, and when the similarity satisfies the specified threshold condition, the to-be-processed data can be processed again. The data is stored, so that the acquired collected data will be screened according to the specified threshold conditions to obtain the collected data that meets the requirements (specified threshold conditions) for storage, so that it is not necessary to directly perform each acquisition of the collected data. storage, saving storage space.

Description

Data acquisition method, device and electronic device

technical field

The present application relates to the field of computer technology, and more particularly, to a data collection method, apparatus, and electronic device.

Background technique

In order to know the running state of the device or program, data during the running process of the device or program can be collected, and then it can be determined whether the device or program is faulty according to the collected data. However, there is still the problem of wasting storage space in related data collection methods.

SUMMARY OF THE INVENTION

In view of the above problems, the present application proposes a data acquisition method, device and electronic device to improve the above problems.

In a first aspect, the present application provides a data collection method, the method includes: in response to a collection instruction, obtaining collection data collected in a current collection period as data to be processed; The similarity of the collected historically collected data; if the similarity meets the specified threshold condition, the data to be processed is stored.

Optionally, the calculating the similarity between the collected data and the historical collection data collected in the historical collection period includes: calculating the data to be processed and the data collected in the historical collection period based on the type of the data to be processed. Similarity of historically collected data.

Optionally, calculating the similarity between the data to be processed and the historically collected data collected in the historical collection period based on the type of the data to be processed includes: if the data to be processed is cumulative data, based on the data to be processed. Calculate similarity between the data to be processed and the rate of change of the historical collection data collected in the historical collection period; if the data to be processed is non-cumulative data, based on the data to be processed and the historical collection collected in the historical collection period Data calculation similarity.

Wherein, optionally, calculating the similarity based on the data to be processed and the rate of change of the historical collection data collected in the historical collection period includes: if the data to be processed is single-dimensional data, acquiring multiple historical collections Obtaining a plurality of historical collection data from the historical collection data collected in the respective periods; obtaining a plurality of pairs of data from the data to be processed and the plurality of historical collection data, and the data in each pair of data has adjacent collection periods; In each pair of data, the difference between the data after the corresponding collection period and the data before the corresponding collection period is used as the reference difference value to obtain the reference difference value corresponding to each pair of data; the reference difference value corresponding to each pair of data and each pair of data The corresponding change rate of each pair of data is obtained by comparing the data corresponding to the previous collection period in Any of distance, cosine distance, Pearson correlation coefficient, modified cosine distance, Hamming distance, Manhattan distance.

Optionally, calculating the similarity based on the data to be processed and the rate of change of the historically collected data collected in the historical collection period includes: if the data to be processed is multi-dimensional data, obtaining the data collected in multiple historical collection periods respectively. obtain a plurality of historical collection data; obtain the data to be processed and the data of each dimension position in the plurality of historical collection data; divide the data of each dimension position into multiple groups based on the corresponding dimension position , obtain multiple sets of data, wherein the dimensional positions corresponding to the same set of data are the same; obtain multiple pairs of data in each set of data, and the data in each pair of data have adjacent collection periods; obtain the corresponding collection periods in each pair of data The difference between the following data and the data corresponding to the previous acquisition period is taken as the reference difference value, so as to obtain the reference difference value corresponding to each pair of data in each group of data; Comparing the data with the previous collection period, the rate of change corresponding to each pair of data is obtained; based on the sampling period of the included data, the multiple pairs of data are divided into multiple sets, wherein the corresponding pairs of data in the same set are Some collection cycles are the same; corresponding multi-dimensional data are generated based on the rate of change corresponding to each data in each set, and multiple multi-dimensional data are obtained, wherein the rate of change corresponding to each pair of data is in the middle dimension of the corresponding generated multi-dimensional data. The dimensional position of the data in the pair of data is the same in the data to be processed or the historically collected data; similarity is calculated based on a second similarity algorithm and the plurality of multidimensional data, and the second similarity algorithm includes Any of Euclidean distance, cosine distance, Pearson correlation coefficient, modified cosine distance, Hamming distance, Manhattan distance.

Optionally, calculating the similarity based on the data to be processed and the historical collection data collected in the historical collection period includes: if the data to be processed is single-dimensional data, acquiring the histories collected by each of the multiple historical collection periods. Collect data to obtain a plurality of historically collected data; calculate similarity based on a first similarity algorithm, the data to be processed, and the plurality of historically collected data, where the first similarity algorithm includes standard deviation, Euclidean distance, and cosine distance , Pearson correlation coefficient, modified cosine distance, Hamming distance, Manhattan distance.

Optionally, the calculating similarity based on the data to be processed and the historical collection data collected in the historical collection period includes: if the data to be processed is multi-dimensional data, acquiring the historical collection data collected by each of the multiple historical collection periods. data to obtain a plurality of historical collection data; calculate similarity based on the second similarity algorithm, the data to be processed and the plurality of historical collection data, the second similarity algorithm includes Euclidean distance, cosine distance, Pearson correlation Coefficient, Modified Cosine Distance, Hamming Distance, Manhattan Distance.

Therefore, when the data to be processed is cumulative data, the similarity can be calculated based on the data to be processed and the rate of change of multiple historical collection data; when the data to be processed is non-cumulative data, the similarity can be calculated based on the data to be processed and multiple historical data. The method of collecting data to calculate the similarity makes it possible to better identify whether the data to be processed has relatively different data characteristics based on the difference of the data types to be processed. High-value data features. In addition, the cumulative/non-cumulative data is further divided into single-dimensional data and multi-dimensional data, and then different similarity algorithms are used to calculate the similarity according to the different dimensions of the data, thereby improving the applicability of the data collection method proposed in this application. Scalability.

Optionally, the calculating the similarity between the data to be processed and the historical collection data collected in the historical collection period includes: calculating the data to be processed based on the dimension of the data to be processed and the data collected in the historical collection period. similarity of historically collected data.

Optionally, if the similarity does not meet the specified threshold condition and the collection time of the data to be processed matches the persistent storage period, the data to be processed is stored.

Data is collected and stored based on a persistent storage period, wherein the persistent storage period is greater than the sampling period.

Optionally, a persistent storage period is determined based on the type of the data to be processed.

Through the above method, the data to be processed can be stored when the collection time matches the persistent storage period or the similarity can meet the specified threshold condition, because the data to be processed will be stored when the persistent storage period arrives, and whether the similarity can meet the specified threshold The threshold condition is random, so that the data to be processed can be persistently stored at a variable frequency, so that both the data to be processed with high value and the conventional data to be processed can be stored persistently, which saves storage space and improves the the security and stability of the device.

Optionally, the specified threshold condition includes that the similarity is smaller than a first similarity threshold, or the absolute value of the difference between the similarity and the similarity corresponding to the previous collection period is smaller than a second similarity threshold.

Through the above method, when the problem caused by the equipment failure represented by the data to be processed is serious (for example: directly causing the equipment to stop working), the specified threshold condition corresponding to the data to be processed can be set as: the similarity of the data to be processed is the same as The absolute value of the difference of the similarity corresponding to the previous collection period is smaller than the second similarity threshold, so that the equipment failure can be found in time; when the problem caused by the equipment failure represented by the data to be processed is relatively minor (for example: a certain The function fails, but other tasks can also be performed), the specified threshold condition corresponding to the data to be processed can be set as: the similarity of the data to be processed is less than the first similarity threshold, so that the operating status of the device can be quickly analyzed. Therefore, the specified threshold condition can be determined based on actual requirements, thereby improving the flexibility of the similarity judgment method.

In a second aspect, the present application provides a data collection method and apparatus, which runs on an electronic device, and the apparatus includes: a to-be-processed data acquisition unit, configured to, in response to a collection instruction, acquire the collection data collected in the current collection cycle as the waiting-to-process data processing data; a similarity calculating unit for calculating the similarity between the data to be processed and the historical collection data collected in the historical collection period; a storage unit for calculating the similarity of the data to be Process data for storage.

In a third aspect, the present application provides an electronic device comprising one or more processors and a memory; one or more programs are stored in the memory and configured to be executed by the one or more processors, The one or more programs are configured to perform the methods described above.

In a fourth aspect, the present application provides a computer-readable storage medium, where a program code is stored in the computer-readable storage medium, wherein the above-mentioned method is executed when the program code is executed.

In a data acquisition method, device, electronic device and storage medium provided by the present application, after acquiring the acquisition data acquired in the current acquisition period in response to the acquisition instruction, the acquisition data acquired in the current acquisition period is used as the data to be processed, The similarity between the data to be processed and the historical collection data collected in the historical collection period is calculated, and if the similarity meets a specified threshold condition, the data to be processed is stored. Thus, by the above method, the similarity between the calculated data to be processed and the historical collection data collected in the historical collection period can be compared with the specified threshold condition, and if the similarity satisfies the specified threshold condition, then treat it again. The processed data is stored, so that the acquired collected data will be screened according to the specified threshold conditions, so as to obtain the collected data that meets the requirements (specified threshold conditions) for storage, so that it is not necessary to directly store the collected data obtained each time. Save storage space.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1 shows a schematic diagram of an application scenario of a data collection method proposed by the present application;

FIG. 2 shows a flowchart of a data collection method proposed by an embodiment of the present application;

FIG. 3 shows a flowchart of a data collection method proposed by another embodiment of the present application;

FIG. 4 shows a flowchart of an embodiment of S220 in FIG. 2 of the present application;

FIG. 5 shows a flowchart of another embodiment of S220 in FIG. 2 of the present application;

6 shows a schematic diagram of a method for calculating the rate of change of multidimensional data proposed by the present application;

FIG. 7 shows a flowchart of an embodiment of S230 in FIG. 2 of the present application;

FIG. 8 shows a flowchart of another embodiment of S230 in FIG. 2 of the present application;

FIG. 9 shows a flowchart of a data collection method proposed by still another embodiment of the present application;

FIG. 10 shows a flowchart of a data collection method proposed by another embodiment of the present application;

FIG. 11 shows a schematic diagram of the flow of a data acquisition method proposed by the present application;

FIG. 12 shows a structural block diagram of a data acquisition device proposed by an embodiment of the present application;

FIG. 13 shows a structural block diagram of an electronic device proposed by the present application;

FIG. 14 is a storage unit for storing or carrying a program code for implementing a parameter acquisition method according to an embodiment of the present application according to an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

In order to know the running state of the device or program, data during the running process of the device or program can be collected, and then it can be determined whether the device or program is faulty according to the collected data. For example, data related to the read/write rate of the disk I/O port may be collected, so as to determine whether the disk has a data read/write failure according to the collected related data.

The inventor found in the related research that the related data collection methods still have the problem of wasting storage space or failing to collect high-value data. For example, in the method of collecting the full amount of device data, a large amount of data needs to be collected, thus occupying a large amount of local storage space, and most of the stored data is repeated low-value data.

Therefore, the inventor proposes a data acquisition method, device and electronic device in the present application. After acquiring the acquisition data acquired in the current acquisition period in response to the acquisition instruction, the acquisition data acquired in the current acquisition period is used as the pending data collection. Processing data, calculating the similarity between the data to be processed and the historical collection data collected in the historical collection period, and storing the data to be processed if the similarity meets a specified threshold condition. Thus, by the above method, the similarity between the calculated data to be processed and the historical collection data collected in the historical collection period can be compared with the specified threshold condition, and if the similarity satisfies the specified threshold condition, then treat it again. The processed data is stored, so that the acquired collected data will be screened according to the specified threshold conditions, so as to obtain the collected data that meets the requirements (specified threshold conditions) for storage, so that it is not necessary to directly store the collected data obtained each time. Save storage space.

Scenario: multiple devices (for example), network, cloud server, multiple devices can collect their respective operating data and upload it to the cloud server (devices can upload responsively or actively), and the cloud server executes the embodiments of this application Provided data collection methods.

In order to better understand the solutions of the embodiments of the present application, an application scenario involved in the embodiments of the present application is first introduced below.

Referring to FIG. 1 , the scenario shown in FIG. 1 includes multiple devices, gateways, cloud platforms, and user terminals. Among them, the device can receive the control command issued by the cloud and the gateway, and report the information corresponding to the control command through the gateway after responding to the control command (for example, the device can report the information corresponding to the control command through the gateway after responding to the data collection command issued by the cloud or the gateway) The operating data corresponding to the device), and the corresponding operating data of the device can also be uploaded through the gateway, so that the gateway and the cloud platform can confirm whether the device is faulty; the gateway can be responsible for processing the uplink information of the IoT device and the downlink commands from the cloud; The platform can be used to execute the data collection method provided by the embodiment of the present application, and can also provide an interface for calling the application program in the user terminal device, so as to realize the uplink and downlink of information, so that when the data collected by the cloud platform indicates that the device may fail When the device is activated, prompt information can be sent to the application program of the user terminal; the application program of the user terminal device can send control commands to the device and read the operation data reported by the device by calling the interface provided by the cloud platform, so that the operation data can be carried out based on the operation data. Troubleshooting and maintenance of IoT devices.

The embodiments involved in the present application will be described below with reference to the accompanying drawings.

Please refer to FIG. 2, a data collection method provided by the present application, the method includes:

S110: In response to the acquisition instruction, acquire the acquisition data acquired in the current acquisition cycle as the data to be processed.

The data to be processed may be data representing the state of a device (eg, a disk, etc.). Exemplarily, when the device is a disk, the data to be processed may be the number of reads per second (r/s), the number of writes per second (w/s), the wear leveling count (Wear Leveling Count), etc., where , r/s and w/s can indicate whether the read and write data functions of the disk I/O port are normal, and the wear leveling count can indicate whether the disk storage function is normal.

In one way, when the time corresponding to the collection cycle arrives, the control unit of the electronic device may issue a collection instruction to the data collection unit, and the collection instruction may include an identifier of the data to be collected. After the data collection unit responds to the collection instruction, it may The identifier of the data to be collected performs a data collection operation, so that the collected data collected in the current collection period is regarded as the data to be processed. Exemplarily, the identifier of the data to be collected may be the number of readings completed per second (r/s), the sampling period may be t, and every time t, the control unit of the electronic device may issue a collection instruction to the data collection unit, and the data collection After the unit responds to the collection instruction, it can perform a data collection operation according to the identifier of the data to be collected, so as to use the number of completed reads per second (r/s) collected in the current collection cycle as the data to be processed.

It should be noted that a collection instruction may include one or more identifiers of data to be collected, and the number of identifiers of data to be collected included in the collection instruction may be determined based on the task requirements of the electronic device.

Optionally, as shown in Table 1, the identifier of the data may be encoded, and each encoding may correspond to a unique identifier.

Table 1

coding identification of data 001 Reads completed per second 002 Writes completed per second ... ...

S120: Calculate the similarity between the data to be processed and the historical collection data collected in the historical collection period.

The similarity represents the degree of similarity between the data to be processed and the historically collected data, and the greater the similarity, the greater the corresponding degree of similarity. As one way, the similarity between the data to be processed and the historically collected data collected in the historical collection period may be calculated based on the type of the data to be processed.

In this embodiment of the present application, the data to be processed can be classified into cumulative data and non-cumulative data. Among them, the cumulative data is data that can produce cumulative effects with historical data, that is, the cumulative data can be data related to the historical state and current state of the device. For example: when the device is a disk, the accumulated data can be the relocation sector count (Reallocated_Sector_Ct), the terminal check error (End-to-End error), the relocation event count (Reallocated_Event_Count), the remaining life percentage (Remain Life Percentage), Available reserved space (Available Reserved Space), percentage of available or remaining available reserved blocks (Reserved BlockCount), total number of program failures (Program Fail Count), remaining percentage of allowed deletion failures (Erase FailCount), wear leveling count (Wear Leveling Count), Total LBAs Write Count (Total LBAs Written), Uncorrectable Sector Count On Line, etc.

Non-cumulative data can be random or burst data, that is to say, non-cumulative data can be data only related to the current state of the device, for example: when the device is a disk, the non-cumulative data can be every The number of merged read requests to the electronic device per second (rrqm/s), the number of merged write requests to the device per second (wrqm/s), the number of reads completed per second (r/s), the number of writes completed per second The number of times (w/s), the amount of data read per second (rkB/s, in kB), the amount of data written per second (wkB/s, in kB), and the average amount of data per I/O operation (avgrq- sz, the number of sectors in units), the average queue length of I/O requests waiting to be processed (avgqu-sz), the average waiting time of each I/O request (await, including waiting time and processing time, in milliseconds), I /O ratio of times when the queue is not empty (%util), etc.

It should be noted that the collection period corresponding to the cumulative data and the non-cumulative data can be different. Since the non-cumulative data is more random and bursty than the cumulative data, the acquisition period corresponding to the The period can be shorter than the acquisition period corresponding to the accumulated data. Exemplarily, the number of times the read requests to the electronic device are merged per second (rrqm/s) is non-cumulative data, the collection period corresponding to rrqm/s can be set to 30s, and the remaining life percentage (Remain Life Percentage) is cumulative. data, the acquisition period corresponding to the percentage of remaining life can be set to 24h. By setting different collection cycles for different types of data, it is possible to analyze the data in time to find out whether the status of the hard disk has changed.

Furthermore, it should be noted that the data type of the operation data to be collected is preset, and the device can determine the type of the collected data based on the currently executed task.

As another way, the similarity between the data to be processed and the historically collected data collected in the historical collection period may be calculated based on the dimensions of the data to be processed.

In this embodiment of the present application, the data to be processed may also be classified into single-dimensional data and multi-dimensional data according to the number of dimensions. Among them, the single-dimensional data is the data containing a kind of identification, for example: the single-dimensional data can be the number of reads per second (r/s); the single-dimensional data can also be the number of writes per second (w/s), etc. . Multidimensional data is data that contains two or more identifiers. For example, multidimensional data may include Remain Life Percentage, Wear Leveling Count, and Total LBAs Written. The data can be used to calculate disk life.

It should be noted that the device can pre-determine whether the data to be collected is single-dimensional data or multi-dimensional data based on different tasks, and the single-dimensional data can be transmitted in the form of a single value in the device, while the multi-dimensional data can be It is transmitted in the form of an array, so that the device can determine whether the acquired data to be processed (collected data) is single-dimensional data or multi-dimensional data according to the data format and the currently executed task.

Furthermore, it should be noted that the types and dimensions of the data to be processed can overlap, that is, when the data to be processed is cumulative/non-cumulative data, the data to be processed can be further divided into single-dimensional cumulative data. /Non-cumulative data and multi-dimensional cumulative/non-cumulative data; when the data to be processed is single/multi-dimensional data, the data to be processed can be further divided into single/multi-dimensional cumulative data and single/multi-dimensional non-cumulative data.

S130: If the similarity meets a specified threshold condition, store the data to be processed.

Wherein, in this embodiment of the present application, the specified threshold condition may be a condition for characterizing high-value data. Therefore, if the similarity satisfies the specified threshold condition, the data to be processed may be determined to be high-value data. In this embodiment of the present application, the high-value data may be data representing a possible failure of the device. For example, when the data to be processed is the number of completed reads per second, the values corresponding to the number of completed reads per second collected in the history of the disk are all is larger, and the value of the pending data is 0, indicating that the disk may have a read data failure.

As one approach, specifying the threshold condition may include that the similarity is less than a first similarity threshold. In this way, when the similarity corresponding to the data to be processed is smaller than the first similarity threshold, it can be indicated that the data to be processed is singular data and may contain information related to the state of the equipment, which has a high value, that is, There is a big difference between the data to be processed and the historically collected data. At this time, the status of the device may change suddenly (for example, the I/O read/write rate of the disk changes from fast to slow, the disk suddenly has a slow card failure, etc.), and the data to be processed can be For persistent storage, that is, stored in the database.

Furthermore, in the embodiment of the present application, as another way, the specified threshold condition includes: the absolute value of the difference between the similarity and the similarity corresponding to the previous collection period is smaller than the second similarity threshold. Exemplarily, it is assumed that the similarity corresponding to the data to be processed is A, and the similarity corresponding to the previous acquisition period that is the same as the data identifier of the data to be processed is B. If |A-B|<the second similarity threshold, it indicates that the data to be processed is The data may be singular data, and the to-be-processed data can be persistently stored, that is, stored in the database.

The above two specified threshold conditions can be used as the basis for whether the data to be processed is stored. The first condition (similarity is less than the first similarity threshold) is simple to calculate; (the absolute value of the difference of the properties is smaller than the second similarity threshold) is more sensitive to the similarity change of two adjacent acquisition periods, and the sudden change of the device state can be found more quickly. Through the above method, when the problem caused by the equipment failure represented by the data to be processed is serious (for example: directly causing the equipment to stop working), the specified threshold condition corresponding to the data to be processed can be set as: the similarity of the data to be processed is the same as The absolute value of the difference of the similarity corresponding to the previous collection period is smaller than the second similarity threshold, so that the equipment failure can be found in time; when the problem caused by the equipment failure represented by the data to be processed is relatively minor (for example: a certain The function fails, but other tasks can also be performed), the specified threshold condition corresponding to the data to be processed can be set as: the similarity of the data to be processed is less than the first similarity threshold, so that the operating status of the device can be quickly analyzed. Therefore, the specified threshold condition can be determined based on actual requirements, thereby improving the flexibility of the similarity judgment method.

It should be noted that the first similarity threshold and the second similarity threshold may be determined according to factors such as the type of data to be processed, experience, and the like. For example: because non-cumulative data has randomness and mutation, cumulative data is more stable than non-cumulative data (that is, the value of cumulative data changes slowly), so the similarity threshold corresponding to cumulative data can be smaller than that of non-cumulative data. Similarity threshold. For another example: because the second similarity threshold is compared with the absolute value of the similarity difference between adjacent periods, and the similarity of adjacent periods changes little, the second similarity threshold may be smaller than the first similarity threshold.

In a data collection method provided by this embodiment, after acquiring the collection data collected in the current collection period in response to the collection instruction, the collection data collected in the current collection period is used as the data to be processed, and the difference between the data to be processed and the data to be processed is calculated. For the similarity of the historically collected data collected in the historical collection period, if the similarity meets the specified threshold condition, the data to be processed is stored. Through the above method, the similarity between the calculated data to be processed and the historical collection data collected in the historical collection period can be compared with the specified threshold condition, and when the similarity satisfies the specified threshold condition, the to-be-processed data can be processed again. The data is stored, so that the acquired collected data will be screened according to the specified threshold conditions to obtain the collected data that meets the requirements (specified threshold conditions) for storage, so that it is not necessary to directly perform each acquisition of the collected data. storage, saving storage space.

Please refer to FIG. 3, a data collection method provided by the present application, the method includes:

S210: In response to the acquisition instruction, acquire the acquisition data acquired in the current acquisition cycle as the data to be processed.

S220: If the data to be processed is cumulative data, calculate similarity based on the data to be processed and the rate of change of the historically collected data collected in the historical collection period.

Wherein, as one way, as shown in Figure 4, if the data to be processed is cumulative data, the similarity is calculated based on the data to be processed and the rate of change of the historical collection data collected in the historical collection period, including:

S2201: If the data to be processed is single-dimensional data, obtain the historical collection data collected by each of a plurality of historical collection periods, and obtain a plurality of historical collection data.

Among them, the number of historically collected data can be determined according to factors such as the task of analyzing the device status and the collection period. According to the actual situation, it can generally be set to obtain 7, 14, 20 or 30 historically collected data. Exemplarily, if the task of analyzing the device status is to check the percentage of the remaining life of the disk and the collection period is 24 hours, then 30 historical collection data can be obtained; if the task of analyzing the device status is to check the time ratio of the disk I/O queue not empty , the collection period is 30s, then 20 historical collection data can be obtained.

S2202: Acquire multiple pairs of data from the data to be processed and the plurality of historical collection data, and the respective collection periods corresponding to the data in each pair of data are adjacent to each other.

Among them, exemplarily, assuming that the acquired historical collection data are in the order of time: A, B, C, D, E, F, G, and the data to be processed is H, the following data pairs can be : AB, BC, CD, DE, EF, FG, GH, where the collection period of each pair of data is adjacent.

S2203: Acquire the difference between the data corresponding to the later acquisition period and the data corresponding to the previous acquisition period in each pair of data as a reference difference value, so as to obtain a reference difference value corresponding to each pair of data.

Wherein, exemplarily, assuming that the data pairs are: AB, BC, CD, DE, EF, FG, GH, the reference difference corresponding to each pair of data can be obtained through BA, CB, DC, ED, FE, GF, HG : X ₁ (corresponding to BA), X ₂ (corresponding to CB), X ₃ (corresponding to DC), X ₄ (corresponding to ED), X ₅ (corresponding to FE), X ₆ (corresponding to GF), X ₇ (corresponds to HG).

S2204: Compare the reference difference value corresponding to each pair of data with the data corresponding to the previous collection period in each pair of data, to obtain a rate of change corresponding to each pair of data.

Wherein, for example, it is assumed that the data pairs are: AB, BC, CD, DE, EF, FG, GH, and the reference difference corresponding to each pair of data is: X ₁ , X ₂ , X ₃ , X ₄ , X ₅ , X ₆ , X ₇ , the corresponding changes of each pair of data can be obtained through X ₁ /A, X ₂ /B, X ₃ /C, X ₄ /D, X ₅ /E, X ₆ /F, X ₇ /G Rate: _Y1 (corresponds to X1/ _A ), _{Y2 (} corresponds to X2/B), _Y3 (corresponds to _X3 /C), Y4 (corresponds to _X4 /D), Y5 ₍ corresponds to _X4 /D) X ₅ /E), Y ₆ (corresponding to X ₆ /F), Y ₇ (corresponding to X ₇ /G).

S2205: Calculate similarity based on a first similarity algorithm and the rate of change corresponding to each pair of data, where the first similarity algorithm includes standard deviation, Euclidean distance, cosine distance, Pearson correlation coefficient, modified cosine distance, Hamming distance, Any of the Manhattan distances.

Wherein, exemplarily, when calculating the similarity using the standard deviation in the first similarity algorithm, it is assumed that the corresponding rate of change of each pair of data is: Y ₁ , Y ₂ , Y ₃ , Y ₄ , Y ₅ , Y ₆ , Y ₇ , then you can first obtain the average value M of the rate of change by (Y ₁ +Y ₂ +...+Y ₇ )/7, and then use the standard deviation formula: sqrt(((Y ₁ -M)^2+( Y ₂ -M)^2+...+(Y ₇ -M)^2)/7) can obtain that the standard deviation corresponding to the data to be processed is N, because the smaller the value of N is, the smaller the value of N indicates that the data to be processed and the historical data The more similar, the larger the value of N, the less similar the data to be processed and the historical data. Therefore, in order not to conflict with the specified threshold condition (the smaller the similarity, the less similar the data to be processed and the historical data are), the similarity of the data to be processed is similar. The sex can be 1/(N+1). Similarly, when other first similarity algorithms (eg, Euclidean distance, etc.) contradict the specified threshold condition, similar processing can also be performed.

Wherein, exemplarily, when calculating the similarity using the cosine distance in the first similarity algorithm, it is assumed that the corresponding change rate of each pair of data is: Y ₁ , Y ₂ , Y ₃ , Y ₄ , Y ₅ , Y ₆ , Y ₇ , then Y ₆ and Y ₇ can be substituted into the cosine distance formula to calculate the similarity corresponding to the data to be processed; Y ₁ and Y ₂ , Y ₂ and Y ₃ , Y ₃ and Y ₄ , Y ₄ and Y can also be respectively ₅ , Y ₅ and Y ₆ , Y ₆ and Y ₇ are substituted into the cosine distance formula to obtain a plurality of cosine distances, and then the standard deviation operation is performed on the plurality of cosine distances to obtain the corresponding similarity of the data to be processed.

As another method, as shown in FIG. 5 , if the data to be processed is cumulative data, the similarity is calculated based on the data to be processed and the rate of change of the historically collected data collected during the historical collection period, including:

S2211 : If the data to be processed is multi-dimensional data, acquire the historical collection data collected by each of a plurality of historical collection periods to obtain a plurality of historical collection data.

S2212: Acquire the data to be processed and the data of each dimension position in the plurality of historically collected data.

Wherein, exemplarily, as shown in FIG. 6 , the acquired historical collection data in chronological order are: {A1, B1, C1, D1}, {A2, B2, C2, D2}, {A3 , B3, C3, D3}, the data to be processed is {A4, B4, C4, D4}, where A, B, C, and D respectively represent the four dimension positions of the multidimensional data, for example, the position of the A dimension of the data to be processed The data is A4, the data in the B dimension is B4, the data in the C dimension is C4, and the data in the D dimension is D4.

S2213: Divide the data of each dimension position into multiple groups based on the corresponding dimension positions, to obtain multiple groups of data, wherein the dimension positions corresponding to the same group of data are the same.

Wherein, exemplarily, as shown in FIG. 6 , the acquired historical collection data in chronological order are: {A1, B1, C1, D1}, {A2, B2, C2, D2}, {A3 , B3, C3, D3}, the data to be processed is {A4, B4, C4, D4}, then the multiple sets of data that can be obtained are: {A1, A2, A3, A4}, {B1, B2, B3, B4} , {C1, C2, C3, C4}, {D1, D2, D3, D4}.

S2214: Acquire multiple pairs of data in each group of data, and the respective corresponding collection periods of the data in each pair of data are adjacent.

Wherein, exemplarily, as shown in FIG. 6 , the following three pairs of data can be obtained from the set of data {A1, A2, A3, A4}: A1A2, A2A3, and A3A4.

S2215: Acquire the difference between the data corresponding to the later acquisition period and the data corresponding to the previous acquisition period in each pair of data as a reference difference value, so as to obtain a reference difference value corresponding to each pair of data in each group of data.

Among them, exemplarily, as shown in Figure 6, in the set of data {A1, A2, A3, A4}, there are data pairs: A1A2, A2A3, A3A4, which can be passed through A2-A1, A3-A2, A4-A3 The obtained reference differences are: X1 (corresponding to A2-A1), X2 (corresponding to A3-A2), X3 (corresponding to A4-A3).

S2216: Compare the reference difference value corresponding to each pair of data with the data corresponding to the previous collection period in each pair of data, to obtain a rate of change corresponding to each pair of data.

Wherein, exemplarily, as shown in FIG. 6 , in the set of data {A1, A2, A3, A4}, the reference difference values corresponding to the data pairs A1A2, A2A3, and A3A4 are: X1, X2, X3, which can be passed through X1/A1, X2/A2, X3/A3 get the corresponding rate of change of each pair of data: Y _A1 (corresponding to X1/A1), Y _A2 (corresponding to X2/A2), Y _A3 (corresponding to X3/A3) .

S2217: Divide the multiple pairs of data into multiple sets based on the sampling periods of the included data, wherein the corresponding collection periods in each pair of data in the same set are the same.

Wherein, exemplarily, as shown in FIG. 6 , the data pairs in group A are: A1A2, A2A3, and A3A4, the data pairs in group B are: B1B2, B2B3, and B3B4, the data pairs in group C are: C1C2, C2C3, and C3C4, and the data pairs in group D are: C1C2, C2C3, and C3C4. The data pairs are: D1D2, D2D3, D3D4, and can be divided into the following sets: {A1A2, B1B2, C1C2, D1D2}, {A2A3, B2B3, C2C3, D2D3}, {A3A4, B3B4, C3C4, D3D4}, where each Each pair of data in each set includes the same two sampling periods.

S2218: Generate corresponding multi-dimensional data based on the rate of change corresponding to each data in each set, and obtain a plurality of multi-dimensional data, wherein the rate of change corresponding to each pair of data is in the middle dimension position of the corresponding generated multi-dimensional data and the pair of data The dimension positions of the data in the data to be processed or the historically collected data are the same.

Wherein, exemplarily, as shown in FIG. 6 , the rate of change of each pair of data in the set {A2A3, B2B3, C2C3, D2D3} is sequentially: Y _A2 , Y _B2 , Y _C2 , Y _D2 , and the above rate of change corresponds to Dimension positions A, B, C, D.

S2219: Calculate similarity based on a second similarity algorithm and the plurality of multidimensional data, where the second similarity algorithm includes Euclidean distance, cosine distance, Pearson correlation coefficient, modified cosine distance, Hamming distance, and Manhattan distance any one.

Wherein, exemplary, as shown in FIG. 5 , the multidimensional data generated based on the rate of change of each pair of data in the set {A2A3, B2B3, C2C3, D2D3} are sequentially: Y _A2 , Y _B2 , Y _C2 , Y _D2 , based on The multi-dimensional data generated by the rate of change of each pair of data in the set {A3A4, B3B4, C3C4, D3D4} are: Y _A3 , Y _B3 , Y _C3 , Y _D3 , the above two multi-dimensional data can be regarded as two vectors, and then The similarity corresponding to the data to be processed can be obtained through the redundant chord distance formula.

Optionally, multiple multi-dimensional rate-of-change data can be generated based on the data to be processed and the acquired multi-dimensional historical collection data, for example: Y _A1 , Y _B1 , Y _C1 , Y _D1 , Y _A2 , Y _B2 , Y _C2 , Y _D2 , Y _A3 , Y _B3 , Y _C3 , Y _D3 , etc., multiple cosine distances can be obtained based on two adjacent multi-dimensional rate-of-change data, and the multiple distances can be used as the standard deviation, and then 1/(The value of the standard deviation + 1) In order to obtain the similarity corresponding to the data to be processed.

S230: If the data to be processed is non-cumulative data, calculate similarity based on the data to be processed and the historical collection data collected in the historical collection period.

Wherein, as a way, as shown in FIG. 7, if the data to be processed is non-cumulative data, the similarity is calculated based on the data to be processed and the historical collection data collected in the historical collection period, including:

S231: If the data to be processed is single-dimensional data, acquire historical collection data collected in multiple historical collection periods respectively, and obtain a plurality of historical collection data.

S232: Calculate similarity based on a first similarity algorithm, the data to be processed, and the plurality of historically collected data, where the first similarity algorithm includes standard deviation, Euclidean distance, cosine distance, Pearson correlation coefficient, modified cosine Any of distance, Hamming distance, Manhattan distance.

Wherein, exemplarily, when using the standard deviation in the first similarity algorithm to calculate the similarity, it is assumed that the acquired historically collected data are: A, B, C, D, E, F in chronological order. , G, the data to be processed is H, then you can first get the average value M by (A+B+...+H)/8, and then use the standard deviation formula: sqrt(((A-M)^2+(B-M)^ 2+...+(H-M)^2)/8), the standard deviation corresponding to the data to be processed is N, because the smaller the value of N, the more similar the data to be processed is with the historical data, and the larger the value of N, the more similar the data to be processed is to the historical data. The data to be processed is less similar to the historical data, so in order not to contradict the specified threshold condition, the similarity of the data to be processed may be 1/(N+1). Similarly, when other first similarity algorithms (eg, Euclidean distance, etc.) contradict the specified threshold condition, similar processing can also be performed.

As another way, as shown in FIG. 8 , if the data to be processed is non-cumulative data, calculating similarity based on the data to be processed and the historical collection data collected in the historical collection period, including:

S236: If the data to be processed is multi-dimensional data, acquire the historical collection data collected by each of a plurality of historical collection periods to obtain a plurality of historical collection data.

S237: Calculate similarity based on a second similarity algorithm, the data to be processed, and the plurality of historically collected data, where the second similarity algorithm includes Euclidean distance, cosine distance, Pearson correlation coefficient, modified cosine distance, Chinese Any of the Ming distance and Manhattan distance.

Wherein, for example, the acquired historical collection data in chronological order are: {A1, B1, C1, D1}, {A2, B2, C2, D2}, {A3, B3, C3, D3 }, the data to be processed is {A4, B4, C4, D4}, when the second similarity algorithm is cosine distance, you can calculate the difference between {A4, B4, C4, D4} and {A3, B3, C3, D3} The cosine distance gives the similarity of the data to be processed.

Optionally, if multiple cosine distances can be obtained based on the data to be processed and the acquired multi-dimensional historical collection data, the multiple cosine distances can be used as the standard deviation, and then 1/(the value of the standard deviation + 1) can be used to obtain the data to be processed. corresponding similarity.

It should be noted that when the value obtained by the second similarity algorithm is larger, it indicates that the data to be processed is less similar to the historically collected data, and the data to be processed can be obtained by 1/(value of the second similarity algorithm+1) The corresponding similarity to compare the similarity to the specified threshold condition.

S240: If the similarity meets a specified threshold condition, store the data to be processed.

In a data collection method provided by this embodiment, in the above manner, the similarity between the calculated to-be-processed data and the historically collected data collected in the historical collection period can be compared with a specified threshold condition, and in the similarity When the specified threshold conditions are met, the data to be processed will be stored, so that the acquired collected data will be screened according to the specified threshold conditions to obtain the collected data that meets the requirements (specified threshold conditions) for storage, and no need The collected data obtained each time is directly stored, which saves storage space. Moreover, in this embodiment, when the data to be processed is cumulative data, the similarity can be calculated based on the data to be processed and the rate of change of a plurality of historically collected data; when the data to be processed is non-cumulative data, the similarity can be calculated based on the data to be processed Similarity is calculated between processed data and multiple historically collected data. Through the above method, since there are differences in the data characteristics of different data types, different data types are used for similarity calculation based on the different data types to be processed. Identify whether the data to be processed has high-value data characteristics. In addition, the cumulative/non-cumulative data is further divided into single-dimensional data and multi-dimensional data, and then different similarity algorithms are used to calculate the similarity according to the different dimensions of the data, thereby improving the applicability of the data collection method proposed in this application. Scalability.

Please refer to FIG. 9, a data collection method provided by the present application, the method includes:

S310: In response to the acquisition instruction, acquire the acquisition data acquired in the current acquisition cycle as the data to be processed.

S320: Calculate the similarity between the data to be processed and the historical collection data collected in the historical collection period.

S330: If the similarity meets a specified threshold condition, store the data to be processed.

S340: If the similarity does not meet the specified threshold condition and the collection time of the data to be processed matches the persistent storage period, store the data to be processed.

Among them, in the embodiment of the present application, for the consideration of the subsequent maintenance of the device and the security of the device, a persistent storage period can be set to periodically store the data to be processed, so that more conventional data can be stored (the device is normal runtime data).

As a method, the persistent storage period can be N times the acquisition period (N>1, and N is an integer). When the similarity corresponding to the data to be processed does not meet the specified threshold condition, the acquisition time of the data to be processed can be judged Whether it matches the persistent storage period. If the collection time of the data to be processed matches the persistent storage period, that is to say, the collection time of the data to be processed happens to be the time corresponding to the persistent storage period, the data to be processed can be persisted. storage.

It should be noted that, in this embodiment of the present application, it is also possible to first determine whether the collection time of the data to be processed matches the persistent storage period. If the collection time of the data to be processed matches the persistent storage period, the data to be processed can be stored. Perform persistent storage; if the collection time of the data to be processed does not match the persistent storage period, it can be judged whether the similarity corresponding to the data to be processed satisfies the specified threshold condition, and if the specified threshold condition is met, the data to be processed can be stored. for persistent storage.

Optionally, in order to facilitate management, the same persistent storage period may be set for all data to be processed. Exemplarily, the persistent storage period of all data to be processed may be set to 1 day.

Optionally, in order to save storage space, the persistent storage period may be determined based on the type of data to be processed. Exemplarily, when the data to be processed is cumulative data, the persistent storage period may be set to 7 days; When the processing data is non-cumulative data, the persistent storage period can be set to 1 day.

In a data collection method provided by this embodiment, the above-mentioned method makes it possible to compare the similarity between the calculated data to be processed and the historical collection data collected in the historical collection period with a specified threshold condition, and compare the similarity between the data to be processed and the historical collection data collected in the historical collection period. When the specified threshold conditions are met, the data to be processed will be stored, so that the acquired collected data will be screened according to the specified threshold conditions to obtain the collected data that meets the requirements (specified threshold conditions) for storage, and no need The collected data obtained each time is directly stored, which saves storage space. In addition, in this embodiment, the data to be processed can be stored when the collection time matches the persistent storage period or the similarity can meet the specified threshold condition, because the data to be processed will be stored when the persistent storage period arrives, and similar performance Whether the specified threshold condition is met is random, so that the data to be processed can be persistently stored at a variable frequency, so that both the data to be processed with high value and the conventional data to be processed can be persistently stored, saving storage space. At the same time, it improves the security and stability of the device.

Please refer to FIG. 10, a data collection method provided by this application, the method includes:

S410: In response to the acquisition instruction, acquire the acquisition data acquired in the current acquisition cycle as the data to be processed.

S420: Calculate the similarity between the data to be processed and the historical collection data collected in the historical collection period.

S430: If the similarity meets a specified threshold condition, store the data to be processed.

S440: Collect and store data based on a persistent storage period, wherein the persistent storage period is greater than the sampling period.

In one way, as shown in FIG. 11 , when the time corresponding to the persistent storage period arrives, the control unit of the electronic device may issue a collection instruction to the data collection unit, and the collection instruction may include the identifier of the data to be collected, the data After the collection unit responds to the collection instruction, it can perform a data collection operation according to the identifier of the data to be collected, and persistently store the collection data collected in the current persistent storage period.

Optionally, in order to facilitate management, the same persistent storage period may be set for all data to be processed. Exemplarily, the persistent storage period of all data to be processed may be set to 1 day.

Optionally, in order to save storage space, the persistent storage period may be determined based on the type of data to be processed. Exemplarily, when the data to be processed is cumulative data, the persistent storage period may be set to 7 days; When the processing data is non-cumulative data, the persistent storage period can be set to 1 day.

As another way, as shown in FIG. 11 , when the time corresponding to the sampling period arrives, the control unit of the electronic device may issue a collection instruction to the data collection unit, and the collection instruction may include an identifier of the data to be collected, and the data collection unit responds to After the collection instruction, a data collection operation can be performed according to the identifier of the data to be collected, so that the collected data collected in the current collection cycle is regarded as the data to be processed; and then the similarity corresponding to the data to be processed is compared with the specified threshold condition. If the similarity corresponding to the processed data satisfies the specified threshold condition, the data to be processed can be persistently stored.

In a data collection method provided by this embodiment, the above-mentioned method makes it possible to compare the similarity between the calculated data to be processed and the historical collection data collected in the historical collection period with a specified threshold condition, and compare the similarity between the data to be processed and the historical collection data collected in the historical collection period. When the specified threshold conditions are met, the data to be processed will be stored, so that the acquired collected data will be screened according to the specified threshold conditions to obtain the collected data that meets the requirements (specified threshold conditions) for storage, and no need The collected data obtained each time is directly stored, which saves storage space. . In addition, in this embodiment, the data to be processed can be stored at the time corresponding to the persistent storage period or when the similarity can meet the specified threshold condition. Since the data to be processed must be stored when the persistent storage period arrives, and the similar performance Whether the specified threshold condition is met is random, so that the data to be processed can be persistently stored at a variable frequency, so that both the data to be processed with high value and the conventional data to be processed can be persistently stored, saving storage space. At the same time, it improves the security and stability of the device.

Please refer to FIG. 12, a data collection device 600 provided by the present application, the device 600 includes:

The to-be-processed data acquisition unit 610 is configured to, in response to the acquisition instruction, acquire the acquisition data acquired in the current acquisition cycle as the to-be-processed data.

The similarity calculation unit 620 is configured to calculate the similarity between the data to be processed and the historical collection data collected in the historical collection period.

The storage unit 630 is configured to store the data to be processed if the similarity satisfies a specified threshold condition.

Wherein, as one way, the similarity calculating unit 620 is specifically configured to calculate the similarity between the data to be processed and the historical collection data collected in the historical collection period based on the type of the data to be processed.

As another way, the similarity calculating unit 620 is specifically configured to calculate the similarity based on the data to be processed and the rate of change of the historical collection data collected in the historical collection period if the data to be processed is cumulative data; The data to be processed is non-cumulative data, and similarity is calculated based on the data to be processed and the historical collection data collected in the historical collection period.

Wherein, optionally, the similarity calculation unit 620 is specifically configured to, if the to-be-processed data is single-dimensional data, obtain the historical collection data collected by each of multiple historical collection periods, and obtain a plurality of historical collection data; from the to-be-processed data Obtain multiple pairs of data from the data and the plurality of historical collection data, and the data in each pair of data have adjacent collection periods; obtain the difference between the data corresponding to the later collection period and the data corresponding to the previous collection period in each pair of data. The difference is used as the reference difference to obtain the reference difference corresponding to each pair of data; the reference difference corresponding to each pair of data is compared with the data corresponding to the collection period in each pair of data to obtain the corresponding change rate of each pair of data; The similarity is calculated based on the first similarity algorithm and the rate of change corresponding to each pair of data. The first similarity algorithm includes standard deviation, Euclidean distance, cosine distance, Pearson correlation coefficient, modified cosine distance, Hamming distance, and Manhattan distance. any of the .

Optionally, the similarity calculation unit 620 is specifically configured to, if the data to be processed is multi-dimensional data, obtain historical collection data collected by multiple historical collection periods, and obtain a plurality of historical collection data; obtain the data to be processed and all the data. The data of each dimension position in the plurality of historically collected data; the data of each dimension position is divided into multiple groups based on the corresponding dimension position, and multiple groups of data are obtained, wherein the dimension positions corresponding to the same group of data are the same; For multiple pairs of data in the group data, the corresponding acquisition periods of each pair of data are adjacent to each other; obtain the difference between the data corresponding to the later acquisition period and the data corresponding to the previous acquisition period in each pair of data as the reference difference, with Obtain the reference difference value corresponding to each pair of data in each group of data; compare the reference difference value corresponding to each pair of data with the data corresponding to the previous acquisition period in each pair of data to obtain the corresponding change rate of each pair of data; The sampling period of the included data, divides multiple pairs of data into multiple sets, wherein the data collection period corresponding to each team of data in the same set is the same; based on the rate of change corresponding to each data in each set Corresponding multi-dimensional data, obtain a plurality of multi-dimensional data, wherein, the rate of change corresponding to each pair of data is in the mid-dimensional position of the corresponding generated multi-dimensional data and the data in the pair of data is in the data to be processed or the historically collected data. The dimensional positions of the two are the same; the similarity is calculated based on the second similarity algorithm and the plurality of multidimensional data, the second similarity algorithm includes Euclidean distance, cosine distance, Pearson correlation coefficient, modified cosine distance, Hamming distance, Manhattan distance any of the distances.

Optionally, the similarity calculation unit 620 is specifically configured to obtain the historical collection data collected by each of a plurality of historical collection periods if the data to be processed is single-dimensional data, and obtain a plurality of historical collection data; based on the first similarity algorithm, The similarity between the data to be processed and the plurality of historically collected data is calculated, and the first similarity algorithm includes standard deviation, Euclidean distance, cosine distance, Pearson correlation coefficient, modified cosine distance, Hamming distance, and Manhattan distance. any of the.

Optionally, the similarity calculation unit 620 is specifically configured to, if the data to be processed is multi-dimensional data, obtain the historical collection data collected by each of multiple historical collection periods, and obtain a plurality of historical collection data; Calculate similarity between the data to be processed and the plurality of historically collected data, and the second similarity algorithm includes any one of Euclidean distance, cosine distance, Pearson correlation coefficient, modified cosine distance, Hamming distance, and Manhattan distance .

As yet another manner, the similarity calculating unit 620 is specifically configured to calculate the similarity between the data to be processed and the historically collected data collected in the historical collection period based on the dimension of the to-be-processed data.

In one way, the storage unit 630 is specifically configured to store the data to be processed if the similarity does not meet the specified threshold condition and the collection time of the data to be processed matches the persistent storage period.

In another manner, the storage unit 630 is specifically configured to collect and store data based on a persistent storage period, wherein the persistent storage period is greater than the sampling period.

Wherein, optionally, a persistent storage period is determined based on the type of the data to be processed.

Optionally, the specified threshold condition includes that the similarity is less than a first similarity threshold, or a difference between the similarity and a similarity corresponding to a previous collection period is less than a second similarity threshold.

An electronic device provided by the present application will be described below with reference to FIG. 13 .

Referring to FIG. 13 , based on the foregoing data collection method and apparatus, another electronic device 1000 that can execute the foregoing data collection method is further provided by the embodiments of the present application. The electronic device 1000 includes one or more (only one is shown in the figure) a processor 102 and a memory 104 that are coupled to each other. Wherein, the memory 104 stores a program that can execute the content in the foregoing embodiments, and the processor 102 can execute the program stored in the memory 104 . The processor 102 may include one or more processing cores. The processor 102 uses various interfaces and lines to connect various parts of the entire electronic device 1000, and executes by running or executing the instructions, programs, code sets or instruction sets stored in the memory 104, and calling the data stored in the memory 104. Various functions of the electronic device 1000 and processing data.

Optionally, the processor 102 may adopt at least one of a digital signal processing (Digital Signal Processing, DSP), a Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), and a Programmable Logic Array (Programmable Logic Array, PLA). A hardware form is implemented. The processor 102 may integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU), a modem, and the like. Among them, the CPU mainly handles the operating system, user interface and application programs; the modem is used to handle wireless communication. It can be understood that, the above-mentioned modem may not be integrated into the processor 102, and is implemented by a communication chip alone.

The memory 104 may include random access memory (Random Access Memory, RAM), or may include read-only memory (Read-Only Memory). Memory 104 may be used to store instructions, programs, codes, sets of codes, or sets of instructions. The memory 104 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.) , instructions for implementing the following method embodiments, and the like. The storage data area may also store data created by the electronic device 1000 during use (such as a phone book, audio and video data, chat record data) and the like.

Please refer to FIG. 14 , which shows a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application. The computer-readable storage medium 800 stores program codes, and the program codes can be invoked by the processor to execute the methods described in the above method embodiments.

The computer readable storage medium 800 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 800 includes a non-transitory computer-readable storage medium. Computer readable storage medium 800 has storage space for program code 810 to perform any of the method steps in the above-described methods. These program codes can be read from or written to one or more computer program products. The program code 800 may, for example, be compressed in a suitable form.

To sum up, the data acquisition method, device, electronic device and storage medium provided by the present application, after acquiring the acquisition data acquired in the current acquisition period in response to the acquisition instruction, the acquisition data acquired in the current acquisition period is collected. As the data to be processed, the similarity between the data to be processed and the historical collection data collected in the historical collection period is calculated, and if the similarity meets a specified threshold condition, the data to be processed is stored. Thus, by the above method, the similarity between the calculated data to be processed and the historical collection data collected in the historical collection period can be compared with the specified threshold condition, and if the similarity satisfies the specified threshold condition, then treat it again. The processed data is stored, so that the acquired collected data will be screened according to the specified threshold conditions, so as to obtain the collected data that meets the requirements (specified threshold conditions) for storage, so that it is not necessary to directly store the collected data obtained each time. Save storage space.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or some technical features thereof are equivalently replaced; and these modifications or replacements do not drive the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions in the embodiments of the present application.

Claims (15)

1. a data collection method, is characterized in that, described method comprises: In response to the acquisition instruction, acquiring the acquisition data acquired in the current acquisition cycle as the data to be processed; Calculate the similarity between the data to be processed and the historical collection data collected in the historical collection period; If the similarity satisfies the specified threshold condition, the data to be processed is stored. 2. The method according to claim 1, wherein the calculating the similarity between the collection data and the historical collection data collected in the historical collection period comprises: The similarity between the data to be processed and the historically collected data collected in the historical collection period is calculated based on the type of the data to be processed. 3. The method according to claim 2, wherein calculating the similarity between the data to be processed and the historical collection data collected in a historical collection period based on the type of the data to be processed comprises: If the data to be processed is cumulative data, the similarity is calculated based on the data to be processed and the rate of change of the historical collection data collected in the historical collection period; If the data to be processed is non-cumulative data, the similarity is calculated based on the data to be processed and the historical collection data collected in the historical collection period. 4. The method according to claim 3, wherein the calculating similarity based on the data to be processed and the rate of change of the historical collection data collected in the historical collection period comprises: If the data to be processed is single-dimensional data, obtain the historical collection data collected by each of a plurality of historical collection periods, and obtain a plurality of historical collection data; Obtain multiple pairs of data from the data to be processed and the plurality of historical collection data, and the respective collection periods corresponding to the data in each pair of data are adjacent; Obtain the difference between the data corresponding to the later acquisition period and the data corresponding to the previous acquisition period in each pair of data as the reference difference value, so as to obtain the reference difference value corresponding to each pair of data; Comparing the reference difference corresponding to each pair of data with the data corresponding to the previous acquisition period in each pair of data, the corresponding change rate of each pair of data is obtained; The similarity is calculated based on the first similarity algorithm and the rate of change corresponding to each pair of data. The first similarity algorithm includes standard deviation, Euclidean distance, cosine distance, Pearson correlation coefficient, modified cosine distance, Hamming distance, and Manhattan distance. any of the . 5. The method according to claim 3, wherein the calculating similarity based on the data to be processed and the rate of change of the historical collection data collected in the historical collection period comprises: If the data to be processed is multi-dimensional data, obtain the historical collection data collected by each of multiple historical collection periods, and obtain a plurality of historical collection data; Acquiring the data to be processed and the data of each dimension position in the plurality of historically collected data; Divide the data of each dimension position into multiple groups based on the corresponding dimension position to obtain multiple groups of data, wherein the dimension positions corresponding to the same group of data are the same; Acquire multiple pairs of data in each group of data, and the data in each pair of data have adjacent acquisition periods; Obtaining the difference between the data corresponding to the following acquisition period and the data corresponding to the previous acquisition period in each pair of data as a reference difference value, so as to obtain a reference difference value corresponding to each pair of data in each group of data; Comparing the reference difference corresponding to each pair of data with the data corresponding to the previous acquisition period in each pair of data, the corresponding change rate of each pair of data is obtained; Based on the sampling period of the included data, the multiple pairs of data are divided into multiple sets, wherein each pair of data in the same set has the same sampling period; Corresponding multi-dimensional data is generated based on the rate of change corresponding to each data in each set, and multiple multi-dimensional data are obtained, wherein the rate of change corresponding to each pair of data is in the middle dimension position of the corresponding generated multi-dimensional data and the pair of data The dimension positions of the data in the data to be processed or the historically collected data are the same; The similarity is calculated based on a second similarity algorithm and the plurality of multidimensional data, the second similarity algorithm includes any one of Euclidean distance, cosine distance, Pearson correlation coefficient, modified cosine distance, Hamming distance, and Manhattan distance item. 6. The method according to claim 3, wherein the calculating similarity based on the data to be processed and the historical collection data collected in the historical collection period, comprising: If the data to be processed is single-dimensional data, obtain the historical collection data collected by each of a plurality of historical collection periods, and obtain a plurality of historical collection data; The similarity is calculated based on a first similarity algorithm, the data to be processed, and the plurality of historically collected data, where the first similarity algorithm includes standard deviation, Euclidean distance, cosine distance, Pearson correlation coefficient, modified cosine distance, Any of Hamming distance and Manhattan distance. 7. The method according to claim 3, wherein the calculating similarity based on the data to be processed and the historical collection data collected in the historical collection period comprises: If the data to be processed is multi-dimensional data, obtain the historical collection data collected by each of multiple historical collection periods, and obtain a plurality of historical collection data; The similarity is calculated based on a second similarity algorithm, the data to be processed, and the plurality of historically collected data, where the second similarity algorithm includes Euclidean distance, cosine distance, Pearson correlation coefficient, modified cosine distance, and Hamming distance , any of the Manhattan distances. 8. The method according to claim 1, wherein the calculating the similarity between the data to be processed and the historical collection data collected in the historical collection period comprises: The similarity between the data to be processed and the historical collection data collected in the historical collection period is calculated based on the dimension of the data to be processed. 9. The method according to any one of claims 1-8, wherein the method further comprises: If the similarity does not satisfy the specified threshold condition and the collection time of the data to be processed matches the persistent storage period, the data to be processed is stored. 10. The method according to claim 9, wherein the method further comprises: Based on the type of the data to be processed, a persistent storage period is determined. 11. The method of claim 1, wherein the specified threshold condition comprises the similarity being less than a first similarity threshold, or The absolute value of the difference between the similarity and the similarity corresponding to the previous collection period is smaller than the second similarity threshold. 12. The method according to any one of claims 1-8, wherein the method further comprises: Data is collected and stored based on a persistent storage period, wherein the persistent storage period is greater than the sampling period. 13. A data acquisition device, characterized in that the device comprises: a data-to-be-processed acquisition unit, configured to, in response to the acquisition instruction, acquire the acquisition data acquired in the current acquisition cycle as the data to be processed; a similarity calculation unit, configured to calculate the similarity between the data to be processed and the historical collection data collected in the historical collection period; A storage unit, configured to store the data to be processed if the similarity satisfies a specified threshold condition. 14. An electronic device, comprising one or more processors and a memory; One or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method of any of claims 1-12. 15. A computer-readable storage medium, wherein a program code is stored in the computer-readable storage medium, wherein the method according to any one of claims 1-12 is executed when the program code is executed.

Images