CN113221862B

CN113221862B - Data filtering method and device, electronic equipment and storage medium

Info

Publication number: CN113221862B
Application number: CN202110773170.3A
Authority: CN
Inventors: 高林明; 范哲权; 张泽民
Original assignee: Thoth Suzhou Medical Technology Co ltd
Current assignee: Thoth Suzhou Medical Technology Co ltd
Priority date: 2021-07-08
Filing date: 2021-07-08
Publication date: 2021-10-15
Anticipated expiration: 2041-07-08
Also published as: CN113221862A

Abstract

The embodiment of the invention discloses a data filtering method and device, electronic equipment and a storage medium. The method comprises the following steps: storing original data sequence to a first buffer area according to data generation time, sorting storage position information of the original data in the first buffer area, and then storing the original data in a second buffer area, further determining a target storage position of a filtering median value in the first buffer area according to the sorting condition of the storage position information stored in the second buffer area, and then searching the filtering median value in the first buffer area based on the target storage position so as to filter the original data stored in the first buffer area through the filtering median value. According to the technical scheme of the embodiment of the invention, the original data and the sorted storage position information are independently stored, so that the filtering median can be quickly positioned and found under the condition of not disturbing the original data sequence, and the filtering efficiency of interference signals in the data is improved.

Description

Data filtering method and device, electronic equipment and storage medium

Technical Field

The embodiment of the invention relates to the technical field of data processing, in particular to a data filtering method and device, electronic equipment and a storage medium.

Background

The baseline drift is one of the common interferences in the electrocardiosignals, and a great deal of influences such as wrong heart beat identification, incomplete waveform display and the like can be caused when the baseline drift is serious. In order to filter the baseline wander in the electrocardiographic signal, the prior art filters the electrocardiographic signal data through a median filter.

The median filter is a nonlinear filter, the principle of the filter is to suppress noise based on a sequencing statistic theory, and the filter has good edge retentivity and impulse noise filtering capability. However, the related median filtering algorithm often involves moving data in multiple batches again in the process of implementing operations such as data insertion, deletion, sorting and the like, so that the filtering efficiency of the median filter is low.

Disclosure of Invention

The embodiment of the invention provides a data filtering method and device, electronic equipment and a storage medium, which are used for realizing rapid filtering of data and improving the filtering efficiency of interference signals in the data.

In a first aspect, an embodiment of the present invention provides a data filtering method, including:

acquiring original data with a preset window length, and storing the original data into a first buffer area based on data generation time;

sorting the storage position information of the original data in the first buffer area based on the data size of the original data, and storing the sorted storage position information into a second buffer area;

determining a target storage position of a filtered median value in the first buffer area based on the sorting result of the storage position information in the second buffer area;

and taking the original data in the first buffer area at the target storage position as the filtering median, and filtering the original data in the first buffer area based on the filtering median to obtain filtered data.

Optionally, the data filtering method further includes:

and when the newly added data is detected, taking the original data with the earliest data storage time in the first buffer area as target replacement data, deleting the target replacement data, and adding the newly added data to the storage position information corresponding to the target replacement data.

Optionally, after the adding the new data to the storage location information corresponding to the target replacement data, the method further includes:

comparing the newly added data with the target replacement data, and determining a sorting adjustment interval of storage position information corresponding to the newly added data in the second buffer area based on a comparison result;

and determining a target sorting position of the storage position information corresponding to the newly added data in the second buffer area based on the newly added data and the original data corresponding to each storage position information in the sorting adjustment interval.

and determining a target sorting position of the storage position information corresponding to the newly added data in the second buffer area based on the newly added data and the original data corresponding to each storage position information in the second buffer area by a binary search method.

Optionally, the determining, by a binary search method, a sorting position of the storage location information corresponding to the newly added data in the second buffer area based on the newly added data and the original data corresponding to each storage location information in the sorting adjustment interval includes:

determining original data corresponding to storage position information located at a middle position in the sorting adjustment interval as reference data;

comparing the newly added data with the reference data, and determining storage position information corresponding to the newly added data to update the sequencing adjustment interval based on a comparison result;

and if the number of the storage position information of the sorting adjustment interval is larger than a preset threshold value, continuously determining the target sorting position of the storage position information corresponding to the newly added data in the second buffer area by a binary search method based on the newly added data and the original data corresponding to each piece of storage position information in the updated sorting adjustment interval.

Optionally, after determining the target sorting position of the storage position information corresponding to the newly added data in the second buffer, the method further includes:

determining a current sorting position of storage position information corresponding to the target replacement data in the second buffer area, and taking the current sorting position as a position to be covered;

determining a data moving interval in a second buffer area based on the current sorting position and the target sorting position, wherein the storage position information in the data moving interval comprises the storage position information currently at the target sorting position and does not comprise the storage position information corresponding to the target replacement data;

moving each piece of storage position information in the data moving interval towards the direction of the position to be covered according to an original sorting mode, so that the storage position information adjacent to the storage position information at the position to be covered in the data moving interval is covered to the position to be covered, and the target sorting position is left;

and inserting the storage position information corresponding to the newly-added data to the target sorting position.

Optionally, the data filtering method further includes:

when at least two newly added data are detected, the newly added data are added to the first buffer one by one from the morning to the evening based on the data generation time of the newly added data.

Optionally, the filtering the original data of the first buffer based on the filtering median to obtain filtered data includes:

determining original data corresponding to the filtering median value in the first buffer area, and taking the original data corresponding to the filtering median value as data to be filtered;

and subtracting the filtering median value from the data to be filtered to obtain filtering data of the data to be filtered.

In a second aspect, an embodiment of the present invention further provides a data filtering apparatus, where the apparatus includes:

the data buffer module is used for acquiring original data with a preset window length and storing the original data to a first buffer area based on data generation time;

the sorting buffer module is used for sorting the storage position information of the original data in the first buffer area based on the data size of the original data and storing the sorted storage position information into a second buffer area;

a median position determination module, configured to determine a target storage position of a filtered median in the first buffer area based on a sorting result of the storage position information in the second buffer area;

and the data filtering module is used for taking the original data in the first buffer area at the target storage position as the filtering median value, and filtering the original data in the first buffer area based on the filtering median value to obtain filtered data.

In a third aspect, an embodiment of the present invention further provides an electronic device, where the electronic device includes:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement a data filtering method as provided by any of the embodiments of the invention.

In a fourth aspect, the embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data filtering method according to any of the embodiments of the present invention.

The embodiment of the invention has the following advantages or beneficial effects:

in the data filtering method provided by the embodiment of the invention, the original data with the preset window length is stored in the first buffer area based on the data generation time, that is, the original data is stored by keeping the original data sequence according to the data generation time, and the storage position information of the original data in the first buffer area is sorted and then stored in the second buffer area, so that the sorting condition of the original data is represented by the sorting of the storage position information of the original data in the first buffer area, and when the data length of the first buffer area is fixed, the storage position information stored in the second buffer area is relatively fixed data, and frequent data replacement is not needed. And then determining the target storage position of the filtering median in the first buffer area according to the sorting condition of the storage position information stored in the second buffer area, searching the filtering median in the first buffer area based on the target storage position, and finally filtering the original data stored in the first buffer area through the filtering median. Compared with the existing filtering method, the method for directly sequencing the original data has the advantages that the original data and the sequenced storage position information are independently stored, the filtering median can be quickly located and found under the condition that the original data sequence is not disturbed, and compared with the original filtering method, the method for filtering the median can only obtain the filtering median and needs to obtain the original data sequence again.

Drawings

In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, a brief description is given below of the drawings used in describing the embodiments. It should be clear that the described figures are only views of some of the embodiments of the invention to be described, not all, and that for a person skilled in the art, other figures can be derived from these figures without inventive effort.

Fig. 1A is a schematic flowchart of a data filtering method according to an embodiment of the present invention;

fig. 1B is a schematic flowchart of a method for storing original data according to an embodiment of the present invention;

fig. 2 is a schematic flowchart of a data filtering method according to a second embodiment of the present invention;

fig. 3 is a schematic flowchart of a data filtering method according to a third embodiment of the present invention;

fig. 4A is a schematic flowchart illustrating a method for inserting storage location information of new data according to a fourth embodiment of the present invention;

fig. 4B is a flowchart illustrating a method for inserting storage location information of new data according to a fourth embodiment of the present invention;

fig. 5 is a schematic structural diagram of a data filtering apparatus according to a fifth embodiment of the present invention;

fig. 6 is a schematic structural diagram of an electronic device according to a sixth embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.

Before describing the data filtering method provided by the embodiment of the present invention in detail, an application scenario of the data filtering method provided by the embodiment is exemplarily described. For example, the data filtering method proposed in this embodiment may be applied to filter out interference noise in the signal. For example, baseline drift caused by unstable device voltage, leakage of coupling capacitor, damage of electronic tube or transistor, etc. in the electrocardiosignal is filtered. Alternatively, baseline drift in the gas chromatography signal, the liquid chromatography signal, or the differential thermal analysis signal is filtered out. Alternatively, salt and pepper noise, gaussian noise, poisson noise, or the like in the image is eliminated. Or eliminating trend terms in the sequence, etc. The present embodiment does not limit the application scenario of the data filtering method, and the above listed application scenarios are only exemplary.

Example one

Fig. 1A is a schematic flow diagram of a data filtering method according to an embodiment of the present invention, which is applicable to a situation where interference information in data needs to be filtered, and is particularly applicable to a situation where original data and storage location information of sorted original data are stored according to a preset first buffer and a preset second buffer, so as to determine a filtering median according to a sorting result of the storage location information stored in the second buffer, and the original data is filtered based on the filtering median, where the method may be executed by a data filtering apparatus, where the apparatus may be implemented by hardware and/or software, and the method specifically includes the following steps:

s110, acquiring original data with a preset window length, and storing the original data in a first buffer area based on data generation time.

The preset window length may be a preset time length of data covered by the window. For example, the preset window length may be 2s, 5s, 1min, or the like for raw data of an electrocardiograph signal type, and may also be 101, 201, 501, or the like, and the preset window length may be 9 × 9, 3 × 3, or the like for raw data of an image type, which is not limited in this application. Optionally, the preset window length is odd, that is, odd number of original data are acquired.

It should be noted that the size of the preset window length affects the efficiency and precision of data filtering, and specifically, if the preset window length is longer, the more the acquired original data is, the larger the calculation amount of the data during filtering is, the lower the filtering efficiency is, and the higher the filtering accuracy is; conversely, if the preset window length is smaller, the acquired original data is smaller, the calculation amount of the data during filtering is smaller, the filtering efficiency is higher, and the filtering accuracy is lower. Therefore, the size of the preset window length can be set according to the requirements of the efficiency and the precision of actual filtering, and the method is not limited in the application.

Specifically, after the original data with the preset window length is acquired, the original data may be stored in the first buffer area according to the data generation time of the original data. In this embodiment, the first buffer may be a buffer space for storing the raw data filtered each time. Optionally, the first buffer area is a ring buffer area; the length of the storage space of the first buffer area is equal to the length of the preset window.

When the original data is stored in the first buffer according to the data generation time of the original data, all the original data stored in the first buffer may be regarded as a segment of data stream, and the generation time of each original data in the data stream is sequentially increased or decreased. Illustratively, the original data t0, t1, t2, … …, tL-1 are stored in the first buffer; wherein, L is the length of the preset window, and t0, t1, t2, … … and tL-1 are the original data generated in the preset window and continuous in time.

Illustratively, as shown in fig. 1B, a flow chart of a method for storing raw data is shown. As shown, the step of storing the raw data into the first buffer is summarized as follows:

step 1, judging whether original data needing to be written into a first buffer area exists, if so, executing step 2, otherwise, stopping writing of the original data, and initializing a second buffer area;

step 2, putting the original data into a first buffer area, and adding 1 to a pointer of the first buffer area;

and 3, judging whether the pointer of the first buffer area is larger than or equal to the storage length of the first buffer area, if so, setting the pointer of the first buffer area to be 0, initializing the second buffer area, and if not, returning to execute the step 1.

Of course, considering that part of the original data is not a discrete signal, if the original data is a continuous signal, the original data may be converted into discrete data before storing the original data, and the discrete data may be stored in the order of the data generation time from early to late. For example, the raw data may be sampled by a preset sampling interval to obtain discrete raw data.

S120, sorting the storage position information of the original data in the first buffer area based on the data size of the original data, and storing the sorted storage position information into the second buffer area.

The method comprises the steps that original data are stored in a first buffer area according to the data generation time sequence of the original data; the storage location information of the original data in the first buffer area may be a specific storage location of the original data in the first buffer area. Along with the above example, the storage location information of the original data t0, t1, t2, … …, tL-1 in the first buffer area can be A0, A1, A2, … … AL-1, respectively. That is, the original data t0 is stored at a0, the original data t1 is stored at a1, and so on.

Specifically, the storage location information of the original data in the first buffer area is sorted based on the data size of the original data, and the storage location information of the original data in the first buffer area may be sorted according to a sorting rule that the original data is sorted from small to large or from large to small.

For example, the original data t0, t1, t2, t3, t4 may be a0, a1, a2, A3, and a4, respectively, and the storage location information of the original data in the first buffer is sorted based on the data size of the original data, e.g., t4> t1> t2> t3> t0, resulting in a4, a1, a2, A3, and a 0.

And storing the sorted storage position information into a second buffer area, wherein the storage position information sorted according to the data size of the original data is stored in the second buffer area. For example, following the above example, the storage location information of the original data in the first buffer is sorted to obtain a4, a1, a2, A3, and a0, and then a4, a1, a2, A3, and a0 are stored at the storage locations B0, B1, B2, B3, and B4 of the second buffer, respectively. From the storage position information a4 taken out at the storage position B0 of the second buffer, it can be known that the original data t4 stored in the storage position a4 of the first buffer is the maximum value, from the storage position information a0 taken out at the storage position B4 of the second buffer, it can be known that the original data t0 stored in the storage position a0 of the first buffer is the minimum value, and the data between the storage positions B0 and B4 of the second buffer is the storage position information of the original data sorted in the descending order in the first buffer.

It should be noted that "sorting the storage location information of the original data in the first buffer area based on the original data" herein essentially sorts the storage location information of the original data, rather than directly sorting the original data. The original data is still stored in the first buffer based on the data generation time. The advantage of this is that the filtering process can continue with the original data in the first buffer after the filtering median is obtained.

And S130, determining a target storage position of the filtering median in the first buffer area based on the sorting result of the storage position information in the second buffer area.

Wherein, the filtered median value can be intermediate data after the sorting of the original data. In one embodiment, determining a target storage location of a filtered median in the first buffer based on the sorting result of the storage location information in the second buffer comprises: and determining an intermediate storage position of the second buffer area, and determining a target storage position of the filtered median value in the first buffer area based on the data stored in the intermediate storage position. Optionally, the filtered median is determined when each storage location in the first buffer stores raw data. In other words, the filtered median is determined when the first buffer is full of data.

The intermediate storage position of the second buffer may be an intermediate position of each storage position storing the sorted storage position information. As in the above example, the storage location B2 of the second buffer is an intermediate storage location, the storage location information stored in B2 is a2, and the original data stored at the storage location a2 in the first buffer is t2, so that the original data t2 is the median of each original data, that is, the filtered median. Specifically, the storage location information stored in the intermediate storage location of the second buffer is determined as the target storage location of the filtering median in the first buffer.

Optionally, the number of the original data stored in the first buffer area is an odd number, and the number of the storage location information stored in the second buffer area is also an odd number, so that a unique intermediate storage location of the second buffer area can be directly determined, and a target storage location of the filtering median value in the first buffer area is determined according to the data stored in the intermediate storage location.

If the number of the original data is an even number, the number of the storage location information stored in the second buffer area is also an even number, at this time, two storage locations in the middle of the second buffer area can be determined as a middle storage location of the second buffer area, and then two target storage locations are determined according to the data stored in the two middle storage locations.

And S140, filtering the original data of the first buffer area based on the filtering median value by taking the original data at the target storage position in the first buffer area as the filtering median value to obtain the filtering data.

Specifically, after a target storage location is determined, that is, after a storage location of a filtering median in the first buffer area is determined, the original data stored at the target storage location is read, the original data is used as the filtering median, and then the original data in the first buffer area is filtered based on the filtering median, so that the filtering data is obtained.

Of course, if the number of target storage locations is 2, that is, the number of raw data is an even number, the raw data stored in any one of the target storage locations may be used as a filtering median; alternatively, the original data stored in each target storage location may be read, the average value of each original data may be processed, and the value after the average processing may be used as the filtering median value.

In an embodiment, optionally, filtering the original data of the first buffer based on the filtering median to obtain filtered data includes: determining original data corresponding to a filtering median value in a first buffer area, and taking the original data corresponding to the filtering median value as data to be filtered; and subtracting the filtering median value from the data to be filtered to obtain filtering data of the data to be filtered.

Wherein, the original data corresponding to the filtered median may be original data in the first buffer at an intermediate time point in the data generation time of all the original data in the first buffer. In this optional embodiment, point-by-point filtering of the original data may be implemented, and if newly added data is stored in the first buffer, the filtering median value needs to be determined again, and the data to be filtered corresponding to the filtering median value needs to be determined again, that is, the original data whose data generation time is the middle time.

Specifically, if the data to be filtered is determined for the first time, at this time, the original data stored in the first buffer area is data sorted according to the data generation time, so that the data to be filtered may be the original data stored in the middle of the first buffer area; if the newly added data is stored in the first buffer, that is, the data to be filtered is not determined for the first time, the data to be filtered may be the original data whose data generation time in the first buffer is the intermediate time.

Optionally, when the data to be filtered is not determined for the first time, the current data to be filtered may be adjacent original data of the data to be filtered determined for the last time, specifically, adjacent original data with a time later than the data generation time of the data to be filtered determined for the last time.

For example, when data to be filtered is determined for the first time, the original data of the first buffer area is: t0, t1, t2, t3 and t4, wherein the sequence is arranged according to the sequence of the data generation time from early to late, at the moment, the data to be filtered is t2, and the filtering of the original data t2 can be realized according to the determined filtering median; when the new data t5 is stored, the original data of the first buffer area is: t5, t1, t2, t3 and t4, the data to be filtered determined at this time is the original data which is adjacent to t2 and has the data generation time larger than the data generation time of t2, that is, t3, and the filtering of the original data t3 is realized according to the filtering median value determined at this time.

By the method, point-by-point filtering of a section of original data can be achieved, offset information in the data to be filtered can be eliminated by means of difference between the data to be filtered and the filtering median corresponding to the data to be filtered, filtering data with accurate baseline and no offset can be obtained, and filtering precision of the data is improved.

In another embodiment, optionally, the filtering the original data of the first buffer based on the filtering median to obtain filtered data includes: determining original data corresponding to a filtering median value in a first buffer area, and taking the original data corresponding to the filtering median value as data to be filtered; and replacing the data to be filtered with a filtering median value.

For the determination of the data to be filtered, reference may be made to the above-described embodiments. Specifically, in this alternative embodiment, smooth and noiseless filtered data may be obtained by replacing the data to be filtered with the filtered median value for the image data with isolated noise.

According to the technical scheme of the embodiment, original data with a preset window length are stored in a first buffer area based on data generation time, namely, the original data are stored by keeping an original data sequence according to the data generation time, storage position information of the original data in the first buffer area is sorted and then stored in a second buffer area, the sorting condition of the original data is represented by the sorting of the storage position information of the original data in the first buffer area, and when the data length of the first buffer area is fixed, the storage position information stored in the second buffer area is relatively fixed data and does not need to be replaced with frequently. And then determining the target storage position of the filtering median in the first buffer area according to the sorting condition of the storage position information stored in the second buffer area, searching the filtering median in the first buffer area based on the target storage position, and finally filtering the original data stored in the first buffer area through the filtering median. Compared with the existing filtering method, the method for directly sequencing the original data has the advantages that the original data and the sequenced storage position information are independently stored, the filtering median can be quickly located and found under the condition that the original data sequence is not disturbed, and compared with the original filtering method, the method for filtering the median can only obtain the filtering median and needs to obtain the original data sequence again.

Example two

Fig. 2 is a schematic flow chart of a data filtering method according to a second embodiment of the present invention, where on the basis of the foregoing embodiments, optionally, the data filtering method provided in this embodiment further includes: and when the newly added data is detected, taking the original data with the earliest data storage time in the first buffer area as target replacement data, deleting the target replacement data, and adding the newly added data to the storage position information corresponding to the target replacement data. Wherein explanations of the same or corresponding terms as those of the above embodiments are omitted. Referring to fig. 2, the data filtering method provided in this embodiment includes the following steps:

s210, acquiring original data with a preset window length, and storing the original data in a first buffer area based on data generation time.

S220, sorting the storage position information of the original data in the first buffer area based on the data size of the original data, and storing the sorted storage position information into the second buffer area.

And S230, determining a target storage position of the filtering median in the first buffer area based on the sorting result of the storage position information in the second buffer area.

S240, taking the original data in the first buffer area at the target storage position as a filtering median, and filtering the original data in the first buffer area based on the filtering median to obtain filtering data.

And S250, when the newly added data is detected, taking the original data with the earliest data storage time in the first buffer area as target replacement data, deleting the target replacement data, and adding the newly added data to the storage position information corresponding to the target replacement data.

Wherein, the new data may be the original data of the new data. Of course, when the new data is detected, it is necessary to delete a part of the original data stored in the first buffer so that the new data can be stored in the first buffer. In the present embodiment, the original data whose data storage time is the earliest in the first buffer is taken as the target replacement data, that is, the data that needs to be deleted. As can be seen from the foregoing, the original data stored in the first buffer are stored sequentially based on the data generation time, and therefore, the original data whose data storage time is the earliest may be the original data whose data generation time is the earliest.

After the target replacement data is deleted, the newly added data may be stored to the storage location information corresponding to the target replacement data. Illustratively, the target replacement data t0 stored at a0 in the first buffer is deleted, and the newly added data is added at a0 in the first buffer.

It is contemplated that the number of newly added data may be plural. Optionally, the data filtering method provided in this embodiment further includes: when at least two newly added data are detected, the newly added data are added to the first buffer one by one from early to late based on the data generation time of the newly added data.

Specifically, when a plurality of new data are added to the first buffer, the new data may be stored one by one according to the data generation time, so that the new data with the earlier data generation time is located before the new data with the later data generation time at the storage position of the first buffer. For example, the new data t6, t7, and t8, the data generation time of the new data is sorted from early to late as t6, t7, and t8, and the order of adding the new data to the first buffer is: t6, t7, t 8.

Of course, when a plurality of pieces of new data are added to the first buffer, a plurality of pieces of target replacement data corresponding to the number of pieces of new data may be determined in the first buffer, and each piece of new data may be added to the storage location information of each piece of target replacement data.

For example, when the new data t6, t7, and t8 are added, the target replacement data is t0, t1, and t2, t0, t1, and t2 are deleted, and t6, t7, and t8 are stored in the storage location information a0, a1, and a2 of t0, t1, and t2, respectively. If the first buffer is a ring buffer, the start point data and the end point data of the original data in the first buffer at this time are t3 and t8 respectively, i.e., the original data with the earliest generation time is t3 and the data with the latest generation time is t 8.

In the scheme of only storing original data in one buffer area, directly sequencing the original data in the buffer area and further determining a median value, the initial sequence of the original data stored in the buffer area is disturbed, and if new data needs to be inserted, the original data and the new data need to be written into the first buffer area again and then sequenced again. In contrast, in the embodiment, the original data and the storage location information sorted according to the original data are respectively stored in the first buffer area and the second buffer area, if new data needs to be inserted, the original data does not need to be rewritten, the original data with the earliest generation time can be directly positioned in the original data stored in the first buffer area, the original data is deleted, and the new data is inserted into the storage location of the original data, so that the data to be deleted is quickly positioned, and the filtering efficiency of the data is further improved. On this basis, the data filtering method according to the embodiment of the present invention may further include: and updating the sequencing result of the second buffer area based on the newly added data, the original data and the storage position information.

That is, the technical solution provided in this embodiment may also determine, according to the storage location information written in the second buffer area, the storage location information that the new data should be written in the second buffer area after the new data is inserted, and it is not necessary to perform integral sorting on the new data and the original data again, thereby reducing the sorting time and further improving the filtering efficiency of the data.

EXAMPLE III

Fig. 3 is a schematic flow chart of a data filtering method according to a third embodiment of the present invention, where on the basis of the foregoing embodiments, optionally after adding the new data to the storage location information corresponding to the target replacement data, the method further includes: comparing the newly added data with the target replacement data, and determining a sorting adjustment interval of storage position information corresponding to the newly added data in the second buffer area based on the comparison result; and determining a target sorting position of the storage position information corresponding to the newly added data in the second buffer area based on the newly added data and the original data corresponding to each storage position information in the sorting adjustment interval. Wherein explanations of the same or corresponding terms as those of the above embodiments are omitted.

Referring to fig. 3, the data filtering method provided in this embodiment includes the following steps:

s310, acquiring original data with a preset window length, and storing the original data to a first buffer area based on data generation time.

S320, sorting the storage position information of the original data in the first buffer area based on the data size of the original data, and storing the sorted storage position information into the second buffer area.

And S330, determining a target storage position of the filtering median in the first buffer area based on the sorting result of the storage position information in the second buffer area.

S340, taking the original data in the first buffer area at the target storage position as a filtering median, and filtering the original data in the first buffer area based on the filtering median to obtain filtering data.

And S350, when the newly added data is detected, taking the original data with the earliest data storage time in the first buffer area as target replacement data, deleting the target replacement data, and adding the newly added data to the storage position information corresponding to the target replacement data.

And S360, comparing the newly added data with the target replacement data, and determining a sorting adjustment interval of the storage position information corresponding to the newly added data in the second buffer area based on the comparison result.

After the new data is added to the first buffer area, the storage location information of the first buffer area stored in the second buffer area needs to be updated. For example, the data size of the newly added data may be compared with the data size of the target replacement data to determine the writing position of the storage position information of the newly added data in the second buffer area based on the storage position information stored in the second buffer area sorted by the data size.

Specifically, in this embodiment, the sorting adjustment interval in which the storage location information corresponding to the newly added data is located in the second buffer may be determined according to the comparison result between the newly added data and the target replacement data. The sorting adjustment interval may be an interval in which storage location information corresponding to newly added data needs to be inserted.

For example, if the original data t0, t1, t2, t3, t4, t5, and t6 are stored in the first buffer, the storage location information of each original data in the first buffer is sorted according to the ascending order of the original data, so as to obtain sorted storage location information A3, a2, A6, a1, a0, a5, and A4 stored in the second buffer, and after the new data t7 is added to the first buffer, the first buffer stores t1, t2, t3, t4, t5, t6, and t 7. At this time, t0 is the target replacement data, and if the comparison result is t7> t0, the storage position information of t7 needs to be inserted after the storage position information of t0, and the sorting adjustment interval is B5-B6; if the comparison result is that t7< t0, the storage position information of t7 needs to be inserted before the storage position information of t0, and the sorting adjustment interval is B0-B3; if the comparison result is t7= t0, the storage location information of t7 needs to be inserted into the storage location information of t0, and the rank adjustment interval is 0.

And S370, determining a target sorting position of the storage position information corresponding to the newly added data in the second buffer area based on the newly added data and the original data corresponding to each storage position information in the sorting adjustment interval.

Specifically, after the sorting adjustment interval is determined, the target sorting position of the storage position information corresponding to the newly added data in the second buffer area may be determined according to the size of the newly added data and the original data corresponding to each storage position information in the sorting adjustment interval.

For example, the newly added data may be compared with the original data corresponding to each storage location information in the sorting adjustment interval one by one, so that the storage location information of the newly added data is written to a location that meets the sorting rule, that is, a target sorting location, based on the sorting rule of the current storage location information.

For example, following the above example, the sorting adjustment interval is B5-B6, B5 and B6 write storage location information a5 and a4 respectively, the original data corresponding to a5 and a4 are t5 and t4, at this time, the new data t7 is compared with t5 and t4 respectively, and the sorting order of t5, t4 and t7 is determined, and if t5< t7< t4, the target sorting position of the storage location information a7 of t7 in the second buffer is determined to be behind a 5.

Of course, after the target sorting position of the storage position information corresponding to the new data in the second buffer area is determined, the storage position information corresponding to the new data may be written in the target sorting position, and each data in the first buffer area is filtered.

According to the technical scheme of the embodiment, after the new data is added to the storage position information corresponding to the target replacement data, the sorting adjustment interval of the storage position information corresponding to the new data in the second buffer area is determined according to the comparison result of the new data and the target replacement data, the target sorting position of the storage position information corresponding to the new data in the second buffer area is determined based on the comparison result of the original data and the new data corresponding to the sorting adjustment interval, the storage position information corresponding to the new data is added to the target sorting position, sorting of all data is not needed, only the data in the sorting adjustment interval in the second buffer area is adjusted, namely, a subset of the second buffer area is adjusted, the determination of the sorting position of the new data is achieved, and the data filtering efficiency is improved.

Compared with the prior art that an insertion point of new data needs to be found, the new data is inserted, the old data is deleted, and two batches of data movement are needed in the data insertion and deletion operations, the embodiment can move one subset of the second buffer area once, so that the data insertion and deletion are completed, and the data filtering efficiency is greatly improved. The embodiment can also maintain the sequencing state by updating the storage position information of the pointing data, optimizes the processing process of positioning insertion of new data and deletion of old data, and further improves the efficiency of filtering interference signals in the data.

It should be noted that this embodiment also provides a method for determining a target sorting position of the storage position information corresponding to the newly added data in the second buffer area based on a binary search method. Illustratively, optionally, after adding the new data to the storage location information corresponding to the target replacement data, the method further includes: and determining the target sorting position of the storage position information corresponding to the newly added data in the second buffer area based on the newly added data and the original data corresponding to each storage position information in the second buffer area by a binary search method.

Specifically, the original data corresponding to the storage location information at the middle position in the second buffer may be used as reference data, the new data is compared with the reference data, the sorting adjustment interval of the storage location information corresponding to the new data is determined based on the comparison result, further, the original data corresponding to the storage location information at the middle position in the sorting adjustment interval is determined, the reference data is updated based on the original data, the updated reference data is compared with the new data, and the sorting adjustment interval is updated based on the storage location information corresponding to the new data determined based on the comparison result until the target sorting location of the storage location information corresponding to the new data in the second buffer is determined.

Of course, a binary search method may be combined with the method for determining the sorting adjustment interval and determining the target sorting position based on the sorting adjustment interval provided in this embodiment. Exemplarily, the determining the sorting position of the storage position information corresponding to the newly added data in the second buffer area based on the newly added data and the original data corresponding to each storage position information in the sorting adjustment interval by a binary search method includes the following steps:

step 1, determining original data corresponding to storage position information located at a middle position in a sorting adjustment interval as reference data.

Step 2: and comparing the newly added data with the reference data, and determining a storage position information updating sequencing adjustment interval corresponding to the newly added data based on the comparison result.

And step 3: and if the number of the storage position information of the sorted adjustment interval is larger than the preset threshold, continuously determining the target sorting position of the storage position information corresponding to the newly added data in the second buffer area by a binary search method based on the newly added data and the original data corresponding to each storage position information in the updated sorted adjustment interval.

In the above optional embodiment, compared with a method in which the section where the target sorting position is located is determined step by step in the whole second buffer area directly through a binary search method, the sorting adjustment section is determined first, and then the section where the target sorting position is located is determined step by step in the sorting adjustment section through the binary search method, the speed of determining the target sorting position is increased, and further, the filtering efficiency of data is increased.

Example four

Fig. 4A is a schematic flow chart of a method for inserting storage location information of newly added data according to a fourth embodiment of the present invention, where this embodiment is based on the foregoing embodiments, optionally, after determining a target sorting location of the storage location information corresponding to the newly added data in the second buffer, the method further includes: determining the current sorting position of the storage position information corresponding to the target replacement data in the second buffer area, and taking the current sorting position as a position to be covered; determining a data moving interval in the second buffer area based on the current sorting position and the target sorting position, wherein the storage position information in the data moving interval comprises the storage position information at the target sorting position at present and does not comprise the storage position information corresponding to the target replacement data; moving each piece of storage position information in the data moving interval towards the direction of the position to be covered according to the original sorting mode, so that the storage position information adjacent to the storage position information at the position to be covered in the data moving interval is covered to the position to be covered, and a target sorting position is left; and inserting the storage position information corresponding to the newly added data to the target sorting position.

Wherein explanations of the same or corresponding terms as those of the above embodiments are omitted. Referring to fig. 4A, the method for inserting the storage location information corresponding to the newly added data provided in this embodiment includes the following steps:

s410, determining the current sorting position of the storage position information corresponding to the target replacement data in the second buffer area, and taking the current sorting position as a position to be covered.

Specifically, the target replacement data is original data to be deleted, and if the target replacement data is deleted, the current sorting position of the storage position information corresponding to the target replacement data in the second buffer area is a position to be covered, that is, a position to be covered, where other storage position information needs to be written.

And S420, determining a data moving interval in the second buffer area based on the current sorting position and the target sorting position, wherein the storage position information in the data moving interval comprises the storage position information at the target sorting position and does not comprise the storage position information corresponding to the target replacement data.

In this embodiment, the storage location information in the data movement interval is storage location information that needs to be moved. Specifically, the storage location information in the moving interval includes storage location information currently at the target sorting location, and storage location information between the target sorting location and the current sorting location.

Illustratively, the new added data is t7, the target sorting position of t7 is B6, the target replacement data is t0, and the current sorting position of to is B4, then the storage position information in the data movement section includes the storage position information written in B5 and the storage position information written in B6.

And S430, moving each piece of storage position information in the data moving interval towards the direction of the position to be covered according to the original sorting mode, so that the storage position information adjacent to the storage position information at the position to be covered in the data moving interval is covered to the position to be covered, and the target sorting position is left.

Specifically, the storage position information in the data moving interval moves towards the to-be-covered position as a whole, and then the storage position information is stored to the position moving towards the to-be-covered position by one bit, the storage position information adjacent to the to-be-covered position is also written into the to-be-covered position, the storage position information at the target sorting position is also written into the adjacent sorting position, and the target sorting position is left.

And S440, inserting the storage position information corresponding to the newly added data to the target sorting position.

Specifically, after the target sorting position is vacated, the storage position information corresponding to the newly added data is filled in the target sorting position to update the storage position information stored in the second buffer area, and then data filtering is performed based on the updated storage position information stored in the second buffer area.

It should be noted that the insertion method of the storage location information corresponding to the newly added data provided in this embodiment is applicable to a situation that after the target sorting location of the newly added data is determined, the storage location information corresponding to the newly added data needs to be filled in the target sorting location.

Illustratively, as shown in fig. 4B, a flow chart of a method for inserting storage location information of new data is shown. As shown in the figure, after the current sorting position and the target sorting position are determined, the sizes of the current sorting position and the target sorting position are judged, if the current sorting position is in front of the target sorting position, the storage position information stored at the current sorting position is deleted, and the storage position information in the data moving interval is moved forwards as a whole; and if the current sorting position is behind the target sorting position, deleting the storage position information stored at the current sorting position, moving the storage position information in the data moving interval backwards as a whole, vacating the target sorting position, and adding the storage position information corresponding to the newly added data to the target sorting position.

According to the technical scheme, after the target sorting position of the storage position information corresponding to the newly added data in the second buffer area is determined, the data moving interval is determined, all the storage position information in the data moving interval is integrally moved to the storage position information of the target replacement data, the new storage position information is stored at the sorting position of the target replacement data, the target sorting position is vacated, the newly added data are placed in the target sorting position, the data insertion and deletion are completed only by moving a subset of the second buffer area once, all the data do not need to be sorted again, the data sorting efficiency is improved, and further, the data filtering efficiency is improved.

EXAMPLE five

Fig. 5 is a schematic structural diagram of a data filtering apparatus according to a fifth embodiment of the present invention, which is applicable to a situation where interference information in data needs to be filtered, and is particularly applicable to a situation where original data and storage location information of the sorted original data are respectively stored according to a preset first buffer area and a preset second buffer area, so as to determine a filtering median according to a sorting result of the storage location information stored in the second buffer area, and filter the original data based on the filtering median, where the apparatus specifically includes: a data buffering module 510, a sorting buffering module 520, a median position determination module 530, and a data filtering module 540.

A data buffer module 510, configured to obtain original data with a preset window length, and store the original data in a first buffer area based on data generation time;

a sorting buffer module 520, configured to sort, based on the data size of the original data, storage location information of the original data in the first buffer, and store the sorted storage location information in a second buffer;

a median location determining module 530, configured to determine a target storage location of a filtered median in the first buffer based on the sorting result of the storage location information in the second buffer;

and a data filtering module 540, configured to take the original data in the first buffer at the target storage location as the filtering median, and filter the original data in the first buffer based on the filtering median to obtain filtered data.

Optionally, the data filtering apparatus further includes a data adding module, configured to, when detecting new data, use original data with the earliest data storage time in the first buffer as target replacement data, delete the target replacement data, and add the new data to storage location information corresponding to the target replacement data.

Optionally, the data filtering device further includes a new data first sorting module, configured to compare the new data with the target replacement data after the new data is added to the storage location information corresponding to the target replacement data, and determine, based on a comparison result, a sorting adjustment interval in which the storage location information corresponding to the new data is located in the second buffer; and determining a target sorting position of the storage position information corresponding to the newly added data in the second buffer area based on the newly added data and the original data corresponding to each storage position information in the sorting adjustment interval.

Optionally, the data filtering device further includes a second new data sorting module, configured to determine, by a binary search method, a target sorting position of the storage location information corresponding to the new data in the second buffer based on the new data and the original data corresponding to each storage location information in the second buffer after the new data is added to the storage location information corresponding to the target replacement data.

Optionally, the data filtering apparatus further includes a storage location inserting module, configured to determine, after the target sorting location of the storage location information corresponding to the newly added data in the second buffer is determined, a current sorting location of the storage location information corresponding to the target replacement data in the second buffer, and use the current sorting location as a location to be covered; determining a data moving interval in a second buffer area based on the current sorting position and the target sorting position, wherein the storage position information in the data moving interval comprises the storage position information currently at the target sorting position and does not comprise the storage position information corresponding to the target replacement data; moving each piece of storage position information in the data moving interval towards the direction of the position to be covered according to an original sorting mode, so that the storage position information adjacent to the storage position information at the position to be covered in the data moving interval is covered to the position to be covered, and the target sorting position is left; and inserting the storage position information corresponding to the newly-added data to the target sorting position.

Optionally, the data filtering apparatus further includes a newly added data adding module, configured to add, when at least two newly added data are detected, the newly added data to the first buffer one by one from the morning to the evening based on data generation time of the newly added data.

Optionally, the data filtering module 540 is specifically configured to determine original data in the first buffer corresponding to the filtering median, and use the original data corresponding to the filtering median as data to be filtered; and subtracting the filtering median value from the data to be filtered to obtain filtering data of the data to be filtered.

In this embodiment, original data with a preset window length is stored in the first buffer area based on the data generation time, that is, the original data is stored while keeping the original data sequence according to the data generation time, and the storage location information of the original data in the first buffer area is sorted and then stored in the second buffer area, so that the sorting condition of the original data is represented by the sorting of the storage location information of the original data in the first buffer area. And then determining the target storage position of the filtering median in the first buffer area according to the sorting condition of the storage position information stored in the second buffer area, searching the filtering median in the first buffer area based on the target storage position, and finally filtering the original data stored in the first buffer area through the filtering median. Compared with the existing filtering method, the method for directly sequencing the original data has the advantages that the original data and the sequenced storage position information are independently stored, the filtering median can be quickly located and found under the condition that the original data sequence is not disturbed, and compared with the original filtering method, the method for filtering the median can only obtain the filtering median and needs to obtain the original data sequence again.

The data filtering device provided by the embodiment of the invention can execute the data filtering method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.

It should be noted that, the units and modules included in the system are merely divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be realized; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the embodiment of the invention.

EXAMPLE six

Fig. 6 is a schematic structural diagram of an electronic device according to a sixth embodiment of the present invention. FIG. 6 illustrates a block diagram of an exemplary electronic device 12 suitable for use in implementing embodiments of the present invention. The electronic device 12 shown in fig. 6 is only an example and should not bring any limitation to the function and the scope of use of the embodiment of the present invention. The device 12 is typically an electronic device that undertakes data filtering functions.

As shown in FIG. 6, electronic device 12 is embodied in the form of a general purpose computing device. The components of electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a memory 28, and a bus 18 that couples the various components (including the memory 28 and the processing unit 16).

Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

Electronic device 12 typically includes a variety of computer-readable media. Such media may be any available media that is accessible by electronic device 12 and includes both volatile and nonvolatile media, removable and non-removable media.

Memory 28 may include computer device readable media in the form of volatile Memory, such as Random Access Memory (RAM) 30 and/or cache Memory 32. The electronic device 12 may further include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only, the storage device 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 6, commonly referred to as a "hard drive"). Although not shown in FIG. 6, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a Compact disk-Read Only Memory (CD-ROM), a Digital Video disk (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. Memory 28 may include at least one program product 40, with program product 40 having a set of program modules 42 configured to carry out the functions of embodiments of the invention. Program product 40 may be stored, for example, in memory 28, and such program modules 42 include, but are not limited to, one or more application programs, other program modules, and program data, each of which examples or some combination may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.

Electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboard, mouse, camera, etc., and display), one or more devices that enable a user to interact with electronic device 12, and/or any devices (e.g., network card, modem, etc.) that enable electronic device 12 to communicate with one or more other computing devices. Such communication may be through an input/output (I/O) interface 22. Also, the electronic device 12 may communicate with one or more networks (e.g., a Local Area Network (LAN), Wide Area Network (WAN), and/or a public Network such as the internet) via the Network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 via the bus 18. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with electronic device 12, including but not limited to: microcode, device drivers, Redundant processing units, external disk drive Arrays, disk array (RAID) devices, tape drives, and data backup storage devices, to name a few.

The processor 16 executes programs stored in the memory 28 to execute various functional applications and data processing, for example, to implement the data filtering method provided by the above-mentioned embodiment of the present invention, including:

Of course, those skilled in the art can understand that the processor can also implement the technical solution of the data filtering method provided by any embodiment of the present invention.

EXAMPLE seven

An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of the data filtering method provided in any embodiment of the present invention, and the method includes:

Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for embodiments of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method of filtering data, comprising:

taking the original data in the first buffer area at the target storage position as the filtering median, and filtering the original data in the first buffer area based on the filtering median to obtain filtered data;

the method further comprises the following steps:

when newly added data are detected, the original data with the earliest data storage time in the first buffer area are used as target replacement data, the target replacement data are deleted, and the newly added data are added to the storage position information corresponding to the target replacement data;

determining a sorting adjustment interval in which the storage position information corresponding to the newly added data is located in the second buffer area based on the newly added data, determining a target sorting position in the second buffer area of the storage position information corresponding to the newly added data based on the original data corresponding to each storage position information in the sorting adjustment interval, and inserting the storage position information corresponding to the newly added data to the target sorting position.

2. The method according to claim 1, further comprising, after the adding the new data to the storage location information corresponding to the target replacement data:

3. The method according to claim 1, further comprising, after the adding the new data to the storage location information corresponding to the target replacement data:

4. The method according to claim 2 or 3, further comprising, after the determining the target sorting position of the storage position information corresponding to the new addition data in the second buffer, the step of:

5. The method of claim 1, further comprising:

6. The method of claim 1, wherein the filtering the raw data of the first buffer based on the filtered median to obtain filtered data comprises:

7. A data filtering apparatus, comprising:

the data filtering module is used for taking the original data in the first buffer area at the target storage position as the filtering median value, and filtering the original data in the first buffer area based on the filtering median value to obtain filtering data;

the device also comprises a data adding module, a data replacing module and a data adding module, wherein the data adding module is used for taking original data with the earliest data storage time in the first buffer area as target replacing data when detecting the newly added data, deleting the target replacing data and adding the newly added data to the storage position information corresponding to the target replacing data; determining a sorting adjustment interval in which the storage position information corresponding to the newly added data is located in the second buffer area based on the newly added data, determining a target sorting position in the second buffer area of the storage position information corresponding to the newly added data based on the original data corresponding to each storage position information in the sorting adjustment interval, and inserting the storage position information corresponding to the newly added data to the target sorting position.

8. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement a data filtering method as claimed in any one of claims 1-6.

9. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the data filtering method of any one of claims 1 to 6.