CN112925950A - Data quality control method and system for continuous star catalogue data - Google Patents

Data quality control method and system for continuous star catalogue data Download PDF

Info

Publication number
CN112925950A
CN112925950A CN202110109011.3A CN202110109011A CN112925950A CN 112925950 A CN112925950 A CN 112925950A CN 202110109011 A CN202110109011 A CN 202110109011A CN 112925950 A CN112925950 A CN 112925950A
Authority
CN
China
Prior art keywords
data
quality control
filtering
curve
variation curve
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110109011.3A
Other languages
Chinese (zh)
Inventor
孟小峰
杨晨
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Renmin University of China
Original Assignee
Renmin University of China
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Renmin University of China filed Critical Renmin University of China
Priority to CN202110109011.3A priority Critical patent/CN112925950A/en
Publication of CN112925950A publication Critical patent/CN112925950A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Image Processing (AREA)

Abstract

The invention relates to a data quality control method and a system for continuous star catalogue data, which comprises the following steps: dynamically generating a smooth light change curve; identifying a difference between the true light variation curve and the smooth light variation curve; and constructing a filtering strategy to discriminate the difference and realize quality control. The invention aims at continuous star catalogue data, can quantify data quality from different dimensions under the condition of not providing a celestial body data quality reference template, and can well distinguish scientific events and noise interference.

Description

Data quality control method and system for continuous star catalogue data
Technical Field
The invention relates to the technical field of astronomy data processing, in particular to a data quality control method and system for continuous star catalogue data.
Background
In time domain astronomy, astronomical big data is generated by taking a picture of massive astronomical bodies through an optical telescope in real time, and high-definition images extract all astronomical information in the picture through technologies such as point source extraction and the like to form a star catalogue. For example, if a high-definition image captured by an optical telescope includes n celestial bodies, a star catalogue including n rows of data can be formed by a point source extraction technology and the like, each row of data represents information collected by one celestial body, such as brightness values, coordinate values and the like, and therefore conversion from unstructured image data to structured star catalogue data is achieved. When the optical telescope continuously shoots the same area, continuous star catalogue data can be formed. The same celestial body in the continuous star catalogue data can be recorded by different star catalogues, the data are extracted to form a time sequence of the change of each celestial body, and the abnormal change of the celestial body can be identified by analyzing the time sequences in real time to find valuable scientific events. However, due to various reasons, such as the clarity of instruments and equipment, cloud and fog shielding, or other light source pollution, etc., data will be interfered, and the accuracy of scientific event discovery will be affected, so it is very necessary to perform data quality control, and currently, the existing data quality control methods mainly include: (1) image noise reduction; (2) and controlling the quality of the star catalogue based on the template.
And (5) reducing the noise of the image. High-definition images shot by an optical telescope often need an image noise reduction algorithm to remove noise points. The currently used image denoising algorithms can be mainly classified into two categories, namely a spatial pixel characteristic denoising algorithm and a transform domain denoising algorithm. The former is processing directly in image space, and the latter is processing indirectly in the image transform domain.
The method based on spatial pixel characteristics is a method for acquiring a new central pixel value by analyzing direct relation between a central pixel and other adjacent pixels in a gray scale space in a window with a certain size. By means of a mathematical transformation, the signal is separated from the noise in the transform domain, and the noise is then filtered out, leaving the signal. Although noise can be eliminated by the image noise reduction method, the interference of noise sources to celestial body data cannot be filtered.
And controlling the quality of the star catalogue based on the template. In order to identify the interference of noise sources to celestial body data, image data can only be converted into star catalogue data, and the star catalogue data is analyzed by taking celestial body data as a basic unit. The current method is a template-based star catalogue quality control method. Firstly, the high-quality observation of a known celestial body can know the mean value and the variance of the fluctuation of the characteristic value of the celestial body, and a celestial body data quality reference template can be constructed by recording the information. When observed in real time, if the change of the acquired celestial feature value exceeds the range defined by the template, the data quality can be considered to be interfered. However, this method is not effective for newly observed unknown celestial bodies. In addition, when a certain scientific event occurs in the antenna, the collected characteristic value may exceed the normal value range in the template, so that the method is difficult to distinguish the difference between the scientific event and the noise interference.
Disclosure of Invention
In view of the above problems, an object of the present invention is to provide a method and a system for controlling data quality of continuous star catalogue data, which can quantify data quality from different dimensions without providing a reference template for celestial body data quality for continuous star catalogue data, and can well distinguish scientific events from noise interference.
In order to achieve the purpose, the invention adopts the following technical scheme: a data quality control method for continuous star catalogue data comprises the following steps: step 1, dynamically generating a smooth light variation curve; step 2, identifying the difference between the real light variation curve and the smooth light variation curve; and 3, constructing a filtering strategy to discriminate the difference and realize quality control.
Further, in step 1, the method for dynamically generating the smooth light change curve includes: optical variable curve by discrete wavelet transform DWT method
Figure BDA0002918575520000021
Real-time estimation is carried out to obtain the light variation curve
Figure BDA0002918575520000022
Real-time estimate of
Figure BDA0002918575520000023
From the real-time estimate
Figure BDA0002918575520000024
A smooth light change curve is constructed.
Further, the estimation method comprises the following steps:
Figure BDA0002918575520000025
wherein the content of the first and second substances,
Figure BDA0002918575520000026
as a curve of light variation
Figure BDA0002918575520000027
At the t ∈ [ t ]k,tp]The value at the time point, wavelet _ func, is the wavelet function, and decomp _ level is the decomposition level.
Further, in step 2, the method for identifying the difference between the real light variation curve and the smooth light variation curve includes:
step 21, acquiring the error between each point of the real light variation curve and each corresponding point on the smooth light variation curve;
step 22, obtaining the mean square error of the whole optical variation curve according to the error of each point on the optical variation curve
Figure BDA0002918575520000028
Further, in step 22, the mean square error
Figure BDA0002918575520000029
Comprises the following steps:
Figure BDA00029185755200000210
wherein the content of the first and second substances,
Figure BDA00029185755200000211
is the length of the light change curve, and e (t) is the error at each point on the light change curve.
Further, in the step 3, the filtering strategy is constructed by using a dispersion quantization method and an outlier quantization method.
Further, the construction method of the filtering strategy adopting the diffusivity quantification comprises the following steps:
311, dividing the observation area into a plurality of sub-partitions according to a uniform grid partition mode;
step 312, further dividing the mag into different brightness intervals in each sub-partition, and dividing similar brightness into the same brightness interval;
313, calculating the median of all K mags in each mag interval of each sub-partition, and taking the median as the true dispersion of the K celestial bodies;
and step 314, determining a filtering threshold value to obtain a filtering rule.
Further, the step 314 includes the following steps:
step 3141, selecting the historical data of a certain day as the reference data, and calculating the data of each celestial body in the reference data in real time
Figure BDA0002918575520000031
A sequence;
step 3142, aggregate all of the same mag interval
Figure BDA0002918575520000032
Value and solve for
Figure BDA0002918575520000033
As a dispersion filter threshold for the mag interval;
step 3143, obtaining a filtering rule according to the filtering threshold value: if it is not
Figure BDA0002918575520000034
If the filtering threshold value is larger than the filtering threshold value, filtering to a point
Figure BDA0002918575520000035
Further, the construction method of the filtering strategy adopting the outlier quantization comprises the following steps:
step 321, dividing the observation area into a plurality of sub-partitions according to a uniform grid partition mode;
322, further dividing the mag into different brightness intervals in each sub-partition, and dividing similar brightness into the same brightness interval;
step 323, assuming the noise is random noise, in each mag interval in each sub-partition
Figure BDA0002918575520000036
Obeying a normal distribution N(μRR) In which μRIs a mean value, σRFiltering outliers using 3-Sigma principle for standard deviation;
step 324, calculate μRIs estimated by
Figure BDA0002918575520000037
σRIs estimated by
Figure BDA0002918575520000038
Determining an outlier;
step 325, obtaining a filtering rule according to the degree of outlier: if it is not
Figure BDA0002918575520000039
Then filter out points
Figure BDA00029185755200000310
A continuous star catalogue data oriented data quality control system comprising: the device comprises a smooth curve generation module, a difference identification module and a filtering discrimination module; the smooth curve generating module is used for dynamically generating a smooth light variation curve; the difference identification module is used for identifying the difference between the real light variation curve and the smooth light variation curve; the filtering discrimination module is used for constructing filtering strategy discrimination differences and realizing quality control.
Due to the adoption of the technical scheme, the invention has the following advantages: 1. the invention does not need to provide a celestial body data quality reference template, can save a large amount of cost for manufacturing the template, and is effective for newly observed unknown celestial bodies. 2. The invention provides two data quality control methods for respectively identifying the dispersion and the outlier of star catalogue data, and can identify different data quality interference conditions caused by noise sources. 3. The two data quality control methods adopted by the invention can well distinguish the difference between the scientific event and the noise interference while identifying the data interference.
Drawings
FIG. 1 is a schematic diagram of a diffusion point in an embodiment of the present invention;
FIG. 2 is a schematic diagram of outliers in an embodiment of the present invention;
FIG. 3 is a schematic diagram of scientific events in an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the drawings of the embodiments of the present invention. It is to be understood that the embodiments described are only a few embodiments of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the described embodiments of the invention, are within the scope of the invention.
The invention provides a data quality control method for continuous star catalogue data. Suppose that the time range of data acquisition of the astronomical telescope is [ t ]k,tp]Wherein t iskIs the start time, tpIs the end time, the acquisition period is t', then at tk,tp]Can be collected (t)p-tk) T' star tables. Further assuming that n celestial data are included in each star table and the characteristic attribute is mag (star, etc., i.e., brightness), then for any celestial object i, at tk,tp]Connecting its mags together in time sequence, called light variation curve
Figure BDA0002918575520000041
For the star table data, many factors contribute to the degradation of the optical change curve, and the present invention distinguishes them into two different types: (1) dispersion of light variation curves caused by external factors; (2) outliers in the light change curve. As shown in fig. 1-3, extreme case diagrams of the dispersion points and outliers of the real light variation curve are given, both of which may be misidentified as scientific events. The present invention proposes two methods, a dispersion quantization method and an outlier quantization method. The two methods can generate two different quality sequences in real time, and the dispersivity and the outlier corresponding to the mag value are respectively marked, so that the quality control is realized. In combination with the following embodimentsThe present invention will be described in detail.
In a first embodiment of the present invention, a method for controlling data quality of continuous star catalogue data is provided, which includes the following steps:
step 1, dynamically generating a smooth light variation curve;
step 2, identifying the difference between the real light variation curve and the smooth light variation curve;
and 3, constructing a filtering strategy to discriminate the difference in the step 2 and realize quality control.
Preferably, in step 1, the method for dynamically generating the smooth light change curve includes: optical variable curve by discrete wavelet transform DWT method
Figure BDA0002918575520000042
Real-time estimation is carried out to obtain the light variation curve
Figure BDA0002918575520000043
Real-time estimate of
Figure BDA0002918575520000044
From the real-time estimate
Figure BDA0002918575520000045
Forming a smooth light variation curve; the DWT method is used for signal processing, and can decompose a base signal into a low-frequency part and a high-frequency part by selecting a proper wavelet function, wherein the low-frequency part can be regarded as a low-frequency part in the embodiment
Figure BDA0002918575520000046
The high frequency part may be considered as a noise signal.
The estimation method comprises the following steps:
Figure BDA0002918575520000047
wherein the content of the first and second substances,
Figure BDA0002918575520000048
as a curve of light variation
Figure BDA0002918575520000049
At the t ∈ [ t ]k,tp]The value at the time point, wavelet _ func, is the wavelet function, and decomp _ level is the decomposition level.
The decomposition level decomp _ level mainly influences the noise reduction effect; through multiple experiments, it is verified that in the present embodiment, the level _ func and the decomp _ level are selected to be sym6 and 4, respectively, so that the information can be obtained in real time
Figure BDA0002918575520000051
Preferably, in step 2, the method for identifying the difference between the real light variation curve and the smooth light variation curve is as follows:
step 21, acquiring the error between each point of the real light variation curve and each corresponding point on the smooth light variation curve;
because the light change curve is compact and smooth under ideal conditions, the light change curve becomes distorted due to dispersion caused by noise interference; i.e. the curve of variation of light
Figure BDA0002918575520000052
Namely the real light change curve. If this distortion can be measured and a dispersion threshold is used to filter out points with high dispersion, a high quality optical variation curve can be obtained. Therefore, the invention describes the dispersion degree of the light change curve by using the error distribution.
Obtaining a light variation curve
Figure BDA0002918575520000053
T time point of (1)
Figure BDA0002918575520000054
Value corresponding to the point in a noise-free environment
Figure BDA0002918575520000055
The error between e' (t) is:
Figure BDA0002918575520000056
will be as in formula (1)
Figure BDA0002918575520000057
From real-time estimates
Figure BDA0002918575520000058
Instead, the error e (t) for each point on the light variation curve can thus be obtained in real time:
Figure BDA0002918575520000059
step 22, obtaining the mean square error of the whole optical variation curve according to the error of each point on the optical variation curve
Figure BDA00029185755200000510
Figure BDA00029185755200000511
Wherein the content of the first and second substances,
Figure BDA00029185755200000512
is the length of the optical variation curve. To ensure that the mean square error is constantly changing in real-time updates, the length of the optical change curve should have a sufficient length, preferably 64 points in this embodiment.
Preferably, in step 3, the filtering strategy is constructed by using both a dispersion quantization method and an outlier quantization method.
(1) Method for quantifying degree of dispersion
For each of the celestial bodies, for example,
Figure BDA00029185755200000513
the physical significance of (a) is primarily a measure of the difference between the true light change curve and the expected smooth trend. If the light change curve of a celestial body is significantly distorted,
Figure BDA00029185755200000514
will become larger. But the occurrence of a scientific event is also essentially a partial value that deviates significantly from the expected smooth trend, and therefore only uses
Figure BDA00029185755200000515
Diffusion and scientific events remain indistinguishable. As shown in fig. 1-3, scientific events and dispersions can be very similar from the point of view of the light change curve itself. In fact, the occurrence of a scientific event is a small probability event, and it is unlikely that all celestial bodies in a certain region will occur together. The noise is likely to cause interference to all celestial bodies in the same region at the same time, such as cloud occlusion. Based on the prior knowledge, all celestial bodies in the same area can be measured
Figure BDA00029185755200000516
And distinguishing noise interference and possible scientific events according to the trend change degree. Therefore, the following filtering strategy metrics are used in this embodiment
Figure BDA00029185755200000517
The degree of change of the trend is the same.
311, dividing the observation area into a plurality of sub-partitions according to a uniform grid partition mode;
for example, into 49 sub-partitions. The dividing aims are mainly that the interference observation area of the noise is possibly uneven, the divided sub-partitions can isolate the unevenness, and the noise interference on the brightness of the celestial body in the same sub-partition is approximately even;
in step 312, the celestial bodies with different brightness have different noise immunity, and obviously, the brighter celestial body has stronger noise immunity. Based on this, the mag is further divided into different luminance sections in each sub-partition (for example, assuming that the luminance sections have [6,7] and [7.8], and the mag set in a certain sub-partition is {6.7,7.8,6.5,7.9}, the mag set can be divided into two mag sets, {6.7,6.5} and {7.8,7.9}, and the definition of the luminance sections is set according to the observation capability of the observation device.), and the similar luminances are divided into the same luminance section;
313, calculating the median of all K mags in each brightness interval of each sub-partition, and taking the median as the true dispersion of the K celestial bodies;
the median solving formula is:
Figure BDA0002918575520000061
in the formula (I), the compound is shown in the specification,
Figure BDA0002918575520000062
the degree of the same trend change in the mag interval is not greatly affected by the individual extreme value. The real noise must be caused
Figure BDA0002918575520000063
Become large without the scientific incident of the individual celestial body causing
Figure BDA0002918575520000064
Significant variations in the temperature of the sample.
Step 314, determining a filtering threshold value to further obtain a filtering rule;
when in use
Figure BDA0002918575520000065
After being able to solve in real time, the next key issue is how to determine the threshold for filtering. The method comprises the following steps:
step 3141, selecting the historical data of a certain day as the reference data, and calculating the data of each celestial body in the reference data in real time
Figure BDA0002918575520000066
A sequence;
among the criteria for selecting the baseline data are low noise interference during observation, e.g. no clouds, good weather and normal instrument operation. This information can be easily determined by looking at the history log.
Step 3142, aggregate all of the same mag interval
Figure BDA0002918575520000067
Value and solve for
Figure BDA0002918575520000068
(three quarters) as a threshold for the dispersivity filter for the mag interval;
if the noise interference of the reference data is too strong, the method can also be adopted
Figure BDA0002918575520000069
(median) as the threshold for the dispersion filter for this mag interval.
Step 3143, obtaining a filtering rule according to the filtering threshold value: if it is not
Figure BDA00029185755200000610
If the filtering threshold value is larger than the filtering threshold value, filtering to a point
Figure BDA00029185755200000611
(2) Method for quantifying degree of outlier
The same as the dispersion quantization, the space sub-partition and the mag interval are divided, and the dividing method is consistent with the dividing method in the dispersion quantization. Comprises the following steps
Step 321, dividing the observation area into a plurality of sub-partitions according to a uniform grid partition mode;
322, further dividing the mag into different brightness intervals in each sub-partition, and dividing similar brightness into the same brightness interval;
step 323, assume the noise is random noise, so in each mag interval in each sub-partition
Figure BDA0002918575520000071
Should obey a normal distribution N (μ)RR) In which μRIs a mean value, σRIs a standard deviation ofOutliers were filtered using 3-Sigma principles.
According to the principle of 3-Sigma,
Figure BDA0002918575520000072
fall in (mu)R-3σRR+3σR) The probability of inner is 0.9974; if it is not
Figure BDA0002918575520000073
Does not fall into (mu)R-3σRR+3σR) Is regarded as an outlier.
Step 324, calculate μRIs estimated by
Figure BDA0002918575520000074
σRIs estimated by
Figure BDA0002918575520000075
Determining an outlier;
to avoid the effects of outliers, use is made of
Figure BDA0002918575520000076
The mean of the front and rear 80% data points was taken as
Figure BDA0002918575520000077
Sample standard deviations of all data points were used as
Figure BDA0002918575520000078
Then degree of outlier
Figure BDA0002918575520000079
Is defined as:
Figure BDA00029185755200000710
step 325, obtaining a filtering rule according to the degree of outlier: if it is not
Figure BDA00029185755200000711
Then filter out points
Figure BDA00029185755200000712
In a second embodiment of the invention, a continuous star catalogue data oriented data quality control system is provided, which comprises a smooth curve generation module, a difference identification module and a filtering discrimination module;
the smooth curve generating module is used for dynamically generating a smooth light variation curve;
the difference identification module is used for identifying the difference between the real light variation curve and the smooth light variation curve;
the filtering discrimination module is used for constructing filtering strategy discrimination differences and realizing quality control.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

Claims (10)

1. A data quality control method for continuous star catalogue data is characterized by comprising the following steps:
step 1, dynamically generating a smooth light variation curve;
step 2, identifying the difference between the real light variation curve and the smooth light variation curve;
and 3, constructing a filtering strategy to discriminate the difference and realize quality control.
2. The data quality control method according to claim 1, wherein in the step 1, the method for dynamically generating the smooth light variation curve comprises: optical variable curve by discrete wavelet transform DWT method
Figure FDA0002918575510000011
Real-time estimation is carried out to obtain the light variation curve
Figure FDA0002918575510000012
Real-time estimate of
Figure FDA0002918575510000013
From the real-time estimate
Figure FDA0002918575510000014
A smooth light change curve is constructed.
3. The data quality control method of claim 2, wherein the estimating method is:
Figure FDA0002918575510000015
wherein the content of the first and second substances,
Figure FDA0002918575510000016
as a curve of light variation
Figure FDA0002918575510000017
At the t ∈ [ t ]k,tp]The value at the time point, wavelet _ func, is the wavelet function, and decomp _ level is the decomposition level.
4. The data quality control method according to claim 1, wherein in the step 2, the method for identifying the difference between the real light variation curve and the smooth light variation curve comprises:
step 21, acquiring the error between each point of the real light variation curve and each corresponding point on the smooth light variation curve;
step 22, obtaining the mean square error of the whole optical variation curve according to the error of each point on the optical variation curve
Figure FDA0002918575510000018
5. The data quality control method according to claim 4, wherein in the step 22, the mean square error
Figure FDA0002918575510000019
Comprises the following steps:
Figure FDA00029185755100000110
wherein the content of the first and second substances,
Figure FDA00029185755100000111
is the length of the light change curve, and e (t) is the error at each point on the light change curve.
6. The data quality control method according to claim 1, wherein in the step 3, the filtering strategy is constructed by using both a dispersion quantization method and an outlier quantization method.
7. The data quality control method according to claim 6, wherein the filtering strategy using the diffusivity quantization is constructed by:
311, dividing the observation area into a plurality of sub-partitions according to a uniform grid partition mode;
step 312, further dividing the mag into different brightness intervals in each sub-partition, and dividing similar brightness into the same brightness interval;
313, calculating the median of all K mags in each mag interval of each sub-partition, and taking the median as the true dispersion of the K celestial bodies;
and step 314, determining a filtering threshold value to obtain a filtering rule.
8. The data quality control method according to claim 7, wherein the step 314 comprises the steps of:
step 3141, selecting the historical data of a certain day as the reference data, and calculating the data of each celestial body in the reference data in real time
Figure FDA0002918575510000021
A sequence;
step 3142, aggregate all of the same mag interval
Figure FDA0002918575510000022
Value and solve for
Figure FDA0002918575510000023
As a dispersion filter threshold for the mag interval;
step 3143, obtaining a filtering rule according to the filtering threshold value: if it is not
Figure FDA0002918575510000024
If the filtering threshold value is larger than the filtering threshold value, filtering to a point
Figure FDA0002918575510000025
9. The data quality control method of claim 6, wherein the filtering strategy using the outlier quantization is constructed by:
step 321, dividing the observation area into a plurality of sub-partitions according to a uniform grid partition mode;
322, further dividing the mag into different brightness intervals in each sub-partition, and dividing similar brightness into the same brightness interval;
step 323, assuming the noise is random noise, in each mag interval in each sub-partition
Figure FDA0002918575510000026
Obeying a normal distribution N (μ)RR) In which μRIs a mean value, σRFiltering outliers using 3-Sigma principle for standard deviation;
step 324, calculate μRIs estimated by
Figure FDA0002918575510000027
σRIs estimated by
Figure FDA0002918575510000028
Determining an outlier;
step 325, obtaining a filtering rule according to the degree of outlier: if it is not
Figure FDA0002918575510000029
Then filter out points
Figure FDA00029185755100000210
10. A data quality control system for continuous star catalogue data, comprising: the device comprises a smooth curve generation module, a difference identification module and a filtering discrimination module;
the smooth curve generating module is used for dynamically generating a smooth light variation curve;
the difference identification module is used for identifying the difference between the real light variation curve and the smooth light variation curve;
the filtering discrimination module is used for constructing filtering strategy discrimination differences and realizing quality control.
CN202110109011.3A 2021-01-27 2021-01-27 Data quality control method and system for continuous star catalogue data Pending CN112925950A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110109011.3A CN112925950A (en) 2021-01-27 2021-01-27 Data quality control method and system for continuous star catalogue data

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110109011.3A CN112925950A (en) 2021-01-27 2021-01-27 Data quality control method and system for continuous star catalogue data

Publications (1)

Publication Number Publication Date
CN112925950A true CN112925950A (en) 2021-06-08

Family

ID=76166809

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110109011.3A Pending CN112925950A (en) 2021-01-27 2021-01-27 Data quality control method and system for continuous star catalogue data

Country Status (1)

Country Link
CN (1) CN112925950A (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105005978A (en) * 2015-07-15 2015-10-28 天津大学 Spectrum real-time filtering method based on Savitzky-Golay filter parameter optimization
CN106777084A (en) * 2016-12-13 2017-05-31 清华大学 For light curve on-line analysis and the method and system of abnormal alarm
CN110427996A (en) * 2019-07-24 2019-11-08 清华大学 The recognition methods of time series abnormal patterns and device based on fuzzy matching
CN111726341A (en) * 2020-06-02 2020-09-29 五八有限公司 Data detection method and device, electronic equipment and storage medium

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105005978A (en) * 2015-07-15 2015-10-28 天津大学 Spectrum real-time filtering method based on Savitzky-Golay filter parameter optimization
CN106777084A (en) * 2016-12-13 2017-05-31 清华大学 For light curve on-line analysis and the method and system of abnormal alarm
CN110427996A (en) * 2019-07-24 2019-11-08 清华大学 The recognition methods of time series abnormal patterns and device based on fuzzy matching
CN111726341A (en) * 2020-06-02 2020-09-29 五八有限公司 Data detection method and device, electronic equipment and storage medium

Similar Documents

Publication Publication Date Title
US10282824B2 (en) Method and system for eliminating stripe noise in infrared images
Oakley et al. Correction of simple contrast loss in color images
CN107404628B (en) Image processing apparatus and method, and monitoring system
CN108763346B (en) Abnormal point processing method for sliding window box type graph median filtering
CN109308460B (en) Article detection method, system and computer readable storage medium
WO2003101120A1 (en) A method and system for estimating sharpness metrics based on local edge statistical distribution
CN109377457B (en) Pill coating image processing method and device, computer equipment and storage medium
CN112085682A (en) Image noise reduction method and device, electronic equipment and storage medium
CN114092355A (en) Self-adaptive infrared image contrast enhancement method and device
Rong et al. Infrared fix pattern noise reduction method based on shearlet transform
CN111353968A (en) Infrared image quality evaluation method based on blind pixel detection and analysis
CN116958127B (en) Silicon wafer quality assessment method based on image features
Prajapati et al. Evaluation of perceptual contrast and sharpness measures for meteorological satellite images
CN112925950A (en) Data quality control method and system for continuous star catalogue data
CN116883412A (en) Graphene far infrared electric heating equipment fault detection method
CN109544593B (en) General image noise estimation method based on self-similarity measurement
CN114724000B (en) Screen shot image moire processing method, device and equipment
Wang et al. A bilateral filtering based ringing elimination approach for motion-blurred restoration image
Sonawane et al. Image quality assessment techniques: An overview
CN106803240B (en) Industrial image light equalizing processing method
CN111445430B (en) Bimodal infrared image fusion algorithm selection method based on difference characteristic amplitude interval fusion validity distribution
Zhu et al. Two-stage non-local means filtering with adaptive smoothing parameter
Bhateja et al. A reduced reference distortion estimation measure for color images
Sheela et al. Comparative analysis of various filtering techniques for Speckle noise suppression in Ultrasound images
US20190197349A1 (en) Image identification method and image identification device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20210608