CN117235464B - Fourier near infrared interference signal virtual generation evaluation method and system - Google Patents
Fourier near infrared interference signal virtual generation evaluation method and system Download PDFInfo
- Publication number
- CN117235464B CN117235464B CN202311508216.4A CN202311508216A CN117235464B CN 117235464 B CN117235464 B CN 117235464B CN 202311508216 A CN202311508216 A CN 202311508216A CN 117235464 B CN117235464 B CN 117235464B
- Authority
- CN
- China
- Prior art keywords
- data
- sample
- virtual
- screening
- virtual data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000011156 evaluation Methods 0.000 title claims abstract description 104
- 238000000034 method Methods 0.000 claims abstract description 89
- 238000009499 grossing Methods 0.000 claims abstract description 38
- 238000012216 screening Methods 0.000 claims description 112
- 238000009826 distribution Methods 0.000 claims description 41
- 238000004422 calculation algorithm Methods 0.000 claims description 36
- 230000008569 process Effects 0.000 claims description 29
- 238000012545 processing Methods 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 13
- 238000011157 data evaluation Methods 0.000 claims description 13
- 238000004364 calculation method Methods 0.000 claims description 9
- 238000000354 decomposition reaction Methods 0.000 claims description 7
- 238000010606 normalization Methods 0.000 claims description 6
- 238000003860 storage Methods 0.000 claims description 5
- 238000013507 mapping Methods 0.000 claims description 4
- 238000005305 interferometry Methods 0.000 claims 3
- 238000004458 analytical method Methods 0.000 abstract description 4
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 208000009119 Giant Axonal Neuropathy Diseases 0.000 description 17
- 201000003382 giant axonal neuropathy 1 Diseases 0.000 description 17
- 238000012549 training Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 10
- 230000003595 spectral effect Effects 0.000 description 10
- 238000004891 communication Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 7
- 238000001228 spectrum Methods 0.000 description 7
- 230000000875 corresponding effect Effects 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 6
- 238000002329 infrared spectrum Methods 0.000 description 6
- 230000000007 visual effect Effects 0.000 description 5
- 230000007274 generation of a signal involved in cell-cell signaling Effects 0.000 description 4
- 238000013256 Gubra-Amylin NASH model Methods 0.000 description 3
- 230000004913 activation Effects 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 3
- 230000000670 limiting effect Effects 0.000 description 3
- 229920000139 polyethylene terephthalate Polymers 0.000 description 3
- 239000005020 polyethylene terephthalate Substances 0.000 description 3
- 230000002829 reductive effect Effects 0.000 description 3
- 239000007787 solid Substances 0.000 description 3
- 239000000126 substance Substances 0.000 description 3
- 238000010521 absorption reaction Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 2
- 230000002596 correlated effect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 229910052739 hydrogen Inorganic materials 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000009776 industrial production Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000010239 partial least squares discriminant analysis Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- -1 polyethylene terephthalate Polymers 0.000 description 2
- 238000013441 quality evaluation Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- UFHFLCQGNIYNRP-UHFFFAOYSA-N Hydrogen Chemical compound [H][H] UFHFLCQGNIYNRP-UHFFFAOYSA-N 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000012569 chemometric method Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 239000001257 hydrogen Substances 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000031700 light absorption Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 150000002894 organic compounds Chemical class 0.000 description 1
- 230000036961 partial effect Effects 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 239000004033 plastic Substances 0.000 description 1
- 229920003023 plastic Polymers 0.000 description 1
- 239000004417 polycarbonate Substances 0.000 description 1
- 229920000515 polycarbonate Polymers 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000013179 statistical model Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000011426 transformation method Methods 0.000 description 1
Abstract
The invention provides a Fourier near infrared interference signal virtual generation evaluation method and a system, wherein the method comprises the steps of obtaining an experimental sample and collecting interference signals of the experimental sample; smoothing the initial sample data to obtain sample real data; inputting the real data of the sample into a preset virtual generation model for data expansion, and taking the data obtained by data expansion as sample virtual data; the method expands the number of real samples by improving a traditional preset virtual generation model, enriches the characteristic information of the samples, fills the information gap between the samples, and simultaneously evaluates and analyzes the real data and the virtual data from the angles of similarity, rationality and diversity so as to be beneficial to subsequent sample analysis.
Description
Technical Field
The invention belongs to the technical field of signal generation and evaluation, and particularly relates to a Fourier near infrared interference signal virtual generation and evaluation method and system.
Background
Near infrared spectrum is widely applied to various fields in recent years due to the advantages of no damage, high analysis speed, environmental friendliness and the like, and mainly reflects the absorption characteristics of the combined frequency and the frequency multiplication of vibration of hydrogen-containing groups (C-H, N-H, O-H and the like) in different chemical environments. In general, the NIR technology needs to be combined with a chemometric method, and by constructing a robust multivariate model, the composition and properties of an organic compound are measured, which is a convenient measurement technology, and in general, a good model requires that a model training contains enough and representative samples, so that the number of samples has a great influence on the robustness and precision of the model, but in real life, scarce data are difficult to obtain, and in the case of fewer counterexample samples, the acquired data cannot completely and accurately express the feature space of the sample, and cannot meet the precision and applicability of data-driven modeling, so that in order to increase the robustness of the model, the construction of virtual samples is required to be carried out.
The generation of the countermeasure network method is generally used for the generation of the image, but in terms of the generation of near infrared spectrum data, near infrared spectrum is a tool for analyzing the composition of a substance, the spectral peak information of which is critical for determining the characteristics of the substance, however, unlike the image, the characteristics of near infrared spectrum are not visual characteristics, but data on the wavelength and absorption intensity, one of the reasons why the difficulty of generating near infrared spectrum is great is that the visual characteristics are insufficient. Typically, the near infrared spectrum has less information about the characteristic peaks, which makes it necessary to generate a model to overcome the lack of information. Because of the high dimensionality and complexity of spectral data, GAN models need to be able to capture and synthesize appropriate wavelength characteristics to generate meaningful spectral data, to overcome these challenges, this can be addressed by near infrared spectral information transformation and improving the architecture and training methods of the generation model, and by choosing reasonable generation evaluation criteria to ensure that the generated virtual sample matches the real sample.
In the prior art, however, the following disadvantages exist:
1. the virtual sample generation method requires that the real samples have a high dimensionality. Virtual sample generation methods are typically developed and optimized for high-dimensional data, such as images. There are indeed challenges and limitations to one-dimensional spectral signal generation, (1) limited information content. One-dimensional spectral signals typically contain relatively little information because they involve only light absorption or reflection data points, in contrast to two-dimensional images that contain rich spatial information, the limited nature of which can make it difficult to capture and synthesize complex features from the emerging patterns; (2) inadequate constraints. The generation of spectral signals is often subject to limited constraints. In image generation, rich visual features may be used to guide the generation of models. But in the spectral signal, there are fewer constraints, which makes it difficult for the model to obtain enough reference information to generate a high quality virtual signal; (3) slow convergence speed. Due to the limited information and the lack of constraint conditions, the one-dimensional spectral signal generation model may take longer to converge to a suitable result. This may require more training iterations and a more complex model architecture; (4) the production quality is low. Due to the above challenges, one-dimensional spectral signal generation models may generate virtual signals of lower quality, where noise, discontinuities, or unreasonable features may be present.
2. Conventional GAN suffers from a number of deficiencies in generating virtual samples. In processing an image generation task, the disadvantages of the conventional GAN method include: (1) training instability. Conventional GANs tend to be more prone to problems with training instability during the training process, such as pattern collapse (generator only generates similar samples) or gradient vanishing/gradient explosion problems. This results in difficulty in convergence of the generator (G) and the arbiter (D) to a proper state. (2) lack of structural features. Conventional GANs generate images without explicit structural constraints, and thus the generated samples may lack advanced semantic structures and specific visual features. (3) difficulty in processing complex data. Conventional GANs tend to perform poorly for processing complex high-dimensional data (e.g., high-resolution images or multi-channel data) because they have no specialized structure to handle these situations. This may lead to a reduced quality of the resulting sample. (4) more training data is required. Conventional GANs typically require more training data to achieve good generation performance. This may be a disadvantage for some applications because it may be difficult and expensive to obtain large-scale training data. (5) sensitivity to hyper-parameters. The performance of conventional GAN is very sensitive to the choice of hyper-parameters, including learning rate, model complexity, regularization parameters, etc. Therefore, more time and effort are required to adjust these super parameters.
3. The generated virtual sample and the real sample have single matching degree evaluation index. The evaluation criteria for the quality of the resulting samples are relatively single. Typically, the quality of the generated samples is evaluated mainly by the following criteria: (1) visual assessment. This is the most intuitive evaluation method, and people judge the quality by observing the generated image. High quality generated images are typically of authenticity, sharpness, detail and diversity. However, this evaluation method is subjective and is not easy to quantify. (2) fixing the statistical index. These indices are used to quantify the quality of the generated image, including pixel-level indices such as Mean Square Error (MSE) and Structural Similarity Index (SSIM), and deep learning-based indices such as acceptance Score and FID. However, these metrics typically capture only some aspects of the generated image, rather than evaluating its quality comprehensively. (3) manual evaluation. By obtaining feedback in a real human reviewer, the authenticity, diversity, and semantic rationality of the generated image can be more fully assessed. But this approach is time consuming, costly and inaccurate.
Disclosure of Invention
In order to solve the technical problems, the invention provides a Fourier near infrared interference signal virtual generation evaluation method and system, which are used for solving the technical problems in the prior art.
On one hand, the invention provides the following technical scheme, namely a Fourier near infrared interference signal virtual generation evaluation method, which comprises the following steps:
acquiring an experimental sample, acquiring an interference signal of the experimental sample by using an FT-NIR spectrometer, and deriving the interference signal of the experimental sample to obtain initial sample data;
smoothing the initial sample data by adopting a polynomial smoothing algorithm, and taking the smoothed data as sample real data;
inputting the sample real data into a preset virtual generation model for data expansion, and taking the data obtained by data expansion as sample virtual data;
performing data evaluation on the sample virtual data based on a preset evaluation algorithm to obtain an evaluation value of the sample virtual data, screening the sample virtual data based on the evaluation value to obtain available virtual data, and adding the available virtual data into the sample real data;
the step of smoothing the initial sample data by adopting a polynomial smoothing algorithm and taking the smoothed data as sample real data comprises the following steps:
performing polynomial decomposition on the data of the initial sample data in the moving window by adopting a preset smoothing formula, and performing data fitting on the data in the moving window by using a least square method to obtain sample real data, wherein the preset smoothing formula is as follows:
，/>；
In the method, in the process of the invention,for sample real data +.>For normalization factor->Is the midpoint wavelength point +.>The number of windows on the left and right sides, +.>Is->Observations of wavelength points, +.>Is a smoothing coefficient.
Compared with the prior art, the beneficial effects of this application are:
1. by generating the interference information, the problems of difficult generation and the like caused by insufficient peak position information of the spectrum signal and larger phase error are avoided, and the condition that the real sample has higher dimension is satisfied in the virtual sample generation method;
2. by improving the traditional GAN, a new virtual sample generation technology based on an interference signal combined CNN-GAN data enhancement method is provided, namely a preset virtual generation model is provided, the CNN-GAN data enhancement method is combined with interference signal spectrum data, so that higher-quality spectrum data can be generated, and the generalization capability and the discrimination accuracy of the model can be improved by carrying out sample generation through the preset virtual generation model;
3. performing quality evaluation on the virtual sample by adopting a pearson correlation coefficient, sample space distribution, data modeling and other sequential multi-loop evaluation criteria, and performing evaluation analysis on the virtual sample aiming at the similarity, rationality and diversity multi-angle of the generated interference signal and the real interference signal;
4. The preset virtual generation model can be directly applied to subsequent industrial production, a network is not required to be built again, repeated iterative training is not required, the method is friendly to users without deep learning basis, and time cost and labor force can be greatly saved.
Preferably, in the step of inputting the sample real data into a preset virtual generation model to perform data expansion, and taking the data obtained by data expansion as sample virtual data, the preset virtual model is:
；
in the method, in the process of the invention,for discriminator(s)>Generator (s)/(s)>For mathematical expectations +.>For sample real data +.>Representation ofDistribution of->Representation->Distribution derived from real data->Representing virtual data generated by the generator, < >>Representation->Distribution of->Representing the random noise signal of the input,/->Representing the probability that the virtual data belongs to the real data.
Preferably, the step of performing data evaluation on the sample virtual data based on a preset evaluation algorithm to obtain an evaluation value of the sample virtual data, and screening the sample virtual data based on the evaluation value to obtain available virtual data includes:
calculating a first evaluation value of data in the sample virtual data by adopting a PCCs algorithm, and carrying out first screening on the data in the sample virtual data based on the first evaluation value to obtain first screened virtual data;
Projecting the first screening virtual data and the sample real data into a two-dimensional space through a t-SNE algorithm, respectively determining a first confidence interval of the first screening virtual data and a second confidence interval of the sample real data, and carrying out second screening on the first screening virtual data based on the first confidence interval and the second confidence interval to obtain second screening virtual data;
and inputting the second screening virtual data and the sample real data into a preset identification model for third screening so as to obtain available virtual data.
Preferably, the step of calculating a first evaluation value of the data in the sample virtual data by using a PCCs algorithm, and performing a first screening on the data in the sample virtual data based on the first evaluation value to obtain first screened virtual data includes:
calculating a first evaluation value of data in the sample virtual data by a PCCs calculation formula:
；
in the method, in the process of the invention,for the first evaluation value, ++>Corresponding to sample real data +.>Point observations->Corresponding to sample dummy data->Point observations->For the number of samples +.>Mean value of sample real data, +. >Mean value of sample virtual data;
and removing the data with the first evaluation value smaller than the evaluation threshold value from the sample virtual data to obtain first screening virtual data.
Preferably, the projecting the first screening virtual data and the sample real data into a two-dimensional space through a t-SNE algorithm, and determining a first confidence interval of the first screening virtual data and a second confidence interval of the sample real data respectively, and performing a second screening on the first screening virtual data based on the first confidence interval and the second confidence interval, so as to obtain second screening virtual data includes:
selecting high-dimensional space outliers from the first screening virtual data and the sample real dataAnd is based on the high-dimensional spatial outlier +.>Calculating the probability of proximity to the data point:
；
；
in the method, in the process of the invention,for other data points +.>Is the variance of Gaussian distribution, +.>Representation->Is->Is used to determine the probability of a neighboring point,for high-dimensional data, ++>Representation->Is->Is>Total number of data points>Is->Is a joint probability distribution of (1);
initializing low-dimensional dataAnd calculating a joint probability distribution of the low-dimensional data:
；
in the method, in the process of the invention, 、/>、/>Respectively->、/>、/>Low dimensional representation of->Is->Is a joint probability distribution of (1);
iteratively updating the low-dimensional data based on joint probability distribution of the low-dimensional data and proximity probability of paired data points to obtain updated low-dimensional data:
；
；
；
in the method, in the process of the invention,as a cost function->、/>、/>Respectively +.>Second, th->Second, th->Low-dimensional data after a number of iterations, +.>For learning rate->Coefficients for momentum terms;
mapping the updated low-dimensional data into a two-dimensional space, respectively drawing a first confidence interval of the first screening virtual data and a second confidence interval of the sample real data, comparing the difference value of the first confidence interval and the second confidence interval to evaluate the quality of the first screening virtual data, and reserving the first screening virtual data in the first confidence interval to obtain second screening virtual data.
Preferably, in the step of inputting the second screening virtual data and the sample real data into a preset recognition model for performing a third screening to obtain the available virtual data, the preset recognition model is specifically a PLS-DA model.
In a second aspect, the present invention provides a fourier near-infrared interference signal virtual generation evaluation system, which includes:
The data acquisition module is used for acquiring an experimental sample, acquiring interference signals of the experimental sample by using an FT-NIR spectrometer, and deriving the interference signals of the experimental sample to obtain initial sample data;
the data processing module is used for carrying out smoothing processing on the initial sample data by adopting a polynomial smoothing algorithm so as to obtain sample real data;
the data expansion module is used for inputting the sample real data into a preset virtual generation model for data expansion, and taking the data obtained by data expansion as sample virtual data;
the data evaluation module is used for performing data evaluation on the sample virtual data based on a preset evaluation algorithm to obtain an evaluation value of the sample virtual data, screening the sample virtual data based on the evaluation value to obtain available virtual data, and adding the available virtual data into the sample real data;
the data processing module is specifically configured to:
performing polynomial decomposition on the data of the initial sample data in the moving window by adopting a preset smoothing formula, and performing data fitting on the data in the moving window by using a least square method to obtain sample real data, wherein the preset smoothing formula is as follows:
，/>；
In the method, in the process of the invention,for sample real data +.>For normalization factor->Is the midpoint wavelength point +.>The number of windows on the left and right sides, +.>Is->Observations of wavelength points, +.>Is a smoothing coefficient.
In a third aspect, the present invention provides a computer, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the processor implements the fourier near infrared interference signal virtual generation evaluation method as described above when executing the computer program.
In a fourth aspect, the present invention provides a storage medium having a computer program stored thereon, the computer program implementing the fourier near infrared interference signal virtual generation evaluation method as described above when executed by a processor.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a fourier near infrared interference signal virtual generation evaluation method according to an embodiment of the present invention;
fig. 2 is a detailed flowchart of step S4 in the fourier near infrared interference signal virtual generation evaluation method according to the first embodiment of the present invention;
fig. 3 is a detailed flowchart of step S41 in the fourier near infrared interference signal virtual generation evaluation method according to the first embodiment of the present invention;
FIG. 4 is a data distribution diagram of the sample virtual data after calculation in step S41;
fig. 5 is a detailed flowchart of step S42 in the fourier near infrared interference signal virtual generation evaluation method according to the first embodiment of the present invention;
FIG. 6 is a data distribution diagram of the first filtered virtual data and the sample real data after step S42;
FIG. 7 is a predicted data distribution diagram of a predetermined recognition model;
fig. 8 is a block diagram of a fourier near-infrared interference signal virtual generation evaluation system according to a second embodiment of the present invention;
fig. 9 is a schematic hardware structure of a computer according to another embodiment of the invention.
Embodiments of the present invention will be further described below with reference to the accompanying drawings.
Detailed Description
Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and intended to illustrate embodiments of the invention and should not be construed as limiting the invention.
In the description of the embodiments of the present invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," and the like indicate orientations or positional relationships based on the orientation or positional relationships shown in the drawings, merely to facilitate description of the embodiments of the present invention and simplify description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present invention.
Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless explicitly defined otherwise.
In the embodiments of the present invention, unless explicitly specified and limited otherwise, the terms "mounted," "connected," "secured" and the like are to be construed broadly and include, for example, either permanently connected, removably connected, or integrally formed; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communicated with the inside of two elements or the interaction relationship of the two elements. The specific meaning of the above terms in the embodiments of the present invention will be understood by those of ordinary skill in the art according to specific circumstances.
Example 1
In a first embodiment of the present invention, as shown in fig. 1, a method for evaluating virtual generation of fourier near infrared interference signals includes:
s1, acquiring an experimental sample, acquiring an interference signal of the experimental sample by using an FT-NIR spectrometer, and deriving the interference signal of the experimental sample to obtain initial sample data;
specifically, in this embodiment, the selected experimental samples are two large plastic particles, namely polyethylene terephthalate (polyethylene terephthalate, PET) and Polycarbonate (PC), the experimental samples are first labeled, then an FT-NIR spectrometer is used to collect interference signals of the experimental samples under the condition of ensuring proper temperature and humidity, and then OPUS software is used to derive the collected interference signal data for later use, so as to obtain initial sample data.
S2, smoothing the initial sample data by adopting a polynomial smoothing algorithm, and taking the smoothed data as sample real data;
specifically, step S2 is a preprocessing process of data to eliminate the influence of noise, baseline drift, etc. in the initial sample data.
The step S2 specifically includes:
performing polynomial decomposition on the data of the initial sample data in the moving window by adopting a preset smoothing formula, and performing data fitting on the data in the moving window by using a least square method to obtain sample real data, wherein the preset smoothing formula is as follows:
，/>；
In the method, in the process of the invention,for sample real data +.>For normalization factor->Is the midpoint wavelength point +.>The number of windows on the left and right sides, +.>Is->Observations of wavelength points, +.>Is a smoothing coefficient;
in this step, the polynomial smoothing algorithm is specifically an S-G convolution smoothing method, which mainly uses a polynomial to perform polynomial decomposition on data in a moving window (window width 2ω+1) of a real data spectrum and uses a least square method to perform data fitting, and is essentially a central function in spectrogram processing of a system expressing a center point of a window of a sample spectrogram by using a weighted average methodAlso for the initial sample data at wavelength +.>The average value of the data after smoothing treatment is the real data of the sample.
S3, inputting the sample real data into a preset virtual generation model for data expansion, and taking the data obtained by data expansion as sample virtual data;
specifically, in the step, a preset virtual generation model is specifically a CNN-GAN model, the original sample real data can be expanded by inputting the sample real data into the preset virtual generation model, and the expanded data is sample virtual data;
the preset virtual model is as follows:
；
In the method, in the process of the invention,for discriminator(s)>Generator (s)/(s)>For mathematical expectations +.>For sample real data +.>Representation ofDistribution of->Representation->Distribution derived from real data->Representing virtual data generated by the generator, < >>Representation->Distribution of (3)，/>Representing the random noise signal of the input,/->Representing the probability that the virtual data belongs to the real data;
wherein, the CNN-GAN model is a convolutional neural network-generating countermeasure network model, and the improvement of CNN-GAN is mainly based on the network structure, namely, on the discriminator #) Sum generator (+)>) The convolutional layer and the convolutional-transpose layer are explicitly used, the CNN-GAN greatly improves the stability of the real GAN training and the quality of the generated result, and the CNN-GAN greatly improves the quality of the generated result>Consists of a convolutional layer, a BN layer and a global pooling layer, using the leak Relu as an activation function for all layers,/a>The inputs of (a) are sample real data and sample virtual data, and finally an s-type activation function is used to distinguish whether they are true or false, +.>Consists of a convolution-transpose layer and a BN layer, using Relu as an activation function for all layers,/o>The input of (2) is Gaussian noise +.>(1 x 100), the output of which is sample dummy data.
S4, carrying out data evaluation on the sample virtual data based on a preset evaluation algorithm to obtain an evaluation value of the sample virtual data, screening the sample virtual data based on the evaluation value to obtain available virtual data, and adding the available virtual data into the sample real data;
As shown in fig. 2, the step S4 includes:
s41, calculating a first evaluation value of data in the sample virtual data by adopting a PCCs algorithm, and carrying out first screening on the data in the sample virtual data based on the first evaluation value to obtain first screened virtual data;
as shown in fig. 3, specifically, the step S41 specifically includes:
s411, calculating a first evaluation value of data in the sample virtual data by a PCCs calculation formula:
；
in the method, in the process of the invention,for the first evaluation value, ++>Corresponding to sample real data +.>Point observations->Corresponding to sample dummy data->Point observations->For the number of samples +.>Mean value of sample real data, +.>Mean value of sample virtual data;
the PCCs algorithm is specifically a Pearson correlation coefficient (Pearson correlation coefficient).
S412, eliminating the data with the first evaluation value smaller than the evaluation threshold value from the sample virtual data to obtain first screening virtual data;
wherein PCCs is a measure for measuring the degree of linear correlation between two variables, between [ -1,1], if PCCs E [ -1, 0), it is indicated that the two sample data are inversely correlated; pccs=0, indicating that the two sample data are uncorrelated, PCCs e (0, 1) indicating that the two sample data are positively correlated, typically, the correlation strength rule defining the variables is as follows, |pccs| >0.8, very strong correlation, 0.6< |pccs| <0.8, medium correlation, 0.2< |pccs| <0.4, weak correlation, |pccs| <0.2, very weak correlation or uncorrelated, whereas in this embodiment the evaluation threshold may be dependent on the number of samples remaining after subsequent screening;
As shown in fig. 4, fig. 4 is a data distribution diagram of sample virtual data after PCCs algorithm calculation, it can be seen from fig. 4 that a dotted line in the diagram is an evaluation threshold, a pearson correlation coefficient in an ordinate in the diagram is a first evaluation value, data below the dotted line in the diagram is output to be removed, and data above the dotted line including a position of the dotted line is first screening virtual data obtained after first screening.
S42, projecting the first screening virtual data and the sample real data into a two-dimensional space through a t-SNE algorithm, respectively determining a first confidence interval of the first screening virtual data and a second confidence interval of the sample real data, and carrying out second screening on the first screening virtual data based on the first confidence interval and the second confidence interval to obtain second screening virtual data;
the t-SNE algorithm is specifically a t-Distributed Stochastic Neighbor Embedding algorithm;
as shown in fig. 5, the step S42 includes:
s421, filtering virtual data and the sample true in the first filtering processSelecting high-dimensional space outliers from real dataAnd is based on the high-dimensional spatial outlier +.>Calculating the probability of proximity to the data point:
；
；
In the method, in the process of the invention,for other data points +.>Is the variance of Gaussian distribution, +.>Representation->Is->Is used to determine the probability of a neighboring point,for high-dimensional data, ++>Representation->Is->Is>Total number of data points>Is->Is described.
S422, initializing low-dimensional dataAnd calculating a joint probability distribution of the low-dimensional data:
；
in the method, in the process of the invention,、/>、/>respectively->、/>、/>Low dimensional representation of->Is->Is described.
S423, iteratively updating the low-dimensional data based on joint probability distribution of the low-dimensional data and adjacent probability of paired data points to obtain updated low-dimensional data:
；
；
；
in the method, in the process of the invention,as a cost function->、/>、/>Respectively +.>Second, th->Second, th->Low-dimensional data after a number of iterations, +.>For learning rate->Is a momentum term coefficient.
S424, mapping the updated low-dimensional data into a two-dimensional space, respectively drawing a first confidence interval of the first screening virtual data and a second confidence interval of the sample real data, comparing the difference value of the first confidence interval and the second confidence interval to evaluate the quality of the first screening virtual data, and reserving the first screening virtual data in the first confidence interval to obtain second screening virtual data;
Specifically, in this step, a second screening is performed on the first screened virtual data using a probability-based method to measure the similarity between the high-dimensional data points and to maintain the similarity as much as possible in the low-dimensional space, a probability distribution is calculated for each data point, which represents the similarity with other data points, an optimization method called KL-divergence (Kullback-Leibler Divergence) is used to measure the difference between the two probability distributions, and the gradient descent algorithm is used to minimize the difference;
after the high-dimensional data is projected to the two-dimensional space, a first confidence interval of the first screening virtual data and a second confidence interval of the sample real data can be respectively drawn, in the actual situation, the first confidence interval and the second confidence interval are elliptical areas, namely sample confidence ellipses, in the actual situation, the first confidence interval is in the range of the second confidence interval, so that in the actual situation, the quality of the first screening virtual data can be evaluated on the spatial distribution through the distribution of each data point and the difference between the two sample confidence ellipses, and as the sample data points are scattered in the two-dimensional space, part of the first screening virtual data can be distributed in the first confidence interval, and the other part of the first screening virtual data can be distributed outside the first confidence interval, and the sample quality of the data distributed in the first confidence interval is higher, so that the sample quality of the data distributed outside the first confidence interval is reserved and used as the second screening virtual data, and the sample quality of the data distributed outside the first confidence interval is lower, and therefore the sample quality of the data is required to be removed;
As shown in fig. 6, fig. 6 is a data distribution diagram of the first screening virtual data and the sample real data after the step S42, in fig. 6, the solid ellipse is a first confidence interval, the dotted ellipse is a second confidence interval, and in this step, the first screening virtual data in the second confidence interval needs to be retained, so as to obtain the first screening virtual data.
S43, inputting the second screening virtual data and the sample real data into a preset identification model for third screening so as to obtain available virtual data;
the preset recognition model is specifically a PLS-DA model, namely a partial least squares discriminant analysis (Partial least squares Discriminant Analysis, PLS-DA) model, the PLS-DA model is a supervised discriminant statistical model capable of distinguishing sample types and inter-group differences, and is essentially a feature transformation method, a new variable is a certain combination of real variables, and the effect of multiple collinearity among the variables can be effectively reduced by projecting high-dimension data into a hidden space through the partial least squares method. According to the method, the information of sample classification is considered in the characteristic selection process, so that the characteristics related to classification can be effectively extracted;
The second screening virtual data and the sample real data are input into the PLS-DA model, if the model identifies the second screening virtual data as the sample real data, the difference between the virtual data generated by the model and the real data is not large, and the model can be used, if the second screening virtual data is identified as the second screening virtual data, the sample real data is identified as the sample real data, the difference between the virtual data and the real data is large, the quality is not high, the identification accuracy of the model can not be improved, in this embodiment, the identification accuracy of the model is set to be 10%, if the accuracy of the PLS-DA model is larger than 10%, the model is invalid, more virtual generation mixed into the sample real data can be identified, the difference between the generated data and the real data is large, the purpose of spurious reality can not be achieved, if the identification accuracy is larger than 10%, the evaluation threshold in the step S41 is required to be increased, or the confidence ellipse difference range in the step S42 is reduced, the sample with higher similarity to the original data is reserved, and the virtual data can be repeatedly executed to judge the second sample data.
As shown in fig. 7, the category of the second screening virtual data, which is the virtual data, is marked as 0, and the sample real data is marked as 1, if the PLS-DA model predicts the second screening virtual data as the sample real data, i.e., predicts the "1" category as the "0" category, which indicates that the generated second screening virtual data is relatively similar to the sample real data, the effect of spurious can be achieved; if the discrimination model correctly identifies the second screening virtual data and the sample real data, namely, predicting the class "1" as the class "1" and predicting the class "0" as the class "0", the difference between the second screening virtual data generated by the generation model and the sample real data is larger, the generation quality is not high, and the discrimination accuracy of the model cannot be improved, so that the second screening virtual data judged as the sample real data is reserved, and the available virtual data can be obtained.
According to the Fourier near infrared interference signal virtual generation evaluation method provided by the embodiment of the invention, by generating the interference information, the problems of difficult generation and the like caused by insufficient peak position information of a spectrum signal and larger phase error are avoided, and the condition that a real sample has higher dimension is satisfied in the virtual sample generation method; by improving the traditional GAN, a new virtual sample generation technology based on an interference signal combined CNN-GAN data enhancement method is provided, namely a preset virtual generation model is provided, the CNN-GAN data enhancement method is combined with interference signal spectrum data, so that higher-quality spectrum data can be generated, and the generalization capability and the discrimination accuracy of the model can be improved by carrying out sample generation through the preset virtual generation model; performing quality evaluation on the virtual sample by adopting a pearson correlation coefficient, sample space distribution, data modeling and other sequential multi-loop evaluation criteria, and performing evaluation analysis on the virtual sample aiming at the similarity, rationality and diversity multi-angle of the generated interference signal and the real interference signal; the preset virtual generation model can be directly applied to subsequent industrial production, a network is not required to be built again, repeated iterative training is not required, the method is friendly to users without deep learning basis, and time cost and labor force can be greatly saved.
Example two
As shown in fig. 8, in a second embodiment of the present invention, there is provided a fourier near infrared interference signal virtual generation evaluation system, the system including:
the data acquisition module 1 is used for acquiring an experimental sample, acquiring an interference signal of the experimental sample by using an FT-NIR spectrometer, and deriving the interference signal of the experimental sample to obtain initial sample data;
the data processing module 2 is used for carrying out smoothing processing on the initial sample data by adopting a polynomial smoothing algorithm, and taking the smoothed data as sample real data;
the data expansion module 3 is used for inputting the sample real data into a preset virtual generation model for data expansion, and taking the data obtained by data expansion as sample virtual data;
and the data evaluation module 4 is used for performing data evaluation on the sample virtual data based on a preset evaluation algorithm to obtain an evaluation value of the sample virtual data, screening the sample virtual data based on the evaluation value to obtain available virtual data, and adding the available virtual data into the sample real data.
Wherein, the data processing module 2 is specifically configured to:
performing polynomial decomposition on the data of the initial sample data in the moving window by adopting a preset smoothing formula, and performing data fitting on the data in the moving window by using a least square method to obtain sample real data, wherein the preset smoothing formula is as follows:
，/>；
In the method, in the process of the invention,for sample real data +.>For normalization factor->Is the midpoint wavelength point +.>The number of windows on the left and right sides, +.>Is->Observations of wavelength points, +.>Is a smoothing coefficient.
The data evaluation module 4 includes:
the first evaluation submodule is used for calculating a first evaluation value of data in the sample virtual data by adopting a PCCs algorithm, and carrying out first screening on the data in the sample virtual data based on the first evaluation value so as to obtain first screening virtual data;
the second evaluation sub-module is used for projecting the first screening virtual data and the sample real data into a two-dimensional space through a t-SNE algorithm, respectively determining a first confidence interval of the first screening virtual data and a second confidence interval of the sample real data, and carrying out second screening on the first screening virtual data based on the first confidence interval and the second confidence interval to obtain second screening virtual data;
and the third evaluation sub-module is used for inputting the second screening virtual data and the sample real data into a preset identification model for third screening so as to obtain available virtual data.
The first evaluation submodule includes:
An evaluation value calculation unit for calculating a first evaluation value of data in the sample virtual data by a PCCs calculation formula:
；
in the method, in the process of the invention,for the first evaluation value，/>Corresponding to sample real data +.>Point observations->Corresponding to sample dummy data->Point observations->For the number of samples +.>Mean value of sample real data, +.>Mean value of sample virtual data;
and the rejecting unit is used for rejecting the data with the first evaluation value smaller than the evaluation threshold value from the sample virtual data so as to obtain first screening virtual data.
The second evaluation submodule includes:
a first calculation unit for selecting high-dimensional space outliers from the first screening virtual data and the sample real dataAnd is based on the high-dimensional spatial outlier +.>Calculating the probability of proximity to the data point:
；
；
in the method, in the process of the invention,for other data points +.>Is the variance of Gaussian distribution, +.>Representation->Is->Is used to determine the probability of a neighboring point,for high-dimensional data, ++>Representation->Is->Is>Total number of data points>Is->Is a joint probability distribution of (1);
a second calculation unit for initializing low-dimensional dataAnd calculating a joint probability distribution of the low-dimensional data:
；
In the method, in the process of the invention,、/>、/>respectively->、/>、/>Low dimensional representation of->Is->Is a joint probability distribution of (1);
an updating unit, configured to iteratively update the low-dimensional data based on a joint probability distribution of the low-dimensional data and a proximity probability of paired data points, so as to obtain updated low-dimensional data:
；
；
；
in the method, in the process of the invention,as a cost function->、/>、/>Respectively +.>Second, th->Second, th->Low-dimensional data after a number of iterations, +.>For learning rate->Coefficients for momentum terms;
and the screening unit is used for mapping the updated low-dimensional data into a two-dimensional space and respectively drawing a first confidence interval of the first screening virtual data and a second confidence interval of the sample real data, comparing the difference value of the first confidence interval and the second confidence interval to evaluate the quality of the first screening virtual data and reserving the first screening virtual data in the first confidence interval so as to obtain second screening virtual data.
In other embodiments of the present invention, a computer is provided in the embodiments of the present invention, which includes a memory 102, a processor 101, and a computer program stored in the memory 102 and executable on the processor 101, where the processor 101 implements the fourier near infrared interference signal virtual generation evaluation method as described above when executing the computer program.
In particular, the processor 101 may include a Central Processing Unit (CPU), or an application specific integrated circuit (Application Specific Integrated Circuit, abbreviated as ASIC), or may be configured to implement one or more integrated circuits of embodiments of the present application.
Memory 102 may include, among other things, mass storage for data or instructions. By way of example, and not limitation, memory 102 may comprise a Hard Disk Drive (HDD), floppy Disk Drive, solid state Drive (Solid State Drive, SSD), flash memory, optical Disk, magneto-optical Disk, tape, or universal serial bus (Universal Serial Bus, USB) Drive, or a combination of two or more of the foregoing. Memory 102 may include removable or non-removable (or fixed) media, where appropriate. The memory 102 may be internal or external to the data processing apparatus, where appropriate. In a particular embodiment, the memory 102 is a Non-Volatile (Non-Volatile) memory. In a particular embodiment, the Memory 102 includes Read-Only Memory (ROM) and random access Memory (Random Access Memory, RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, abbreviated PROM), an erasable PROM (Erasable Programmable Read-Only Memory, abbreviated EPROM), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, abbreviated EEPROM), an electrically rewritable ROM (Electrically Alterable Read-Only Memory, abbreviated EAROM), or a FLASH Memory (FLASH), or a combination of two or more of these. The RAM may be Static Random-Access Memory (SRAM) or dynamic Random-Access Memory (Dynamic Random Access Memory DRAM), where the DRAM may be a fast page mode dynamic Random-Access Memory (Fast Page Mode Dynamic Random Access Memory FPMDRAM), extended data output dynamic Random-Access Memory (Extended Date Out Dynamic Random Access Memory EDODRAM), synchronous dynamic Random-Access Memory (Synchronous Dynamic Random-Access Memory SDRAM), or the like, as appropriate.
Memory 102 may be used to store or cache various data files that need to be processed and/or communicated, as well as possible computer program instructions for execution by processor 101.
The processor 101 reads and executes the computer program instructions stored in the memory 102 to implement the above-described fourier near infrared interference signal virtual generation evaluation method.
In some of these embodiments, the computer may also include a communication interface 103 and a bus 100. As shown in fig. 9, the processor 101, the memory 102, and the communication interface 103 are connected to each other via the bus 100 and perform communication with each other.
The communication interface 103 is used to implement communication between modules, devices, units, and/or units in the embodiments of the present application. The communication interface 103 may also enable communication with other components such as: and the external equipment, the image/data acquisition equipment, the database, the external storage, the image/data processing workstation and the like are used for data communication.
Bus 100 includes hardware, software, or both, coupling components of a computer device to each other. Bus 100 includes, but is not limited to, at least one of: data Bus (Data Bus), address Bus (Address Bus), control Bus (Control Bus), expansion Bus (Expansion Bus), local Bus (Local Bus). By way of example, and not limitation, bus 100 may include a graphics acceleration interface (Accelerated Graphics Port), abbreviated AGP, or other graphics Bus, an enhanced industry standard architecture (Extended Industry Standard Architecture, abbreviated EISA) Bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an industry standard architecture (Industry Standard Architecture, ISA) Bus, a wireless bandwidth (InfiniBand) interconnect, a Low Pin Count (LPC) Bus, a memory Bus, a micro channel architecture (Micro Channel Architecture, abbreviated MCa) Bus, a peripheral component interconnect (Peripheral Component Interconnect, abbreviated PCI) Bus, a PCI-Express (PCI-X) Bus, a serial advanced technology attachment (Serial Advanced Technology Attachment, abbreviated SATA) Bus, a video electronics standards association local (Video Electronics Standards Association Local Bus, abbreviated VLB) Bus, or other suitable Bus, or a combination of two or more of the foregoing. Bus 100 may include one or more buses, where appropriate. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.
The computer can execute the Fourier near infrared interference signal virtual generation evaluation method based on the obtained Fourier near infrared interference signal virtual generation evaluation system, so that virtual generation and evaluation of interference signals are realized.
In still other embodiments of the present invention, in combination with the above-mentioned fourier near infrared interference signal virtual generation evaluation method, embodiments of the present invention provide a technical solution, a storage medium storing a computer program, where the computer program when executed by a processor implements the above-mentioned fourier near infrared interference signal virtual generation evaluation method.
Those of skill in the art will appreciate that the logic and/or steps represented in the flow diagrams or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.
It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.
The technical features of the above-described embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above-described embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples merely represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the invention. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application is to be determined by the claims appended hereto.
Claims (6)
1. The method for evaluating the virtual generation of the Fourier near infrared interference signal is characterized by comprising the following steps of:
acquiring an experimental sample, acquiring an interference signal of the experimental sample by using an FT-NIR spectrometer, and deriving the interference signal of the experimental sample to obtain initial sample data;
smoothing the initial sample data by adopting a polynomial smoothing algorithm, and taking the smoothed data as sample real data;
Inputting the sample real data into a preset virtual generation model for data expansion, and taking the data obtained by data expansion as sample virtual data;
performing data evaluation on the sample virtual data based on a preset evaluation algorithm to obtain an evaluation value of the sample virtual data, screening the sample virtual data based on the evaluation value to obtain available virtual data, and adding the available virtual data into the sample real data;
the step of smoothing the initial sample data by adopting a polynomial smoothing algorithm and taking the smoothed data as sample real data comprises the following steps:
performing polynomial decomposition on the data of the initial sample data in the moving window by adopting a preset smoothing formula, and performing data fitting on the data in the moving window by using a least square method to obtain sample real data, wherein the preset smoothing formula is as follows:
，/>；
in the method, in the process of the invention,for sample real data +.>For normalization factor->Is the midpoint wavelength point +.>The number of windows on the left and right sides, +.>Is->Observations of wavelength points, +.>Is a smoothing coefficient;
in the step of inputting the sample real data into a preset virtual generation model for data expansion, and taking the data obtained by data expansion as sample virtual data, the preset virtual model is as follows:
；
In the method, in the process of the invention,for discriminator(s)>Generator (s)/(s)>For mathematical expectations +.>For sample real data +.>Representation->Distribution of->Representation->Distribution derived from real data->Representing virtual data generated by the generator, < >>Representation ofDistribution of->Representing the random noise signal of the input,/->Representing the probability that the virtual data belongs to the real data;
the step of performing data evaluation on the sample virtual data based on a preset evaluation algorithm to obtain an evaluation value of the sample virtual data, and screening the sample virtual data based on the evaluation value to obtain available virtual data includes:
calculating a first evaluation value of data in the sample virtual data by adopting a PCCs algorithm, and carrying out first screening on the data in the sample virtual data based on the first evaluation value to obtain first screened virtual data;
projecting the first screening virtual data and the sample real data into a two-dimensional space through a t-SNE algorithm, respectively determining a first confidence interval of the first screening virtual data and a second confidence interval of the sample real data, and carrying out second screening on the first screening virtual data based on the first confidence interval and the second confidence interval to obtain second screening virtual data;
Inputting the second screening virtual data and the sample real data into a preset identification model for third screening to obtain available virtual data;
and in the step of inputting the second screening virtual data and the sample real data into a preset identification model for third screening to obtain available virtual data, the preset identification model is specifically a PLS-DA model.
2. The method for virtually generating and evaluating the near infrared fourier interference signal according to claim 1, wherein the step of calculating a first evaluation value of the data in the sample virtual data using the PCCs algorithm and performing a first screening on the data in the sample virtual data based on the first evaluation value to obtain first screened virtual data comprises:
calculating a first evaluation value of data in the sample virtual data by a PCCs calculation formula:
；
in the method, in the process of the invention,for the first evaluation value, ++>Corresponding to sample real data +.>Point observations->Corresponding to sample dummy data->Point observations->For the sampleQuantity of->Mean value of sample real data, +.>Mean value of sample virtual data;
and removing the data with the first evaluation value smaller than the evaluation threshold value from the sample virtual data to obtain first screening virtual data.
3. The method for evaluating virtual generation of near infrared interference signals according to claim 2, wherein the step of projecting the first screening virtual data and the sample real data into a two-dimensional space by a t-SNE algorithm and determining a first confidence interval of the first screening virtual data and a second confidence interval of the sample real data, respectively, and performing a second screening on the first screening virtual data based on the first confidence interval and the second confidence interval to obtain second screening virtual data comprises:
selecting high-dimensional space outliers from the first screening virtual data and the sample real dataAnd is based on the high-dimensional spatial outlier +.>Calculating the probability of proximity to the data point:
；
；
in the method, in the process of the invention,for other data points +.>Is the variance of Gaussian distribution, +.>Representation->Is->Is>For high-dimensional data, ++>Representation->Is->Is>Total number of data points>Is->Is a joint probability distribution of (1);
initializing low-dimensional dataAnd calculating a joint probability distribution of the low-dimensional data:
；
in the method, in the process of the invention,、/>、/>respectively->、/>、/>Low dimensional representation of->Is->Is a joint probability distribution of (1);
Iteratively updating the low-dimensional data based on joint probability distribution of the low-dimensional data and proximity probability of paired data points to obtain updated low-dimensional data:
；
；
；
in the method, in the process of the invention,as a cost function->、/>、/>Respectively +.>Second, th->Second, th->Low-dimensional data after a number of iterations, +.>For learning rate->Coefficients for momentum terms;
mapping the updated low-dimensional data into a two-dimensional space, respectively drawing a first confidence interval of the first screening virtual data and a second confidence interval of the sample real data, comparing the difference value of the first confidence interval and the second confidence interval to evaluate the quality of the first screening virtual data, and reserving the first screening virtual data in the first confidence interval to obtain second screening virtual data.
4. A fourier near infrared interferometry signal virtual generation evaluation system, the system comprising:
the data acquisition module is used for acquiring an experimental sample, acquiring interference signals of the experimental sample by using an FT-NIR spectrometer, and deriving the interference signals of the experimental sample to obtain initial sample data;
the data processing module is used for carrying out smoothing processing on the initial sample data by adopting a polynomial smoothing algorithm so as to obtain sample real data;
The data expansion module is used for inputting the sample real data into a preset virtual generation model for data expansion, and taking the data obtained by data expansion as sample virtual data;
the data evaluation module is used for performing data evaluation on the sample virtual data based on a preset evaluation algorithm to obtain an evaluation value of the sample virtual data, screening the sample virtual data based on the evaluation value to obtain available virtual data, and adding the available virtual data into the sample real data;
the preset virtual model is as follows:
；
in the method, in the process of the invention,for discriminator(s)>Generator (s)/(s)>For mathematical expectations +.>For sample real data +.>Representation->Distribution of->Representation->Distribution derived from real data->Representing virtual data generated by the generator, < >>Representation ofDistribution of->Representing the random noise signal of the input,/->Representing the probability that the virtual data belongs to the real data;
the data processing module is specifically configured to:
performing polynomial decomposition on the data of the initial sample data in the moving window by adopting a preset smoothing formula, and performing data fitting on the data in the moving window by using a least square method to obtain sample real data, wherein the preset smoothing formula is as follows:
，/>；
In the method, in the process of the invention,for sample real data +.>For normalization factor->Is the midpoint wavelength point +.>The number of windows on the left and right sides, +.>Is->Observations of wavelength points, +.>Is a smoothing coefficient;
wherein, the data evaluation module includes:
the first evaluation submodule is used for calculating a first evaluation value of data in the sample virtual data by adopting a PCCs algorithm, and carrying out first screening on the data in the sample virtual data based on the first evaluation value so as to obtain first screening virtual data;
the second evaluation sub-module is used for projecting the first screening virtual data and the sample real data into a two-dimensional space through a t-SNE algorithm, respectively determining a first confidence interval of the first screening virtual data and a second confidence interval of the sample real data, and carrying out second screening on the first screening virtual data based on the first confidence interval and the second confidence interval to obtain second screening virtual data;
the third evaluation sub-module is used for inputting the second screening virtual data and the sample real data into a preset identification model for third screening so as to obtain available virtual data;
the preset recognition model is specifically a PLS-DA model.
5. A computer comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the fourier near infrared interferometry signal virtual generation evaluation method according to any of claims 1 to 3 when the computer program is executed.
6. A storage medium having stored thereon a computer program which, when executed by a processor, implements the fourier near infrared interferometry signal virtual generation evaluation method of any of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311508216.4A CN117235464B (en) | 2023-11-14 | 2023-11-14 | Fourier near infrared interference signal virtual generation evaluation method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311508216.4A CN117235464B (en) | 2023-11-14 | 2023-11-14 | Fourier near infrared interference signal virtual generation evaluation method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117235464A CN117235464A (en) | 2023-12-15 |
| CN117235464B true CN117235464B (en) | 2024-02-23 |
Family
ID=89096973
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311508216.4A Active CN117235464B (en) | 2023-11-14 | 2023-11-14 | Fourier near infrared interference signal virtual generation evaluation method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117235464B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112949846A (en) * | 2021-03-26 | 2021-06-11 | 电子科技大学 | Method for constructing generated deception jamming signal suitable for direct sequence spread spectrum system |
| CN113114408A (en) * | 2021-04-06 | 2021-07-13 | 电子科技大学 | Method and device for generating and evaluating generated deception jamming signal |
| CN113642666A (en) * | 2021-08-29 | 2021-11-12 | 浙江工业大学 | Active enhanced soft measurement method based on sample expansion and screening |
| WO2022083536A1 (en) * | 2020-10-21 | 2022-04-28 | 华为技术有限公司 | Neural network construction method and apparatus |
| CN114882955A (en) * | 2022-04-08 | 2022-08-09 | 广州国家实验室 | Transcriptome image generation device, method and application |
| CN116415652A (en) * | 2023-03-29 | 2023-07-11 | 深圳市优必选科技股份有限公司 | Data generation method and device, readable storage medium and terminal equipment |
-
2023
- 2023-11-14 CN CN202311508216.4A patent/CN117235464B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022083536A1 (en) * | 2020-10-21 | 2022-04-28 | 华为技术有限公司 | Neural network construction method and apparatus |
| CN112949846A (en) * | 2021-03-26 | 2021-06-11 | 电子科技大学 | Method for constructing generated deception jamming signal suitable for direct sequence spread spectrum system |
| CN113114408A (en) * | 2021-04-06 | 2021-07-13 | 电子科技大学 | Method and device for generating and evaluating generated deception jamming signal |
| CN113642666A (en) * | 2021-08-29 | 2021-11-12 | 浙江工业大学 | Active enhanced soft measurement method based on sample expansion and screening |
| CN114882955A (en) * | 2022-04-08 | 2022-08-09 | 广州国家实验室 | Transcriptome image generation device, method and application |
| CN116415652A (en) * | 2023-03-29 | 2023-07-11 | 深圳市优必选科技股份有限公司 | Data generation method and device, readable storage medium and terminal equipment |
Non-Patent Citations (4)
| Title |
|---|
| AN ADAPTATION OF CNN FOR SMALL TARGET DETECTION IN THE INFRARED;Zhao, D et al.;IGARSS 2018 - 2018 IEEE INTERNATIONAL GEOSCIENCE AND REMOTE SENSING SYMPOSIUM;全文 * |
| Evaluation for infrared scene simulation based on self-learning framework;Fang Hao et al.;Systems Engineering and Electronics;全文 * |
| 基于SERED-GAN的多时段红外图像生成方法研究;王祯瑞;中国优秀硕士学位论文集信息科技辑;全文 * |
| 基于虚拟样本生成技术的多组分机械信号建模;汤健;乔俊飞;柴天佑;刘卓;吴志伟;;自动化学报(第09期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117235464A (en) | 2023-12-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Gutmann et al. | Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. | |
| US9898649B2 (en) | Face authentication method and device | |
| Chen et al. | SmartTracing: self-learning-based neuron reconstruction | |
| Silva et al. | Time series analysis via network science: Concepts and algorithms | |
| Ebbels et al. | Processing and modeling of nuclear magnetic resonance (NMR) metabolic profiles | |
| CN109934278B (en) | High-dimensionality feature selection method for information gain mixed neighborhood rough set | |
| CN116595463B (en) | Construction method of electricity larceny identification model, and electricity larceny behavior identification method and device | |
| Tompkins et al. | Fourier feature approximations for periodic kernels in time-series modelling | |
| US11666286B2 (en) | Classification of polyps using learned image analysis | |
| Silva et al. | How to produce complementary explanations using an ensemble model | |
| Betancourt | Calibrating model-based inferences and decisions | |
| CN104616022A (en) | Classification method of near infrared spectrum | |
| Wendt | Contributions of Wavelet Leaders and Bootstrap to Multifractal Analysis: Images, Estimation Performance, Dependence Structure and Vanishing Moments. Confidence Intervals and Hypothesis Tests. | |
| CN117235464B (en) | Fourier near infrared interference signal virtual generation evaluation method and system | |
| Pei et al. | Effective algorithm for determining the number of clusters and its application in image segmentation | |
| Koul et al. | Machine Learning and Biomedical Sub-Terahertz/Terahertz Technology | |
| Kaya et al. | SAGA: A novel signal alignment method based on genetic algorithm | |
| Chen et al. | Launching drifter observations in the presence of uncertainty | |
| Monti et al. | Decoding time-varying functional connectivity networks via linear graph embedding methods | |
| CN106485286B (en) | Matrix classification model based on local sensitivity discrimination | |
| CN115455407A (en) | Machine learning-based GitHub sensitive information leakage monitoring method | |
| Ray et al. | Inferring the shape of data: a probabilistic framework for analysing experiments in the natural sciences | |
| Lee et al. | Probabilistic orthogonal-signal-corrected principal component analysis | |
| Ramirez-Hereza et al. | Gaussianization of LA-ICP-MS features to improve calibration in forensic glass comparison | |
| Mu | Feature network methods for machine learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |