WO2011087083A1

WO2011087083A1 - Data processing method, data processing device and data processing program

Info

Publication number: WO2011087083A1
Application number: PCT/JP2011/050534
Authority: WO
Inventors: 茂樹広林; 貴博加古
Original assignee: 国立大学法人富山大学
Priority date: 2010-01-14
Filing date: 2011-01-14
Publication date: 2011-07-21

Abstract

Provided is a data processing device in which, without the occurrence of block noise, each type of data can be analyzed and interpolated and/or extrapolated, and, post-interpolated and/or post-extrapolation data can be considerably natural. When the original image data, which becomes the processing target, is input, the data processing device calculates a frequency f', an amplitude A' and an initial phase φ as acyclic Fourier transform parameters such that the sum of squares of the difference between an arbitrary dimensional signal based on the original data and a sinusoidal model signal represented by the amplitude A' and a phase using the frequency f' and the initial phase φ, becomes the least value, and extracts a spectrum. The data processing device interpolates and/or extrapolates the data on the basis of the extracted spectrum and calculates the reconstruction data.

Description

Data processing method, data processing apparatus, and data processing program

The present invention relates to a data processing method, a data processing apparatus, and a data processing program for analyzing and processing various types of data such as image data, and in particular, correcting missing portions of data by data interpolation processing and / or extrapolation processing. The present invention relates to a data processing method, a data processing apparatus, and a data processing program.

For example, when a subject is imaged, an area with high brightness in the obtained image data causes so-called “overexposed” due to an excessive aperture of the imaging device, and vice versa. In a region with low lightness in image data, so-called “blackout” or the like may occur, and information that is inherently lost may be lost. In particular, in the image processing technique, a process of interpolating missing information based on adjacent macroblocks to generate a predicted image or performing various image corrections is performed during compression encoding. ing.

As such an interpolation technique, for example, there is a technique to which an interpolation method is applied like the techniques described in Patent Documents 1 to 6 and the like.

Patent Document 1 discloses a technique for reducing image quality degradation due to judder at a low cost in an image interpolation device that converts a frame frequency using an interpolated image generated from an original image.

Further, Patent Document 2 discloses a technique for appropriately generating pixels in any region of an image of an interpolation frame when performing frame interpolation on a video signal.

Furthermore, in Patent Document 3, when interpolating an interpolated frame on an interpolated frame surface between temporally adjacent frames, there is no gap or overlap between images in the interpolated frame, and further block distortion occurs. A technique for reducing the above is disclosed.

Furthermore, Patent Document 4 discloses a technique for expanding the resolution by interpolating between two points inside the frame subjected to frequency analysis.

Patent Document 5 discloses a technique for applying an interpolation method when creating a three-dimensional color information conversion table for color-converting RGB signal processing system image information to YMCK signal processing system image information. Yes.

Furthermore, Patent Document 6 discloses a technique for applying an interpolation method when smoothing the periphery of a texture image.

In addition, as an interpolation technique, there is a technique that applies an extrapolation method, such as the technique described in Patent Document 7 to Patent Document 12, for example.

Patent Document 7 discloses a technique for linearly interpolating the total code amount calculated for each quantization step in order to realize image compression that can efficiently control the total code amount of the entire frame. .

Also, Patent Document 8 discloses a technique for efficiently generating a prediction signal for an image having a complex pattern when generating an intra-screen prediction signal by applying an extrapolation method.

Furthermore, in Patent Document 9, when an input image is divided into block units and a difference from a predicted image generated according to a prediction mode selected by intra prediction is encoded, pixel values are extrapolated. A technique for generating a predicted image is disclosed.

Furthermore, Patent Document 10 discloses a technique for removing block artifacts from a compressed image by extrapolation.

Patent Document 11 discloses a technique for obtaining a texture map value using extrapolation instead of a mipmap that cannot be used when mapping a texture image.

Further, Patent Document 12 discloses a technique for performing extrapolation processing when mapping a texture image and performing image correction processing.

Furthermore, Patent Document 5 described above also discloses a technique for applying an extrapolation method when creating a three-dimensional color information conversion table for color-converting RGB signal processing system image information into YMCK signal processing system image information. In addition, Patent Document 6 described above also discloses a technique for applying an extrapolation method when smoothing the periphery of a texture image.

Also, when a subject is imaged, an unnatural image may be obtained due to a small dynamic range. In application programs and the like currently on the market, gamma correction is generally used to correct such unnaturalness of the image. Many existing standards describe images with 8 bits per color, but in recent years, standards with more than 8 bits, such as JPEG XR, have appeared. There is also a demand for extending the dynamic range of an already captured image.

JP 2009-253626 A Japanese Patent Laid-Open No. 2009-200760 JP 2004-357215 A JP 2000-299862 A JP 2005-354219 A Japanese translation of PCT publication No. 2003-504697 JP 2008-42943 A JP 2007-300380 A JP 2006-295408 A Japanese National Patent Publication No. 11-504173 JP 2008-305408 A JP 2004-206672 A

However, the conventional techniques related to interpolation described in Patent Document 1 to Patent Document 6 described above are linear interpolation, cubic convolution interpolation, and interpolation using vector information, and the range in which interpolation is possible is limited. There is a problem that block noise occurs because it is not suitable for predicting a wide range.

In addition, the conventional technique related to extrapolation described in Patent Document 5 and Patent Document 7 to Patent Document 10 described above applies a quadratic linear extrapolation or an extrapolation method using other functions. Therefore, there is a problem that the extrapolable range is limited.

Furthermore, the conventional techniques described in Patent Document 6, Patent Document 11, and Patent Document 12 described above are techniques for extrapolating high frequency components in the frequency domain in order to obtain high resolution. In order to improve the performance, prediction is performed from adjacent macroblocks, and the error is encoded. This is not suitable for predicting a wide range, and there is a problem that block noise occurs.

Here, there is also a technique to reduce block noise by correcting the spectrum obtained by Fourier transform of image data. Discrete Fourier transform (DFT) and discrete cosine transform widely used in the engineering field. It is applied to extrapolation processing using (Discrete Cosine Transform; DCT). There is also a technique for extracting an optimum frequency component by performing Fourier transform and performing inverse Fourier transform on the obtained image, as in the technique described in Japanese Patent Application Laid-Open No. 2007-188721.

However, in such processing based on Fourier transform, only a signal having a frequency that is an integral multiple of f _s / k (f _s is the sampling frequency, k is the analysis window length) can be accurately analyzed. When a signal having a frequency other than is analyzed, there is a problem that an error occurs due to spectrum leakage. In the process based on the Fourier transform, since it is assumed that the analyzed frame is repeated infinitely, the same information as the analyzed frame is also interpolated or extrapolated in the adjacent frame. Therefore, in such a technique, there is a problem that an error occurs between frames when interpolation or extrapolation is performed.

The present invention has been made in view of such circumstances, and can analyze various data without generating block noise, and can perform interpolation and / or extrapolation over a wide range. Alternatively, it is an object to provide a data processing method, a data processing device, and a data processing program that can make data after extrapolation significantly natural.

Some of the inventors of the present application disclosed Non-Harmonic Analysis (NHA), which is an aperiodic signal analysis technique, as a new frequency analysis technique in International Publication No. 2009/038056. This NHA has a frequency f ′ at which the sum of squares of the difference between the signal to be analyzed and the sine wave model signal represented by the phase using the frequency f ′ and the initial phase φ ′ and the amplitude A ′ becomes a minimum value. , Amplitude A ′, and initial phase φ ′ are calculated as parameters of the Fourier transform equation of the aperiodic signal. The present invention uses NHA having a high frequency resolution and a small influence on the analysis window length, thereby accurately extracting and interpolating and / or extrapolating the spectrum of the cut out data. The technology that essentially solves the problem is established.

That is, the data processing method according to the present invention that achieves the above-described object is a data processing method that analyzes and processes various data, and inputs the original data to be processed to the data processing device and stores it in the memory. A data input step and an arithmetic means of the data processing device reads the original data inputted in the data input step and stored in the memory, and an arbitrary dimension signal based on the original data, a frequency f ′ and an initial value The frequency f ′, the amplitude A ′, and the initial phase φ ′ at which the sum of squares of the difference between the phase using the phase φ ′ and the sinusoidal model signal represented by the amplitude A ′ is minimized. The spectrum is extracted as a Fourier transform parameter of the aperiodic signal, the data is interpolated and / or extrapolated based on the extracted spectrum, and the reconstructed data is extracted. Including an interpolation and / or extrapolation step for obtaining the data.

A data processing apparatus according to the present invention that achieves the above-described object is a data processing apparatus that analyzes and processes various data, and includes data input means for inputting original data to be processed, and the data input means. The sum of squares of the difference between the arbitrary dimension signal based on the original data inputted via the sine wave model signal represented by the phase using the frequency f ′ and the initial phase φ ′ and the amplitude A ′ is the minimum value A spectrum is extracted by obtaining the frequency f ′, the amplitude A ′, and the initial phase φ ′ as follows as parameters of a Fourier transform formula of an aperiodic signal, and data interpolation processing is performed based on the extracted spectrum And / or an extrapolation process to obtain reconstruction data.

Furthermore, a data processing program according to the present invention that achieves the above-described object is a computer-executable data processing program that analyzes and interpolates various data, and inputs the original data to be processed to the computer. A sine wave model represented by a data input means, an arbitrary dimension signal based on the original data input via the data input means, and a phase and an amplitude A ′ using a frequency f ′ and an initial phase φ ′ The frequency f ′, the amplitude A ′, and the initial phase φ ′ that minimize the sum of squares of the difference from the signal are obtained as parameters of the Fourier transform equation of the aperiodic signal, and a spectrum is extracted and extracted. It is characterized by functioning as an arithmetic means for obtaining reconstructed data by performing data interpolation processing and / or extrapolation processing based on the spectrum obtained. The

The data processing method, the data processing apparatus, and the apparatus in which the data processing program according to the present invention is mounted use a frequency analysis method that has a high frequency resolution and is less affected by the analysis window length. In addition, interpolation and / or extrapolation based on the original data can be performed without impairing the characteristics of the original data and without causing an error between frames.

In the present invention as described above, it is possible to interpolate the missing information with high accuracy without generating block noise regardless of the shape and size of the portion where the information is missing. Data can be significantly more natural. Similarly, in the present invention, it is possible to extrapolate unknown information with high accuracy without generating block noise regardless of any data and any portion of the data. In addition, when an image is configured with a small spectrum, high frequency components are insufficient and ringing occurs at the rising edge of the edge.In the present invention, the dynamic range at the time of data reconstruction is expanded to save the data to be stored. Since rounding can be performed according to the standard, occurrence of such ringing can be reduced. Therefore, in the present invention, only a small number of spectra need be retained as a predetermined number of spectra necessary for reconstructing data from which ringing has been removed, and surrounding unknown information is interpolated and / or extrapolated. Thus, a wide range of data can be expressed with a small amount of information.

It is a block diagram which shows the structure of the data processor shown as embodiment of this invention. It is a figure for demonstrating the difference between this frequency analysis method, DFT, and GHA, and is a figure which shows the result of having calculated | required the error of each method. 6 is a flowchart showing a series of processing when image data is reconstructed by interpolation and / or extrapolation based on original image data in the data processing apparatus shown as an embodiment of the present invention. It is a figure for demonstrating the image data processed in the data processor shown as embodiment of this invention, (a) shows the image data in which some information was missing, (b) shows (a (C) is a figure which shows the image data which substituted the average value of the image data shown to (a) with respect to an unknown area | region. It is a figure for demonstrating the effectiveness of the interpolation process in the data processor shown as embodiment of this invention, (a) is a figure which shows the image data used in order to verify effectiveness, ( (b) is a diagram showing image data in which information is missing from the image data shown in (a), and (c) is reconstructed by applying this frequency analysis method based on the image data shown in (b). It is a figure which shows image data. It is a figure for demonstrating the effectiveness of the interpolation process in the data processor shown as embodiment of this invention, (a) is a figure which shows the image data which provided three missing parts from which a shape and a magnitude | size differ (B) is a diagram showing image data reconstructed by applying this frequency analysis method based on the image data shown in (a). It is a figure for demonstrating the effectiveness of the interpolation process in the data processor shown as embodiment of this invention, (a) is an enlarged view of three missing parts shown to (a) of FIG. FIG. 6B is a diagram showing a portion corresponding to the missing portion shown in FIG. 6A in the reconstructed image data shown in FIG. It is a figure which shows the image data used in order to verify the effectiveness of the extrapolation process in the data processor shown as embodiment of this invention. It is a figure which shows the original image data which cut out the part enclosed with the rectangular frame line among the image data shown in FIG. FIG. 10 is a diagram showing image data reconstructed by extrapolation processing by applying FFT based on the original image data shown in FIG. 9. It is a figure which shows the image data reconfigure | reconstructed by the extrapolation process by applying this frequency analysis method based on the original image data shown in FIG. It is a figure which shows the image data reconfigure | reconstructed by extrapolation processing by applying this frequency analysis method based on other original image data. It is a figure which shows the image data reconfigure | reconstructed by extrapolation processing by applying this frequency analysis method based on the original image data shown in FIG. 9, (a) is the same image data as the original image data shown in FIG. (B) shows image data reconstructed based on 50 spectra extracted by analyzing the image data shown in (a) by this frequency analysis method, and (c) shows constants (a ) Shows the image data reconstructed based on the 50 spectra extracted by the frequency analysis method after being subtracted from the image data shown in (), and (d) shows the reconstructed image data shown in (c) The reconstructed image data is shown by expanding the dynamic range at the time. It is a flowchart which shows a series of processes of the "exemplar" based "method" algorithm. It is a figure for demonstrating the effectiveness at the time of applying this frequency-analysis method to an "exemplar (base) based (method)" algorithm, and is a figure which shows the original image data. FIG. 16 is a diagram for explaining the effectiveness when this frequency analysis method is applied to the “exemplar based method” algorithm, and is a diagram showing image data obtained by deleting information over a wide area with respect to the image data shown in FIG. 15. It is a figure for demonstrating the effectiveness at the time of applying this frequency-analysis method to the "exemplar" based "method" algorithm, and is a figure which shows the image data interpolated by the "exemplar" based "method" algorithm. It is a figure for demonstrating the effectiveness at the time of applying this frequency-analysis method to an "exemplar-based-method" algorithm, and is a figure which shows the image data interpolated by applying this frequency-analysis method.

Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings.

This embodiment is a data processing device that analyzes and / or extrapolates various data such as image data. In particular, this data processing apparatus applies a new frequency analysis method in which the frequency resolution does not depend on the analysis window length by estimating a Fourier coefficient by solving a nonlinear equation, and performs data interpolation and / or extrapolation. Is. Here, a case will be mainly described in which reconstruction is performed by performing interpolation based on original image data by performing interpolation processing and / or extrapolation processing.

[Data processor configuration]
The data processing apparatus is composed of, for example, a computer or the like. As shown in FIG. 1, a CPU (Central Processing Unit) 11 that centrally controls each unit and a read-only ROM (Read Read ROM that stores various types of information including various programs). Only Memory 12 and RAM (Random) that functions as a work area
(Access Memory) 13, a storage unit 14 that stores various information in a readable and / or writable manner, an input operation control unit 15 that performs processing and control of an input operation via a predetermined operation device (not shown) as a user interface, And a display unit 16 for displaying various information.

The CPU 11 executes various programs including various application programs stored in the storage unit 14 and the like, and comprehensively controls each unit.

The ROM 12 stores various information including various programs. Information stored in the ROM 12 is read under the control of the CPU 11.

The RAM 13 functions as a work area when the CPU 11 executes various programs. Under the control of the CPU 11, the RAM 13 temporarily stores various information and reads the stored various information.

The storage unit 14 stores various types of information including original image data and mask data to be processed in addition to application programs such as a data processing program according to the present invention. For example, a hard disk or a non-volatile memory can be used as the storage unit 14. The storage unit 14 also includes a drive device that reads and / or writes various types of information on a storage medium such as a flexible disk or a memory card that can be attached to and detached from the main body. Various types of information stored in the storage unit 14 are read out under the control of the CPU 11.

The input operation control unit 15 accepts an input operation via a predetermined operation device (not shown) as a user interface such as a keyboard, a mouse, a keypad, an infrared remote controller, a stick key, or a push button, for example, and indicates operation contents. A control signal is supplied to the CPU 11.

The display unit 16 is, for example, a liquid crystal display (Liquid Crystal).
Various display devices such as a display (LCD), a plasma display panel (PDP), an organic electroluminescence (Organic ElectroLuminescent) display, or a CRT (Cathode Ray Tube). Display information. For example, when the data processing program is activated by the CPU 11, the display unit 16 displays the screen, and the input original image data as the processing target and the image data after interpolation as the interpolation and / or extrapolation result Etc. are displayed.

When the data processing apparatus including each unit executes a data processing program under the control of the CPU 11, the spectrum is extracted by performing frequency analysis of the input image data under the control of the CPU 11. Then, the image data is reconstructed based on the obtained spectrum. Note that a signal to be subjected to frequency analysis, that is, image data to be processed is input to the CPU 11 via a data input unit (not shown). For example, when the data processing apparatus interpolates image data captured by an imaging apparatus such as a digital camera by interpolation processing and / or extrapolation processing, a predetermined interface that connects the data processing apparatus and the imaging apparatus The image data as the signal to be processed is input by storing it in the storage unit 14. In addition, the data processing apparatus may input arbitrary data created by the user by storing it in the storage unit 14 as a processing target signal. That is, the data input unit is a part having a function of causing the CPU 11 to input image data as a processing target. Needless to say, the data input unit also has a function of performing A / D conversion and converting it into a digital signal when an analog signal is input. At this time, the data input unit may be an A / D converter including an anti-aliasing filter as necessary. Under the control of the CPU 11, the data processing apparatus performs interpolation processing and / or extrapolation processing by performing frequency analysis of the image data as the processing target input in this manner, and reconstructed image data. And the like are stored in the storage unit 14 via an output unit (not shown) or output to other devices.

[Frequency analysis algorithm]
First, prior to description of a series of data processing algorithms in the data processing device, a frequency analysis algorithm used when performing data interpolation and / or extrapolation will be described in detail. Needless to say, the image data is a two-dimensional signal, but here, for convenience of explanation, a frequency analysis algorithm for a one-dimensional analysis target signal will be described.

In the frequency analysis method applied to the data processing apparatus (hereinafter referred to as the present frequency analysis method), the problem of obtaining the frequency parameter of the Fourier transform equation of the non-periodic signal shown in the following equation (1) is obtained as the optimal solution of the nonlinear equation. Replaced with a problem.

Specifically, in this frequency analysis method, as shown in the following equation (2), as an optimal solution of the nonlinear equation represented by the sum of squares of the difference between the analysis target signal x (n) and the sine wave model signal: Then, the frequency f ′, the amplitude A ′, and the initial phase φ ′ are obtained so that the right side of the nonlinear equation becomes the minimum value. In the following equation (2), L is a frame length (analysis window length), and f _s is a sampling frequency [Hz]. In this frequency analysis method, the effect of the analysis window and aliasing are eliminated by reducing the problem of finding the optimal solution of the nonlinear equation by the least square method, and the analysis window length is less than one cycle. In addition, it is not necessary to be an integral multiple of the period, and furthermore, it is possible to realize flexible frequency analysis processing such as unequal intervals.

Now, in order to actually obtain the optimal solution of the nonlinear equation shown in the above equation (2), the following method can be taken.

In this frequency analysis method, appropriate initial values are obtained for each of the amplitude A ′, the frequency f ′, and the initial phase φ ′, and the initial value is converged to an optimal solution using a nonlinear equation solving method. In this nonlinear problem, the above equation (2) is a minimization problem with a cost function. Appropriate initial values can be obtained by performing any arbitrary method such as discrete Fourier transform (DFT) or wavelet transform, or obtaining an approximate register by filtering. Can be obtained by applying.

First, in this frequency analysis method, the so-called steepest descent method is applied to the frequency parameters f ′ and φ ′ constituting the phase of the sine wave model signal in the above equation (2), and the frequency parameters f _m ′ and φ _m ′ are applied. Is obtained by the following equations (3) and (4).

In the above formulas (3) and (4), the following formula (5) is abbreviated. Further, mu _m, a weighting factor based on the so-called reduction method, to a cost function determined by the recursion formula monotonically decreasing sequence takes a value of timely 0-1.

If the frequency parameters f _m ′ and φ _m ′ can be obtained, the frequency parameter A ′ as a coefficient of the sine wave model signal in the above equation (2) can be uniquely obtained. The frequency parameter A _m ′ is converged by equation (6).

In this frequency analysis method, it is possible to converge the amplitude A ′, the frequency f ′, and the initial phase φ ′ with high accuracy by repeatedly performing these series of calculations. In particular, in this frequency analysis method, the calculation is performed by separately obtaining the frequency parameters f ′ and φ ′ constituting the phase of the sine wave model signal in the above equation (2) and the frequency parameter A ′ as a coefficient. It can be performed simply.

However, although the steepest descent method converges from a relatively wide range, accuracy is low in one iteration and it takes time to converge.

Therefore, in this frequency analysis method, it is desirable to converge the frequency parameters f _m ′ and φ _m ′ to some extent by applying the steepest descent method, and then to converge with high accuracy by applying the so-called Newton method. . Specifically, in this frequency analysis method, the frequency parameters f _m ′ and φ _m ′ are obtained by the recurrence formulas shown in the following equations (7) and (8) as the Newton method.

However, in the above formulas (7) and (8), J is the following formula (9) and is abbreviated as the following formula (10). Also, [nu _m is also a weighting coefficient based on the reduction method in the same manner as mu _m, taking the value of timely 0-1.

In the present frequency analysis technique, the above equation (7) and the equation (8) frequency parameter f _m by ', phi _m' sought after, like the steepest descent method, frequency parameter by the above equation (6) A _m 'Is converged and this series of calculations is repeated further.

As described above, in this frequency analysis method, the frequency parameters A ′, f ′, and φ ′ are estimated at high speed and with high accuracy by using a hybrid method combining the steepest descent method and the Newton method. Can do.

Further, in this frequency analysis method, even when the analysis target signal x (n) is a composite sine wave, the spectral parameter can be approximately derived by performing successive subtraction processing. Here, it is assumed that the analysis target signal x (n) is the sum of a plurality of sine waves and is expressed as the following equation (11).

According to the Parseval theorem, if the frequency _fk of the signal to be analyzed x (n) and the frequency parameter f ′ of the sine wave model signal do not coincide at all, that is, if The nonlinear equation shown in the equation (2) becomes the following equation (13). When the set of frequency parameters f ′ and φ ′ matches either of the set of the frequency f _k and the initial phase φ _k , the nonlinear equation shown in the above equation (2) becomes the following equation (14). . Furthermore, when the amplitude A _j matches the frequency parameter A ′, the frequency component related to the estimated spectrum can be completely eliminated from the analysis target signal. Therefore, the problem of obtaining the optimum solution is independent of the frequency, and can be applied to signals represented by a plurality of sine waves if individually estimated from the analysis target signal.

That is, in this frequency analysis method, even if the analysis target signal x (n) is a composite sine wave, it is possible to extract a plurality of sine waves by performing the same process on the residual signal successively. .

In order to express a signal such as an audio signal or an acoustic signal with a composite sine wave, a large number of spectra (the number of sine waves) has been required so far. Even if it exists, it can express with few errors and without an error. That is, being able to express a signal with a smaller number of spectra indicates that it is effective for information compression.

[Effectiveness of this frequency analysis method]
Hereinafter, the effectiveness of this frequency analysis method will be specifically described.

This frequency analysis method can determine the frequency f ′, the amplitude A ′, and the initial phase φ ′ of the sine wave model signal at high speed and with high accuracy by obtaining the optimal solution of the nonlinear equation. In order to verify the specific accuracy, the inventor of the present application verified accuracy by comparing DFT and GHA (Generalized Harmonic Analysis), which is said to have the highest analysis accuracy among the developed types of DFT. .

Note that DFT and GHA apparently have a plurality of window lengths in one analysis window length, so the frequency resolution depends on the analysis window length, but the resolution frequency is finite, and the signal to be analyzed If the analysis signal is different from the frequency that can be analyzed accurately, the analysis signal cannot be analyzed when the frequency is other than the decomposition frequency. A frequency (sideband component) appears, and a plurality of frequencies appear.

Whether or not such a phenomenon occurs also in the present frequency analysis method, that is, in order to verify the frequency resolution of the present frequency analysis method, the analysis window length is one second (1024 samples) and is very short. A single sine wave was analyzed, one sine wave was extracted by each method, and the square error from the original signal was examined. The result is shown in FIG.

As shown in FIG. 2, in DFT, analysis accuracy deteriorated at frequencies other than integer multiples of the fundamental frequency. In GHA, the accuracy was improved by about 2 to 5 digits compared to DFT at a frequency of 1 Hz or higher. On the other hand, in this frequency analysis method, an accuracy improvement of 10 digits or more compared to DFT and 5 digits or more compared to GHA was observed at a frequency of 1 Hz or more. That is, this frequency analysis method showed an improvement in accuracy of 100,000 to 10 billion times or more compared with the existing frequency analysis method. In particular, the fact that a frequency of 1 Hz or less can be accurately estimated indicates that even a long periodic signal exceeding the analysis window length can be analyzed.

Thus, this frequency analysis method can perform analysis with surprisingly high accuracy even compared to GHA, which is said to have the highest analysis accuracy. When the data processing apparatus reconstructs image data by interpolation and / or extrapolation based on the original image data which is a two-dimensional signal, this frequency analysis method is applied and shown in FIG. A series of processes are performed.

First, as shown in FIG. 3, the data processing apparatus converts the original image data I _org (n ₁ , n ₂ ) into a two-dimensional signal via a data input unit (not shown) under the control of the CPU 11 in step S1. When image data I (n ₁ , n ₂ ) is input and stored in a memory such as the RAM 13, the minimum L required to reconstruct the image data under the control of the CPU 11 in step S 2. It is determined whether or not a book spectrum has been extracted. When the data processing apparatus has not extracted all the L spectra, in step S3, the portion of the image data I (n ₁ , n ₂ ) that lacks information under the control of the CPU 11 To the image data I (n ₁ , n ₂ ), the mask data M (n ₁ , n ₂ ) corresponding to the missing information is applied to the image data I (n ₁ , n ₂ ) as shown in the following equation (15): New image data I ′ (n ₁ , n ₂ ) is generated.

Note that the mask data M (n ₁ , n ₂ ) is M (n ₁ , n ₂ ) = 0 for unknown regions and M (n () for known regions in the image data I (n ₁ , n ₂ ). n ₁ , n ₂ ) = 1 and can be generated by using an arbitrary mask data generation algorithm, image processing software, or the like. In the above equation (15), M ′ is the inverted data of the mask data M. Further, K is a predetermined constant, and takes the average value or median value of the image data I in the case of interpolation processing. On the other hand, the constant K in the case of extrapolation processing is not practically used because the inverted data M ′ always takes a value of 0. However, in order to implement an algorithm similar to the interpolation processing, an image is used here. An arbitrary value such as an average value or a median value of the data I is assumed. That is, the data processing apparatus substitutes the value K for an unknown area in the image data I (n ₁ , n ₂ ) and proceeds with the process. Specifically, for example, as shown in FIG. 4A, the data processing apparatus performs mask data M (n) when processing the image data I (n ₁ , n ₂ ) with some information missing. ₁ , n ₂ ), data is prepared that takes a value of 0 for an unknown region lacking information as shown in FIG. 4B and a value of 1 for a known region. When the data processing apparatus substitutes the average value K (= 149.33) of the image data I for the unknown area in the image data I (n ₁ , n ₂ ), the data processing apparatus shown in FIG. The image data I ′ (n ₁ , n ₂ ) as shown in FIG. Note that the image data shown in FIG. 4A is the same as the image data area shown in the upper right of FIG. The data processing apparatus may subtract the constant C from the data I · M + K · M ′ under the control of the CPU 11. In other words, the data processing apparatus is able to perform extrapolation over a wide area by extracting a spectrum of only the AC component of the texture from which information unnecessary for extrapolated information is removed while reducing the amount of information. The low-frequency component with little deterioration is removed by quantization.

Further, in the data processing apparatus, when extracting a plurality of spectra, it is necessary to set a new initial value for each spectrum extraction in order to perform the calculations of the above equations (2) to (10). . Therefore, after setting an initial value under the control of the CPU 11 in step S4, the data processing apparatus sets the image data I ′ (n ₁ , n ₁ , n ₂ , obtained in step S3 consisting of a two-dimensional signal in step S5. The frequency analysis method described above is applied to n ₂ ), and the above formulas (2) to (10) are calculated.

Next, in step S6, the data processing apparatus extracts the l-th spectrum as shown in the following equation (16) under the control of the CPU 11. In the following equation (16), f _xs and f _ys are sampling frequencies [Hz] in the horizontal axis direction and the vertical axis direction of the image data, respectively, and A _l ′, f _xl ′, f _yl ′, φ _l ′ is the amplitude of the spectrum to be extracted, the frequency corresponding to each axis of the image data, and the initial phase, respectively. Here, as described above, for each of the amplitude A _l ′, the frequencies f _xl ′, f _yl ′, and the initial phase φ _l ′, the data processing apparatus performs DFT, Fast Fourier Transform (FFT), etc. When obtaining an appropriate initial value using, it is necessary to remove a portion where information is missing from the processing target. Therefore, the data processing apparatus applies the frequency analysis method described above to a portion of the image data I ′ (n ₁ , n ₂ ) where the pixel value of the mask data M (n ₁ , n ₂ ) is not zero. To extract a spectrum. The data processing device expresses image data that is a two-dimensional signal using a sine wave model function, changes parameters so that the difference between the actual signal and the sine wave model signal is minimized, and obtains each frequency.

In step S7, the data processing device converts the image data I ″ (n ₁ , n ₂ ) represented by the extracted spectrum from the image data I (n ₁ , n ₂ ) under the control of the CPU 11. The subtracted residual signal is substituted for the image data I (n ₁ , n ₂ ), and in step S8, the image data I ′ (n ₁ , n ₂ ) and the image data I ″ (n ₁ , n ₂ ) Is added to the image data I ′ (n ₁ , n ₂ ), and in step S9, l is incremented, the constant K is updated, and the processing from step S2 is repeated. That is, in the data processing apparatus, when the first spectrum is extracted, the image data I (n ₁ , n ₂ ) is the original image data I _org (n ₁ , n ₂ ) itself, but the second and subsequent ones are extracted. In the case of extracting a spectrum, the image data I (n ₁ , n ₂ ) is the image data I ″ (n) composed of the original image data I (n ₁ , n ₂ ) and the spectrum extracted so far. ₁ , n ₂ ).

The data processing apparatus does not extract this L spectrums (A _l ′, f _xl ′, f _yl ′, φ _l ′; l = 1 to L) that are at least necessary for reconstructing the image data. When the above processing is performed and L spectra are extracted, in step S10, under the control of the CPU 11, the interpolation and / or extrapolation processing of the image data is performed as shown in the following equation (17). The reconstructed image data I ′ ″ (n ₁ , n ₂ ) is output, and the series of processes is terminated. At this time, if the data processing apparatus subtracts the constant C corresponding to the number of removed quantization bits from the data I · M + K · M ′, the image data is obtained by reconstructing the constant C ′ based on the L spectra. Add to. Here, the constant C ′ = C may be satisfied. This is because the expansion of the dynamic range is used to correct the edge portion of the image, and the output image is displayed with the same gradation as the input image (256 gradations of 0 to 255 in the case of 8 bits). , Because constant C ′ = C. For example, when an image having 256 gradations (0 to 255) is changed to 512 gradations (0 to 511), the constant C ′ = 2C may be set.

Note that when the image is constructed by extracting only a small number of spectra by applying this frequency analysis method, the high frequency components of the image data reconstructed based on the information are insufficient, and the edge rises. Ringing is likely to occur. Therefore, the data processing device removes the ringing of the texture image by extending the dynamic range when reconstructing the image data to enhance the image intensity and rounding it with a predetermined threshold. That is, the data processing apparatus, for example, in the case of image data quantized with 8 bits, an arbitrary constant k is applied to the image data constituted by the extracted spectrum so that 0 and 255 are set as threshold values. It is desirable to reconstruct the image data while expanding the dynamic range of the image data by multiplying.

The data processing apparatus reconstructs the image data I ′ ″ (n ₁ , n ₂ ) based on the original image data I _org (n ₁ , n ₂ ) by performing such a series of processes. Can do. Since the dynamic range of the reconstructed image data I ′ ″ (n ₁ , n ₂ ) is rounded according to the standard of the image data, ringing can be reduced. In addition, the data processing apparatus can extrapolate information outside the measurement region by extending the possible range of (n ₁ , n ₂ ) to a length exceeding the measurement region.

[Effectiveness of data processing equipment]
The inventor of the present application actually analyzed image data using such a data processing apparatus, and performed interpolation processing and extrapolation processing.

First, an example of performing interpolation processing will be described.

For the image data as shown in FIG. 5 (a), as shown in FIG. 5 (b), image data in which information is lost due to a plurality of dots dispersed over the entire image area was prepared. The peak SNR of the image data shown in FIG. 5B with respect to the image data shown in FIG. 5A is 21.2079 dB. Further, mask data having values corresponding to these dots was prepared. Then, reconstructed image data was obtained by interpolating the missing information based on information around the portion of the image data where the information is missing.

This result is shown in FIG. In a data processing apparatus to which the present frequency analysis method that has high frequency resolution and can be analyzed with little influence by the analysis window length, the characteristics of the original image data are impaired as shown in FIG. In addition, interpolation based on the original image data was performed without causing an error between frames. The peak SN ratio of the image data shown in FIG. 5C with respect to the image data shown in FIG. 5A was a high value of 43.5708 dB.

The inventor of the present application also examined the influence of the shape and size of a portion where information is missing when the frequency analysis method is applied. Specifically, for the same image data as the image data shown in FIG. 5 (a), as shown in FIG. 6 (a), image data provided with three missing portions having different shapes and sizes are prepared. did. Then, reconstructed image data was obtained by interpolating the missing information based on information around the portion of the image data where the information is missing.

As a result, in the data processing apparatus, as shown in FIG. 6B, regardless of the shape and size of the missing part, the characteristics of the original image data are not impaired, and errors occur between frames. And interpolation based on the original image data could be performed. More specifically, as shown in FIG. 7 (b), reconstructed image data reflecting ground patterns and shades substantially the same as the original image data for all three missing portions shown in FIG. 7 (a). Obtained. In other words, in the data processing apparatus, an accurate analysis is performed without impairing the characteristics of the original image data, regardless of the shape and size of the missing part, and any part of the image area. Even when the missing information is interpolated by interpolation, no error is generated between frames.

In this way, the data processing apparatus to which this frequency analysis method is applied performs interpolation processing that eliminates the generation of block noise that could not be achieved by applying the conventional frequency analysis method, and significantly reconstructed image data is obtained. It is possible to obtain.

Next, an example of performing extrapolation processing will be described.

Image data having the same pattern as shown in FIG. 8 is prepared, and a portion surrounded by a rectangular frame (FIG. 9) is cut out as an analysis object, that is, original image data. Based on this original image data Then, the reconstructed image data was obtained by extrapolating the surrounding unknown information. Note that the size of the original image data shown in FIG. 9 is 140 pixels × 140 pixels, and based on this, reconstructed image data having a size of 420 pixels × 420 pixels was obtained.

Conventional extrapolation methods extrapolate unknown information using quadratic linear prediction or other functions, or apply frequency analysis methods such as discrete Fourier transform focusing on the periodicity of texture images. Then, the spectrum is extracted, and unknown information around it is extrapolated. Therefore, for comparison, the inventor of the present application analyzed the same original image data using FFT instead of the present frequency analysis method to obtain reconstructed image data.

These results are shown in FIG. 10 and FIG. First, when FFT is applied, the same information as the analysis result of the original image data is repeatedly extrapolated in the surroundings as shown in FIG. 10, resulting in unnatural reconstructed image data between frames. . Note that, when performing analysis, an error between frames can be reduced by using a window function. However, even in this case, the original signal cannot be obtained when the image data is reconstructed. In other words, in the extrapolation method to which the conventional frequency analysis method such as FFT is applied, there is a problem of how to determine the analysis section, that is, which part of the texture image is cut out and analyzed and the frame length. It is very difficult to solve this.

On the other hand, in the data processing apparatus to which the present frequency analysis method is applied, as shown in FIG. 11, the original image data is not lost without losing the characteristics of the original image data and without generating an error between frames. Extrapolation based on In addition, even when a portion surrounded by a rectangular frame line in FIG. 12 is cut out as other original image data and surrounding unknown information is extrapolated by a data processing apparatus to which the present frequency analysis method is applied, the characteristics of the original image data The extrapolation based on the original image data can be performed without impairing the image quality and without causing an error between frames. In other words, the data processing apparatus can perform accurate analysis without damaging the characteristics of the original image data, regardless of what texture image or any portion of the texture image is cut out. No error is generated between frames when extrapolating. As described above, the data processing apparatus can extrapolate information outside the measurement region by extending the range that (n ₁ , n ₂ ) can take beyond the measurement region. The image data shown in FIG. 9 has a value in the range of (n ₁ , n ₂ ) from 0 to 1, whereas the image data shown in FIG. 11 has a value of (n ₁ , n ₂ ) from −1 to 2. It is the result of restructuring by range.

In addition, when the present frequency analysis method is applied, the inventor of the present application also examined the influence of the removal of the low frequency component in step S3 in FIG. 3 and the expansion of the dynamic range in step S4.

FIG. 13A shows the same original image data consisting of 140 pixels × 140 pixels as shown in FIG. Image data reconstructed based on the 50 spectra extracted by analyzing the original image data by the frequency analysis method, as shown in FIG. 13B, lacks high frequency components and ringing occurs. It will be a thing.

Further, as shown in step S3 in FIG. 3, after subtracting the constant C of the number of quantization bits / 2 from the original image data, reconstruction is performed based on 50 spectra extracted by analysis by this frequency analysis method. Then, by adding the subtracted constant C, image data in which ringing occurred at the edge portion of the image was obtained as shown in FIG.

On the other hand, when the dynamic range at the time of reconstructing the image data shown in FIG. 13C is expanded, as shown in FIG. 13D, image data with reduced ringing was obtained. For this reason, in the data processing apparatus, by expanding the dynamic range when reconstructing the image data, it can be rounded according to the standard of the image data to be stored, and the number of spectra extracted by frequency analysis can be reduced. It can be seen that it is possible to obtain image data in which both ringing due to lack of high frequency components due to lack and ringing due to removal of low frequency components by quantization are reduced.

In this way, the data processing apparatus to which this frequency analysis method is applied performs extrapolation processing that eliminates the generation of block noise that could not be achieved by applying the conventional frequency analysis method, and significantly reconstructed image data is obtained. It is possible to obtain.

Further, the inventor of the present application stated that “Antonio
Criminisi, Patrick Perez and Kentaro Toyama, “Region Filling and Object Removal by Exemplar-Based Image
Inpainting ”, IEEE Trans. On Image Process., Vol.
13, No. 9, 2004, p.1200-1212 ”, and this frequency analysis method is applied to the“ exemplar based method ”algorithm known as a technique for performing wide area interpolation. The effectiveness was verified.

Here, the “exemplar based method” algorithm is as shown in FIG. That is, when the “exemplar based method” algorithm extracts the initial contour δΩ ⁰ of the region Ω selected as the interpolation target for the original data, the contour δΩ ^t is identified and prioritized in the t-th iteration in step S21. The degree P (p) is calculated. In the "exemplar based method" algorithm, if the contour [Delta] [omega ^t is 0 and ends the series of processes. Further, the priority P (p) is a data term (data) that is a function of a confidence term (confidence term) C (p) that is reliability information around the pixel p and the intensity of the isoluminance line reaching the contour δΩ. term) D (p) and P (p) = C (p) D (p), which satisfies the relationship ∀pεδΩ ^t .

Subsequently, in step S22, the “exemplar based method” algorithm obtains a point p ′ having the highest priority, that is, a patch Ψ _{p ′} that is a square template at p ′ = argmax _pεδΩt P (p), In step S23, a sample Ψ _{q ′} ∈Φ that minimizes the distance d (Ψ _{p ′} , Ψ _{q ′} ) between the two patches Ψ _{p ′} and Ψ _{q ′} is searched, and in step S24, from the patch Ψ _{q ′.} Copy the image data to the patch Ψ _{p ′} . Note that the relationship of ∀p∈Ψ _{p ′} Ω is satisfied.

Then, "exemplar based method" algorithm, in step S25, the updated reliability term C a (p) such that ∀p∈Ψ _{p '∩Ω.} The “exemplar based method” algorithm repeats such processing t = T times, and generates image data obtained by interpolating a region lacking information.

The inventor of the present application tried to interpolate over a wide area by applying this frequency analysis method by replacing the processing of step S22 to step S24 in the “exemplar based method” algorithm with this frequency analysis method.

The image data shown in FIG. 15 was processed to create image data in which information was lost over a wide area, as indicated by A to I in FIG. 16, and this was input and processed as original image data.

As a result, in the image data interpolated by the “exemplar based method” algorithm, as shown in FIG. 17, the missing portions shown in A to I in FIG. 16 are not sufficiently reproduced. As shown in FIG. 18, the image data interpolated by applying the method reproduced any missing portions with high accuracy and became very close to the image data shown in FIG.

Thus, the data processing apparatus to which the present frequency analysis method is applied is extremely effective when performing interpolation over a wide area.

[Effect of data processing device]
As described above, the data processing apparatus to which this frequency analysis method is applied can interpolate the missing information over a wide range with high accuracy without generating block noise regardless of the shape and size of the missing portion. And can make the data after interpolation much more natural. In addition, the data processing apparatus can extrapolate unknown information with high accuracy without generating block noise regardless of what texture image or any portion of the texture image is cut out. In addition, this data processing device can reduce ringing because it can be rounded in accordance with the standard of data to be stored by extending the dynamic range at the time of data reconstruction. As a result, in this data processing apparatus, only a small number of spectra need be retained in order to reconstruct a texture image from which ringing has been removed. By extrapolating surrounding unknown information, a wide range of information can be obtained with less information. A texture image can be expressed.

Such a data processing apparatus is applied to a purpose of highly accurately interpolating unknown information lost due to “whiteout” or “blackout” of captured image data based on surrounding information or remaining information. It is also possible to convert an image represented by 8 bits per color into an arbitrary dynamic range. Therefore, this data processing apparatus is extremely useful in view of the recent expansion trend of the digital camera market.

Furthermore, in recent years, computer graphics technology has become indispensable for the production of movies and games, and there is a need for more realistic expression using computer graphics. For this purpose, the texture or the like that is a pattern to be pasted on the surface of the three-dimensional object needs to be more realistic, and much research in this field has been conducted. Considering such a background, a data processing apparatus to which the frequency analysis method is applied is a photograph taken by a user with a digital camera or the like when generating a texture used in 3D modeling software or image processing software. It is possible to freely cut out an arbitrary part included in the image and paste it as a texture on the surface of a three-dimensional object or the like, and to bring a more realistic texture expression. Further, this data processing apparatus can be expanded to an arbitrary size by performing extrapolation processing even when the cut out texture is small, and can be mapped to an arbitrary surface. Thus, this data processing and insertion device is extremely useful in light of the demands of the computer graphics market.

Note that the present invention is not limited to the embodiment described above. For example, in the above-described embodiments, the interpolation processing and / or extrapolation processing of image data has been described. However, the present invention can be applied to audio data such as a purpose of repairing audio data in which a burst error has occurred due to noise mixing. The present invention can be applied to interpolation processing and / or extrapolation processing of arbitrary data such as moving image data that is three-dimensional data as well as one-dimensional data such as data.

In addition, as described above, the present invention can perform interpolation by interpolation regardless of the shape and size of a portion where information is lost. For example, this can be performed by converting an analog signal into a digital signal. In this case, even when the sampling interval of the A / D converter becomes unequal due to the influence of the jitter of the sampling clock, it is shown that the processing can be performed with high accuracy. In other words, the present invention can interpolate the missing information with high accuracy over a wide range even when using an inexpensive A / D converter with low accuracy, and contributes to reduction of system cost. be able to.

Furthermore, in the process shown in FIG. 3, it has been described that the process proceeds while synthesizing the extracted spectra, but the present invention may synthesize these after extracting all L spectra, and the data It is not limited to the timing of the analysis and the extraction of the spectrum and the synthesis of the extracted spectrum.

Furthermore, in the above-described embodiment, the data processing apparatus has been described as performing frequency analysis by software. However, the present invention implements an algorithm for data interpolation processing and / or extrapolation processing including this frequency analysis method. If a product-sum operation can be performed, such as a DSP (Digital Signal Processor), it can also be realized by hardware.

Thus, it goes without saying that the present invention can be modified as appropriate without departing from the spirit of the present invention.

Claims

A data processing method for analyzing and processing various data,
A data input step of inputting original data to be processed into a data processing device and storing it in a memory;
The calculation means of the data processing device reads the original data input in the data input step and stored in the memory, and uses an arbitrary dimension signal based on the original data, the frequency f ′, and the initial phase φ ′. The frequency f ′, the amplitude A ′, and the initial phase φ ′ such that the sum of squares of the difference between the phase and the amplitude A ′ expressed by the phase and the amplitude A ′ is the minimum value are set to An interpolation and / or extrapolation step for obtaining reconstructed data by performing a data interpolation process and / or extrapolation process based on the extracted spectrum and obtaining a spectrum obtained as a Fourier transform parameter. A data processing method characterized by the above.
In the interpolation and / or extrapolation step, the calculation means applies mask data corresponding to the missing information to the original data read from the memory, and an arbitrary dimension signal of the data multiplied by the mask data The frequency f ′, the amplitude A ′, and the sum of squares of the differences between the phase using the frequency f ′ and the initial phase φ ′ and the sinusoidal model signal represented by the amplitude A ′, and The data processing method according to claim 1, wherein a spectrum is extracted by obtaining the initial phase φ 'as a parameter of a Fourier transform formula of an aperiodic signal.
The said interpolation means and / or extrapolation process WHEREIN: The said calculating means extracts a spectrum about the part in which the value of the said mask data is not zero among the data which applied the said mask data. Data processing method.
In the interpolation and / or extrapolation step, the calculation means multiplies the data constituted by the extracted spectrum by an arbitrary constant k to expand the dynamic range of the data and / or The data processing method according to any one of claims 1 to 3, wherein the reconstruction data is obtained by performing extrapolation processing.
In the interpolation and / or extrapolation step, the calculation means subtracts a constant C from the arbitrary dimension signal based on the original data read from the memory to remove low frequency components by quantization, and subtracts the constant C. For the obtained data, the spectrum is extracted by obtaining the frequency f ′, the amplitude A ′, and the initial phase φ ′, and a constant C ′ is added to the data reconstructed based on the extracted spectrum. The data processing method according to any one of claims 1 to 4, wherein reconstructed data is obtained.
In the interpolation and / or extrapolation step, the calculation means specifies the contour of the region to be interpolated selected for the original data read from the memory, and based on the contour, based on the original data The frequency f ′ and the amplitude such that the sum of squares of the difference between the arbitrary dimension signal and the sine wave model signal represented by the phase using the frequency f ′ and the initial phase φ ′ and the amplitude A ′ becomes a minimum value. The data processing method according to any one of claims 1 to 5, wherein a spectrum is extracted by obtaining A 'and the initial phase φ' as parameters of a Fourier transform equation of an aperiodic signal. .
The data processing method according to any one of claims 1 to 6, wherein the data is image data that is a two-dimensional signal.
The data processing method according to any one of claims 1 to 6, wherein the data is audio data which is a one-dimensional signal.
A data processing device that analyzes and processes various data,
Data input means for inputting the original data to be processed;
The square of the difference between the arbitrary dimension signal based on the original data input via the data input means and the phase and amplitude A ′ using the frequency f ′ and the initial phase φ ′. The spectrum f is extracted by obtaining the frequency f ′, the amplitude A ′, and the initial phase φ ′ such that the sum becomes the minimum value as a parameter of the Fourier transform formula of the aperiodic signal, and data is obtained based on the extracted spectrum. A data processing apparatus comprising: an arithmetic means for performing reconstruction processing and / or extrapolation processing to obtain reconstruction data.
A computer-executable data processing program that analyzes and interpolates various data,
The computer,
Data input means for inputting the original data to be processed; and
The square of the difference between the arbitrary dimension signal based on the original data input via the data input means and the phase and amplitude A ′ using the frequency f ′ and the initial phase φ ′. The spectrum f is extracted by obtaining the frequency f ′, the amplitude A ′, and the initial phase φ ′ such that the sum becomes the minimum value as a parameter of the Fourier transform formula of the aperiodic signal, and data is obtained based on the extracted spectrum. A data processing program that functions as an arithmetic means for performing reconstruction processing and / or extrapolation processing to obtain reconstruction data.