JP2011197954A - Signal processing apparatus and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To quicken calculation and generate a signal of high quality, when enhancing a resolution of the signal.SOLUTION: A state shifter 50 regularizes total variation, using a probability density function P easy to generate a hypothesis with the variation getting low in its value, and generates a predicted hypothetical group {x}. An observation value predictor 60 generates an observation predicted value z) by bringing the predicted hypothetical group {x} into a low resolution state, and a likelihood calculator 70 compares the observation predicted value z) with a low-resolution image y, and calculates a likelihood Lof a value getting high along with highness in similarity between the images. An observation updating device 80 updates a weight w, based on the likelihood L, A representative value calculator 30 calculates a representative value, based on the predicted hypothetical group {x} and an updated weight group {w}, to output a super resolution image. The processing of high calculation load in accompaniment to repetition calculation is not required thereby, although required in the prior art.

Description

  The present invention relates to a signal processing apparatus and program for images, sounds, and the like, and more particularly to a technique for generating a high-quality signal.

  Conventionally, in the field of image processing, as a method for restoring or estimating a high-resolution image from a low-resolution image, a method for generating a high-resolution image by interpolating between pixels of the low-resolution image It has been known. In this method, the sharpness of the image signal can be improved by providing the interpolation filter for performing the interpolation process or the filter for performing the post-processing with a high frequency emphasis characteristic.

  In addition, a technique for generating a high-resolution image by a single super-resolution method focusing on the local luminance pattern structure (for example, edge, straight line, fractal property) of the image is known (see Patent Document 1). ). This method generates a high resolution image using a non-linear filter (directional filter or the like). Specifically, optimum interpolation processing is performed on each of the skeleton component corresponding to the global edge structure extracted from the original image signal and the texture component corresponding to the fine vibration component. As a result, high-definition interpolation processing can be realized, and high-frequency components at the edge portion can be virtually generated.

  There is also a method of generating a high resolution image by using a multiple-resolution super-resolution method in which local image correlation between frames is obtained with a small number of pixel position accuracy for moving image signals and images are superimposed while performing motion compensation. Known (Patent Document 2). According to this method, high-speed super-resolution processing can be realized, aliasing distortion can be canceled, and high-frequency components of edge portions can be generated.

  In addition, in the method using the single-frame super-resolution method or the multiple-frame super-resolution method described above, a high-resolution image is converted from a low-resolution image by introducing a regularization term that is a cost function for the signal characteristics of the image. There is also known a method for solving the problem of setting a defect by estimating. For example, there is a method for solving the optimization problem of total variation regularization by iterative calculation (Non-Patent Document 1).

JP 2009-130632 A JP 2006-309649 A

Sina Farsiu, M. Dirk Robinson, Michael Elad, and Peyman Milanfar, “Fast and Robust Multiframe Super Resolution,” IEEE Transactions on Image Processing, Vol.13, No.10, pp.1327-1344, October 2004.

  However, in principle, the above-described interpolation method cannot reproduce or generate a high-frequency component exceeding the Nyquist frequency of a low-resolution image signal. Even if the high-frequency emphasis processing is performed, since there is no effective signal component in the first place, a high-frequency component exceeding the Nyquist frequency cannot be reproduced or generated.

  In the single-frame super-resolution method, for example, whether the local region of the image is an edge or flat is determined by a case according to the luminance pattern. Distortion may occur. In addition, since there is no uniform guideline regarding conditions for dividing cases, threshold values, etc., the design thereof is difficult, and there is a problem that a method using the single-frame super-resolution method cannot be easily realized.

  In addition, the total variation regularization method makes it possible to clearly define physical processes such as image blurring and sampling as cost functions, so design is easy, but this optimization problem is nonlinear. For this reason, there is a problem that the calculation load such as repetitive calculation is high.

  Thus, in the field of image processing, in order to improve the resolution of a signal, when generating a high resolution image from a low resolution image, various image quality degradation, high calculation load, etc. There was a problem. Also in the field of audio processing, there is a problem similar to that in the field of image processing when generating high quality audio from low quality audio.

  Therefore, the present invention has been made to solve the above-described problems, and the object thereof is to realize high-speed calculation and to generate a high-quality signal when improving the signal resolution. The object is to provide a signal processing apparatus and a program.

  In order to solve the above problems, the invention of claim 1 is a signal processing device that generates a high resolution signal from a low resolution signal, and holds a plurality of candidates for the high resolution signal as a hypothesis group, and the candidate Holding means for holding weights as weight groups, state transition means for causing a state transition of the hypothesis group held in the holding means and generating a new hypothesis group, and a new hypothesis group generated by the state transition means Is converted into an observation signal having the same resolution as the low-resolution signal, and the observation signal converted by the observation conversion means is compared with the low-resolution signal, and the similarity between the signals is estimated as likelihood. A likelihood calculation means for calculating as follows, and an observation update for updating the weight group based on the likelihood calculated by the likelihood calculation means and generating a new weight group for the new hypothesis group And a new hypothesis group generated by the state transition means, and a new weight group updated by the observation update means, the representative value of the hypothesis in the new hypothesis group is converted into the high-resolution signal. And a representative value calculating means for calculating as follows.

  According to the first aspect of the present invention, the hypothesis changes state by the state transition unit, and the weight of the hypothesis that matches the low resolution signal is increased by the likelihood calculating unit and the observation updating unit. This makes it possible to generate a high resolution signal from a hypothesis that matches the low resolution signal. In addition, the processing efficiency can be improved, and there is no need to perform processing with a high calculation load associated with repetitive calculations as in the conventional case.

  According to a second aspect of the present invention, in the signal processing device according to the first aspect, the state transition means changes the signal value of each signal point constituting the hypothesis of the hypothesis group based on a predetermined rule, and a new A hypothesis group is generated.

  According to the invention of claim 2, when a high resolution signal is generated from a low resolution signal, the signal value of the signal point constituting the low resolution signal is sufficiently affected and the signal value does not vary greatly. Regularization can be realized.

  According to a third aspect of the present invention, in the signal processing device according to the second aspect, the state transition means is configured to determine a gradient of an adjacent signal point with respect to a signal value of each signal point constituting a hypothesis of the hypothesis group. A new hypothesis group is generated so that the total variation value indicating the sum of absolute values becomes smaller.

  According to the third aspect of the present invention, since regularization by total variation is realized, it is possible to generate a high resolution signal with less signal edge degradation.

  According to a fourth aspect of the present invention, in the signal processing device according to the third aspect, the state transition means includes a signal value of a point of interest in a signal constituting a hypothesis of the hypothesis group and within a predetermined range from the point of interest. The state of the hypothesis group is changed using a probability density function that is defined so that the total variation value calculated from the signal value of the signal point becomes smaller when the signal value of the target point is changed. A hypothesis group is generated.

  According to the invention of claim 4, since the probability density distribution model of the probability density function that realizes regularization by the total variation is used, the change of the total variation value can be grasped from a small local region of the signal, Can be speeded up.

  The signal processing device according to any one of claims 1 to 4, wherein the signal is a still image signal, and the state transition means, the observation conversion means, and the likelihood calculation means. And the observation updating means generates a new hypothesis group and a new weight group by a predetermined number of iterations.

  According to the invention of claim 5, a high resolution still image can be efficiently generated from a low resolution still image.

  According to a sixth aspect of the present invention, in the signal processing device according to any one of the first to fourth aspects, the signal is a moving image signal composed of a plurality of frames, and the state transition means, the observation A new hypothesis group and a new weight group are generated for each frame or by a predetermined number of iterations in the frame by the conversion means, likelihood calculation means, and observation update means. .

  According to the invention of claim 6, a moving image whose shape changes every moment can be predicted by state transition, and a high-resolution moving image can be efficiently converted from a low-resolution moving image to an object in the moving image. It can be generated according to the movement.

  Furthermore, the invention of claim 7 resides in a signal processing program for causing a computer to function as the signal processing device according to any one of claims 1 to 6.

  According to the invention of claim 7, the hypothesis makes a state transition, and the weight of the hypothesis matching the low resolution signal is increased. This makes it possible to generate a high resolution signal from a hypothesis that matches the low resolution signal. In addition, the processing efficiency can be improved, and there is no need to perform processing with a high calculation load associated with repetitive calculations as in the conventional case.

  As described above, according to the present invention, when improving the resolution of a signal, it is possible to realize high-speed calculation and to generate a high-quality signal.

1 is a block diagram illustrating an overall configuration of a signal processing device according to an embodiment of the present invention. It is a flowchart explaining operation | movement of a signal processing apparatus. It is a figure which shows the example of the tap coefficient of an interpolation filter. It is a figure explaining the probability density function which implement | achieves regularization by a total variation.

The best mode for carrying out the present invention will be described below with reference to the drawings.
[Configuration of signal processing device]
First, the overall configuration of a signal processing apparatus according to an embodiment of the present invention will be described. FIG. 1 is a block diagram showing the overall configuration of the signal processing apparatus 1. This signal processing apparatus 1 includes switching means 10-1 and 10-2, initialization means 20, representative value calculation means 30, latch means (holding means) 40, state transition means 50, observation value prediction means (observation conversion means). 60, likelihood calculating means 70, observation updating means 80 and resampling means 90. The signal processing apparatus 1 is intended for image signals and performs super-resolution processing for improving image quality. The signal processing device 1 receives the low-resolution image y and the initialization signal, and generates and outputs a super-resolution image s from the low-resolution image y by super-resolution processing. The super-resolution image s is a high-quality high-resolution image having a higher resolution (resolution) than the low-resolution image y.

Hereinafter, an image group that is internally held to generate the super-resolution image s is referred to as a hypothesis group, the number of hypotheses constituting the hypothesis group is N (N is a natural number), and each hypothesis (image) is x ^(N) (n is an integer of 0 or more and less than N), and the hypothesis group is {x ⁽ⁿ⁾ }. The low-resolution image y, the super-resolution image s, and the hypothesis x ⁽ⁿ⁾ are pixel values of all the pixels constituting the image (scalar values of luminance in a grayscale image, each color component in a color image or multiband image, or This is expressed by arranging the luminance values of the respective wavelength components in a predetermined order as vectors (column vectors) (hereinafter referred to as image vectors). In the image vector, the order in which the pixel values are arranged is arbitrary, for example, the order of raster scanning.

Further, the image coordinates are set to p = [u, v] ^T. u is a horizontal coordinate, and v is a vertical coordinate. The pixel value of the low resolution image y at the image coordinate p is y (p), the pixel value of the super-resolution image s is s (p), and the pixel value of the hypothesis x ⁽ⁿ⁾ is x ⁽ⁿ⁾ (p). . For example, when the low-resolution image y is an image having a width L _x and a height L _y and the arrangement of pixel values in the image vector is the order of raster scanning from the upper left pixel to the lower right pixel of the image, the low resolution The image image y is represented by the following formula.

  When the switching unit 10-1 receives the low-resolution image y and the initialization signal and inputs the initialization signal having a true value for initializing the super-resolution processing, the input low-resolution image When y is output to the initialization unit 20 and a false value initialization signal for performing super-resolution processing is input, the input low-resolution image y is output to the likelihood calculation unit 70.

  The switching means 10-2 receives the output signal from the initialization means 20, the output signal from the observation update means 80, and the initialization signal. When a true value initialization signal is input, the switching means 10-2 Is output to the representative value calculation means 30 and the resampler 90, and when an initialization signal of a false value is input, the output signal from the observation update means 80 is represented by the representative value calculator 30 and the resampler. Output to 90.

The initialization unit 20 receives the low resolution image y and initializes the N hypotheses x ⁽ⁿ⁾ (higher resolution than the low resolution image y) in order to initialize the super-resolution processing. A candidate for the super-resolution image s) and N weights w ⁽ⁿ⁾ are generated, and the hypothesis group {x ⁽ⁿ⁾ } and the weight group {w ⁽ⁿ⁾ } are represented by the representative value calculating means 30 and the re-sampling means 90. Output to.

The representative value calculation means 30 inputs the hypothesis group {x ⁽ⁿ⁾ } and the weight group {w ⁽ⁿ⁾ } from the initialization means 20, or the prediction hypothesis group {x ⁽ⁿ⁾ } and the observation update means 80. The update weight group {w ⁽ⁿ⁾ } is input, the representative value of the hypothesis x ⁽ⁿ⁾ is calculated using the weight w ⁽ⁿ⁾ , and this representative value is output as the super-resolution image s.

The latch means 40 inputs a new hypothesis group {x ⁽ⁿ⁾ } and a new weight group {w ⁽ⁿ⁾ } resampled (reconstructed) by the resampler 90 described later, and temporarily Hold it and output it to the state transition means 50.

The state transition means 50 inputs the hypothesis group {x ⁽ⁿ⁾ } and the weight group {w ⁽ⁿ⁾ } from the latch means 40 and uses the probability density function P to determine the next cycle or the next time (frame). predict hypotheses, and outputs the observation value prediction unit 60 as prediction hypotheses ^{{x (n)},} observing updating means predicted hypotheses ^{{x (n)}} and input weighting group ^{{w (n)}} Output to 80. As the probability density function P, for example, a function that easily generates a hypothesis in which the value of the total variation is small, or a process in which an image moves, deforms, blurs and blurs, changes in luminance, and noise is added. A function that formulates is used. This probability density function P is sufficiently affected by the pixel values constituting the low resolution image when generating a high resolution image from the low resolution image, so that the pixel value does not vary so much. This is a function for realizing regularization. Details of the probability density function P will be described later.

The observation value predicting means 60 receives the prediction hypothesis group {x ⁽ⁿ⁾ } from the state transition means 50, and uses the observation function h that simulates the image degradation process, from the hypothesis x ⁽ⁿ⁾ that is a high resolution image. The observed predicted value z ⁽ⁿ⁾ , which is a low-resolution image, is predicted, and the observed predicted value z ⁽ⁿ⁾ is output to the likelihood calculating means 70.

The likelihood calculating means 70 inputs the observed predicted value z ⁽ⁿ⁾ from the observed value predicting means 60, and at the same time the cycle of the hypothetical group {x ⁽ⁿ⁾ } predicted by the low-resolution image y (the state transition means 50 ⁾ or The low resolution image y) corresponding to the time (frame) is input, the observed predicted value z ⁽ⁿ⁾ is compared with the low resolution image y, the similarity between the images is quantified, and the likelihood L ⁽ⁿ⁾ Is output to the observation update means 80. The likelihood L ⁽ⁿ⁾ becomes a larger value as the similarity between images is higher.

The observation update unit 80 receives the likelihood L ⁽ⁿ⁾ from the likelihood calculation unit 70 and also inputs the prediction hypothesis group {x ⁽ⁿ⁾ } and the weight group {w ⁽ⁿ⁾ } from the state transition unit 50. The weight w ⁽ⁿ⁾ is updated based on the likelihood L ⁽ⁿ⁾ , and the prediction hypothesis group {x ⁽ⁿ⁾ } and the update weight group {w ⁽ⁿ⁾ } are transferred to the representative value calculation unit 30 and the re-sampling unit 90. Output.

The resampling unit 90 inputs the hypothesis group {x ⁽ⁿ⁾ } and the weight group {w ⁽ⁿ⁾ } from the initialization unit 20, or the prediction hypothesis group {x ⁽ⁿ⁾ } and the observation update unit 80. The update weight group {w ⁽ⁿ⁾ } is input, and based on the weight w ⁽ⁿ⁾ , the hypothesis x ⁽ⁿ⁾ is copied or deleted, and the new hypothesis group {x _new ⁽ⁿ⁾ } and the new weight group { Resample to w _new ⁽ⁿ⁾ } and output to the latch means 40.

[Operation of signal processor]
Next, the operation of the signal processing apparatus 1 shown in FIG. 1 will be described. FIG. 2 is a flowchart for explaining the operation of the signal processing apparatus 1. In FIG. 2, the low resolution image y input by the signal processing apparatus 1 is represented by frame numbers t = 0, 1,. . . , T, and the signal processing apparatus 1 generates and outputs a super-resolution image s in one processing cycle for each frame constituting the low-resolution image y as super-resolution processing. Shall.

First, the signal processing device 1 sets the frame number t = 0 for the input low resolution image y (step S1). Then, the initialization unit 20 of the signal processing device 1 uses a hypothesis group {x ⁽ⁿ⁾ } and a weight group as initial hypotheses based on the frame (frame 0) of the frame number t = 0 in the input low resolution image y. {W ⁽ⁿ⁾ } is generated (step S2).

The resampling means 90 duplicates or deletes the hypothesis x ⁽ⁿ⁾ based on the weight w ^(n), and creates a new hypothesis group {x _new ⁽ⁿ⁾ } and a new weight group {w _new ⁽ⁿ⁾ }. Resampling is performed (step S3). The resampled new hypothesis group {x _new ⁽ⁿ⁾ } and the new weight group {w _new ⁽ⁿ⁾ } are output to the latch means 40 and held. Then, the signal processing device 1 increments the frame number t (t = t + 1) (step S4).

The state transition means 50 inputs the hypothesis group {x ⁽ⁿ⁾ } held by the latch means 40, causes the hypothesis group {x ⁽ⁿ⁾ } to undergo state transition using the probability density function P, and is incremented in step S4. A new hypothesis group with a new frame number t is predicted to generate a prediction hypothesis group {x ⁽ⁿ⁾ } (step S5). Then, the observed value predicting unit 60 substitutes the prediction hypothesis group {x ⁽ⁿ⁾ } generated by the state transition unit 50 into an observation formula using the observation function h, and the observed predicted value z that is a low-resolution image. ^(N) is predicted (step S6).

The likelihood calculating means 70 compares the observed predicted value z ^{(n) of} the frame number t generated by the observed value predicting means 60 with the frame of the frame number t in the low resolution image y, and the similarity between the images. And the likelihood L ⁽ⁿ⁾ is calculated (step S7).

The observation update unit 80 updates the weight w ⁽ⁿ⁾ based on the likelihood L ⁽ⁿ⁾ calculated by the likelihood calculation unit 70 (step S8).

The representative value calculating means 30 uses the weight w ⁽ⁿ⁾ and the hypothesis x from the prediction hypothesis group {x ⁽ⁿ⁾ } generated in step S5 and the update weight group {w ⁽ⁿ⁾ } updated in step S8. The representative value of ⁽ⁿ⁾ is calculated, and this representative value is output as the super-resolution image s (step S9).

  The signal processing device 1 determines whether or not the frame number t is equal to the final frame number T (t = T) (step S10). That is, it is determined whether or not the processing of all frames (frame numbers t = 0, 1,..., T) in the low resolution image y is completed. When the signal processing apparatus 1 determines in step S10 that t <T (step S10: N), that is, when it is determined that the processing of all the frames in the low resolution image y is not completed, the step The process proceeds to S3, and the next frame processing (steps S3 to S9) is performed. On the other hand, if the signal processing apparatus 1 determines in step S10 that t = T (step S10: Y), that is, determines that all the frames in the low-resolution image y have been processed, Exit.

The signal processing apparatus 1 uses the internal state (resampled new hypothesis group {x _new ⁽ resampled)) generated in the processing cycle before the current processing cycle, which is a series of processing units in steps S3 to S10 shown in FIG. ⁿ⁾ } and the new weight group {w _new ⁽ⁿ⁾ }) are held in the latch means 40. Then, the signal processing device 1 performs sequential processing while referring to the internal state held in the latch unit 40, and generates a super-resolution image s in the current processing cycle.

  As described above, when the low-resolution image y is a moving image with frame numbers t = 0, 1,..., T, the signal processing device 1 performs the super-resolution processing shown in FIG. A super-resolution image s is generated and output in one processing cycle for each frame constituting the low-resolution image y.

  Here, the processing cycle of step S3 to step S10 shown in FIG. 2 may be linked to physical time, may not be linked, or may be a combination thereof. .

  When the processing cycle is linked to physical time, the signal processing apparatus 1 inputs a low-resolution image y that is a moving image and configures a moving image, for example, as shown in the flowchart of FIG. The processing of the super-resolution method is realized by making one cycle of the processing cycle with the update of the frame (still image) to be performed.

When the processing cycle is not synchronized with the physical time, the signal processing apparatus 1 inputs, for example, the low-resolution image y that is a still image, repeats one or more processing cycles, and performs this repeated calculation. Super-resolution processing is realized by converging at a predetermined number of times, or by converging when the difference between the super-resolution images s is within a predetermined value even after repeated calculations. In this case, in the flowchart shown in FIG. 2, the likelihood calculating means 70 compares the observed predicted value z ⁽ⁿ⁾ generated by the observed value predicting means 60 with the low resolution image y that is a still image. Then, the similarity between the images is quantified to calculate the likelihood L ⁽ⁿ⁾ . As the observed predicted value z ⁽ⁿ⁾ , a different value is generated for each processing cycle by the observed value predicting unit 60. However, the low resolution images y compared in the likelihood calculating unit 70 are the same regardless of the processing cycle. It is an image.

  Furthermore, when the case where the processing cycle is linked to the physical time and the case where the processing cycle is not linked are combined, the signal processing device 1 inputs the low resolution image y which is a moving image, and the individual frames are inputted. Super-resolution processing is realized by repeating one or more processing cycles to converge the repetitive operation and performing the same repetitive operation for the next new frame.

  Next, switching means 10-1 and 10-2, initialization means 20, representative value calculation means 30, latch means 40, state transition means 50, observation value prediction means 60, likelihood of the signal processing apparatus 1 shown in FIG. Each constituent unit of the degree calculation unit 70, the observation update unit 80, and the resampling unit 90 will be described in detail.

[Switching means]
First, the switching means 10-1 and 10-2 shown in FIG. 1 will be described in detail. The switching means 10-1 and 10-2 receive the initialization signal and change the output destination according to the value of the initialization signal while interlocking with each other according to the initialization signal.

  The initialization signal usually takes a false value and is the timing at which the super-resolution processing by the signal processing device 1 is initialized, for example, the first timing when the low-resolution image y is input to the signal processing device 1. Takes a true value. When the initialization process of the initialization unit 20 described later is completed, the value becomes a false value. That is, the initialization signal generation means (not shown) switches the initialization signal of the true value to the switching means 10-1, 10- at the timing when the signal processing apparatus 1 first inputs the low resolution image y or immediately before it. Output to 2. The initialization signal generation means monitors the timing when the initialization process by the initialization means 20 is completed, and outputs an initialization signal with a false value to the switching means 10-1 and 10-2 at that timing.

  It should be noted that the initialization input signal may be reversed to a true value. Further, the value of the initialization input signal may be changed by, for example, manual operation or automatic operation by an externally connected device. For example, it is necessary when the operator (or an external operating device) determines that the high-resolution super-resolution image s is not generated in order to initialize the super-resolution processing by the signal processing device 1. At the timing when the determination is made, the initialization signal generating means outputs a true value initialization signal to the switching means 10-1 and 10-2.

  When a true value initialization signal is input, the switching unit 10-1 uses a terminal to which the low resolution image y is input as an input terminal of the initialization unit 20 in order to initialize the super-resolution processing. And the input low resolution image y is output to the initialization means 20. On the other hand, when an initialization signal having a false value is input, a terminal to which the low resolution image y is input is connected to an input terminal of the likelihood calculating means 70 in order to perform super-resolution processing. The low resolution image y is output to the likelihood calculating means 70.

  When the true value initialization signal is input, the switching means 10-2 uses the output terminal of the initialization means 20 as the representative value calculating means 30 and the resampling means in order to initialize the super-resolution processing. The input signal of the initialization means 20 is output to the representative value calculation means 30 and the resampling means 90. On the other hand, when a false value initialization signal is input, the output terminal of the observation update means 80 is connected to the input terminals of the representative value calculation means 30 and the resampling means 90 in order to perform super-resolution processing. The output signal of the observation update means 80 is output to the representative value calculation means 30 and the resampling means 90.

[Initialization means]
Next, the initialization means 20 shown in FIG. 1 will be described in detail. The initialization unit 20 inputs a low resolution image y in order to initialize the super-resolution processing, and assumes a hypothesis x ⁽ⁿ⁾ and a weight w ⁽ higher resolution image than the low resolution image y. ⁿ⁾ are respectively generated (N is a natural number), and the hypothesis group {x ⁽ⁿ⁾ } and the weight group {w ⁽ⁿ⁾ } are output to the representative value calculation means 30 and the resampling means 90. Here, n = 0, 1,. . . , N−1, and each hypothesis x ⁽ⁿ⁾ constituting the hypothesis group {x ⁽ⁿ⁾ } may be the same value (image) or may include different values. The weight w ⁽ⁿ⁾ is a real number greater than or equal to zero. For example, the initialization unit 20 performs an interpolation process on the low resolution image y having the width L _x and the height L _y to generate a hypothesis x ⁽ⁿ⁾ having the width H _x and the height H _y .

前記式（２）において、以下の関数は、ｚより大きくない最大の整数（床関数により小数を丸めた数）を示す。

Initialization means 20, all when generating the hypothesis x ⁽ⁿ⁾ of the same value, for example, by the following equation, to generate a hypothesis x ⁽ⁿ⁾ at zero-order interpolation.

In the above equation (2), the following function represents a maximum integer (number obtained by rounding a decimal by a floor function) not larger than z.

前記式（３）において、Ｚは整数環であり、Ｚ^２は２次元の格子点の集合を示す。また、ａ（ｐ）は、画像座標ｐにおける補間フィルタａのタップ係数である。演算子＊は、２次元の畳み込み演算を示す。 Further, when the initialization unit 20 generates the hypothesis x ⁽ⁿ⁾ having the same value, for example, the initialization unit 20 generates the hypothesis x ⁽ⁿ⁾ using the interpolation filter a according to the following equation.

In the formula (3), Z is an integer ring, and Z ² represents a set of two-dimensional lattice points. Further, a (p) is a tap coefficient of the interpolation filter a at the image coordinate p. The operator * indicates a two-dimensional convolution operation.

としたものと等価である。 The zero-order interpolation of the equation (2) is performed by the interpolation filter a of the equation (3).

Is equivalent to

FIG. 3 is a diagram illustrating an example of the tap coefficient a (p) of the interpolation filter a. In this example, the tap coefficient a (p) (p = [u, v] ^T ) multiplied by 65536 is expressed with respect to the horizontal coordinate u and the vertical coordinate v, and the numerical value in the bold frame is divided by 65536. The result is the tap coefficient. Note that for all image coordinates [u, v] ^T (points other than grid points, | u |> 3 or | v |> 3) other than the horizontal coordinate u and the vertical coordinate v shown in FIG. Is assumed to be 0.

尚、雑音ｅ（ｐ）は、画像座標ｐ毎及び仮説ｘ^（ｎ）毎に、ある確率過程に基づいて生成され、画像座標ｐ毎及び仮説ｘ^（ｎ）毎には必ずしも同一の値をとらない。 In addition, when generating a hypothesis x ⁽ⁿ⁾ that is not all the same value, the initialization unit 20 adds a noise e to the equation (3 ⁾ to generate a hypothesis x ^(n), for example. When the noise e has additive characteristics, a hypothesis x ⁽ⁿ⁾ is generated by the following equation.

Incidentally, the noise e (p), for each image coordinates p and for each hypothesis x ^(n), is generated based on some stochastic process, not necessarily take the same value for each image coordinates p and for each hypothesis x ⁽ⁿ⁾ Absent.

For example, the noise e may be Gaussian noise having an average of 0 and a variance σ ² as shown in the following equation.

Furthermore, the initialization means 20, when all generates a hypothesis x ⁽ⁿ⁾ are not identical values, for example, for each hypothesis x ^(n), to generate a hypothesis x ⁽ⁿ⁾ by changing the interpolation method. Specifically, the initialization unit 20 uses another interpolation filter a in the equation (3) or the equation (4) (that is, one having a different tap coefficient ^{) for} each hypothesis x ^(n). , Generate hypothesis x ⁽ⁿ⁾ .

On the other hand, the initialization unit 20 generates a real number of 0 or more as a weight w ⁽ⁿ⁾ for each hypothesis x ⁽ⁿ⁾ . The initialization unit 20 may generate the weight w ⁽ⁿ⁾ at random for each hypothesis x ⁽ⁿ⁾ or may generate a weight w ⁽ⁿ⁾ having a different value. Further, the initialization unit 20 may generate the same weight w ⁽ⁿ⁾ for all hypotheses x ⁽ⁿ⁾ by the following formula.

[Representative value calculation means]
Next, the representative value calculation means 30 shown in FIG. 1 will be described in detail. The representative value calculation means 30 inputs the hypothesis group {x ⁽ⁿ⁾ } and the weight group {w ⁽ⁿ⁾ } from the initialization means 20, or the prediction hypothesis group {x ⁽ⁿ⁾ } and the observation update means 80. The update weight group {w ⁽ⁿ⁾ } is input, the representative value of the hypothesis x ⁽ⁿ⁾ is calculated using the weight w ⁽ⁿ⁾ , and this representative value is output as the super-resolution image s.

The representative value calculating means 30 calculates a weighted average by the following formula, for example, and obtains a representative value of the hypothesis x ⁽ⁿ⁾ .

The representative value calculation means 30 may obtain a hypothesis x ⁽ⁿ⁾ having the maximum weight w ⁽ⁿ⁾ as a representative value from the hypothesis group {x ⁽ⁿ⁾ }. Further, the representative value calculating means 30 clusters the hypothesis group {x ⁽ⁿ⁾ }, obtains a cluster having the maximum sum of the weights w ⁽ⁿ⁾ from the clustered clusters, and the hypothesis x ⁽ belonging to the cluster ^). Only for ⁿ⁾ , a weighted average may be calculated and the calculation result may be a representative value of the hypothesis x ⁽ⁿ⁾ .

[Latch means]
The latch means 40 receives the new hypothesis group {x _new ⁽ⁿ⁾ } and the new weight group {w _new ⁽ⁿ⁾ } resampled (reconstructed) by the resampling means 90 and temporarily receives them. Hold it and output it to the state transition means 50.

[State transition means 50]
Next, the state transition means 50 shown in FIG. 1 will be described in detail. Hereinafter, hypotheses and weights in the c-th cycle at time t (cε {0, 1,..., C _t −1}, C _t is a natural number) are _{expressed as} x _{t, c} ⁽ⁿ⁾ and w _{t, c} ^{( n)} . It is assumed that there are a total of C _t cycles at each time t. Further, as described above, when the processing cycle is not linked to the physical time, it is only necessary to fix t and operate so that only c is counted. On the other hand, as described above, when the processing cycle is linked to the time, the operation may be performed so that only t is counted while c = 0 is fixed (that is, C _t = 1).

The state transition means 50 inputs the hypothesis group {x ⁽ⁿ⁾ } and the weight group {w ⁽ⁿ⁾ } from the latch means 40, predicts the hypothesis group of the next cycle or the next time (frame), and predicts the hypothesis. The group {x ⁽ⁿ⁾ } is output to the observation value prediction unit 60 and the prediction hypothesis group {x ⁽ⁿ⁾ } and the input weight group {w ⁽ⁿ⁾ } are output to the observation update unit 80 as they are. Specifically, the state transition unit 50 inputs the hypothesis group {x _{t, c} ⁽ⁿ⁾ } and the weight group {w _{t, c} ⁽ⁿ⁾ } at the time t and the cycle c from the latch unit 40, and the hypothesis group From {x _{t, c} ⁽ⁿ⁾ }, a hypothesis group {x _{t, c + 1} ⁽ⁿ⁾ } at time t, cycle c + 1, or a hypothesis group {x _{t + 1,0} ⁽ⁿ⁾ } at time t + 1, cycle 0 is predetermined. The probability density function P is used for prediction. Then, the state transition means 50 outputs the predicted hypothesis group {x _{t, c + 1} ⁽ⁿ⁾ } or the hypothesis group {x _{t + 1,0} ⁽ⁿ⁾ } to the observation value prediction means 60, and the hypothesis group {x _{t, c + 1} ^(N) } or the hypothesis group {x _{t + 1,0} ⁽ⁿ⁾ } and the input weight group {w _{t, c} ⁽ⁿ⁾ } are output to the observation update means 80. Hereinafter, the prediction calculation by the state transition using the probability density function P will be described in detail.

(When predicting a hypothesis group {x _{t, c + 1} ⁽ⁿ⁾ })
First, a case where a hypothesis group {x _{t, c + 1} ⁽ⁿ⁾ } at time t and cycle c + 1 is predicted from a hypothesis group {x _{t, c} ⁽ⁿ⁾ } at time t and cycle c will be described. When the cycle c is c <C _t −1 (and the super-resolution image s to be output at the time t has not converged), the state transition means 50 determines the hypothesis group at the time t and the cycle c + 1. Predict {x _{t, c + 1} ⁽ⁿ⁾ }.

ここで、Ｐ（ｘ_{ｔ，ｃ＋１}｜ｘ_ｔ，ｃ）は、仮説ｘ_ｔ，ｃが与えられたときに、次サイクルｃ＋１における仮説ｘ_{ｔ，ｃ＋１}のとる確率密度関数をモデル化したものである。 The update (state transition) of the hypothesis group when increasing from cycle c to c + 1 is defined by equation (9).

Here, P (x _{t, c + 1} | x _{t, c} ) models the probability density function taken by the hypothesis x _{t, c + 1} in the next cycle c + 1 when the hypothesis x _{t, c} is given. .

つまり、トータルバリエーションＶは、画素値が増加する区間についてはそのままの値を用い、画素値が減少する区間については符号を反転した値を用いて、それぞれの傾きを積分して得られた値である。 Here, the probability density function P (x _{t, c + 1} | x _{t, c} ) is a function having a function of stabilizing the hypothesis x so that the hypothesis x does not become an excessively complicated pixel value pattern as an image as the cycle c increases. Is used. That is, a probability density function P (x _{t, c + 1} | x _{t, c} ) that changes the pixel value of the hypothesis x based on a predetermined rule is used. For example, as the probability density function P (x _{t, c + 1} | x _{t, c} ), a function that can easily generate a hypothesis x _{t, c + 1 in} which the value of the total variation of the image is small is used. That is, a function is used that generates a hypothesis x _{t, c + 1} of the next cycle by making a state transition of the original hypothesis x _{t, c} so that the value of the total variation of the image becomes small. Here, the total variation means a sum of absolute values of gradients (differences between pixel values between adjacent pixels) in pixel values in an image. The total variation V is expressed by the following formula in the case of one dimension.

That is, the total variation V is a value obtained by integrating each slope using a value as it is for a section where the pixel value increases and a value obtained by inverting the sign for a section where the pixel value decreases. is there.

式（１１）において、Ｒは仮説の画像内における全画素位置の集合である。第１項のｅｘｐ部は、水平方向の画素においてトータルバリエーションの値を小さくする式であり、第２項のｅｘｐ部は、垂直方向の画素においてトータルバリエーションの値を小さくする式であり、第３項のｅｘｐ部は、現在の画素値から極端に変化させないようにする式である。また、Λ関数は、台形の特性を、富士山型のように裾広がりの特性に変換する関数である。つまり、台形の傾斜部の直線を裾広がりの曲線に変換する。このような確率密度関数Ｐ（ｘ_{ｔ，ｃ＋１}｜ｘ_ｔ，ｃ）を用いることにより、仮説ｘ_ｔ，ｃを構成する所定の注目点の画素値、及び、その注目点から所定範囲内の信号点の画素値により演算されるトータルバリエーションの値が、注目点の画素値を変更したときに小さくなるように、新たな仮説ｘ_{ｔ，ｃ＋１}が生成される。 As the probability density function P (x _{t, c + 1} | x _{t, c} ), for example, the following equation is used.

In Expression (11), R is a set of all pixel positions in the hypothetical image. The exp part of the first term is an expression for reducing the value of the total variation in the pixels in the horizontal direction, and the exp part of the second term is an expression for reducing the value of the total variation in the pixels in the vertical direction. The exp part of the term is an expression that prevents an extreme change from the current pixel value. The Λ function is a function that converts the trapezoidal characteristic into a flared characteristic like the Mount Fuji type. That is, the straight line of the trapezoidal inclined portion is converted into a curve with a skirt spread. By using such a probability density function P (x _{t, c + 1} | x _{t, c} ), a pixel value of a predetermined attention point constituting the hypothesis x _{t, c} and a signal within a predetermined range from the attention point. A new hypothesis x _{t, c + 1} is generated so that the total variation value calculated from the pixel value of the point becomes smaller when the pixel value of the target point is changed.

FIG. 4 is a diagram illustrating a probability density function that realizes regularization by total variation. As shown in FIG. 4, the characteristic of the probability density function P (x _{t, c + 1} | x _{t, c} ) is the first term characteristic, the second term characteristic, and the third term characteristic according to equation (11). It is synthesized. That is, regularization noise that reduces the total variation of the image is generated by the probability density function P (x _{t, c + 1} | x _{t, c} ) shown in Expression (11).

  When the hypothesized pixel value takes a vector value, such as a color image (or multiband image), Equation (11) is independently applied to each color (or each band) component.

(When predicting a hypothesis group {x _{t + 1,0} ⁽ⁿ⁾ })
Next, a case where the hypothesis group {x _{t + 1,0} ⁽ⁿ⁾ } at time t + 1 and cycle 0 is predicted from the hypothesis group {x _{t, c} ⁽ⁿ⁾ } at time t and cycle c will be described. When the cycle c is c = C _t −1 (or when the super-resolution image to be output at time t has converged), the state transition means 50 causes the hypothesis group {at time t + 1, cycle 0 { x _{t + 1,0} ⁽ⁿ⁾ } is predicted.

ここで、前記式（１２）の確率密度関数Ｐは、仮説

が与えられたときに、次時刻における仮説ｘ_{ｔ＋１，０}のとる確率密度関数をモデル化したものである。 The update (state transition) of the hypothesis group at this time is defined by the following equation.

Here, the probability density function P of the equation (12) is a hypothesis.

Is a model of the probability density function taken by the hypothesis x _{t + 1,0 at} the next time.

式（１３）では、まず、仮説

を関数φにより変換する。Σは共分散行列である。つまり、式（１３）は、関数φの変換結果を平均値にもつ正規分布の確率密度となる。すなわち、式（１２）により生成される仮説ｘ_{ｔ＋１，０}は、平均が

共分散行列がΣのガウス雑音である。 Further, the probability density function P is assumed to formulate a process in which the image represented by the hypothesis x moves, deforms, blurs and blurs, changes in luminance, and adds noise as time t increases. For example, the following equation can be obtained from the normal distribution N.

In equation (13), first, hypothesis

Is converted by the function φ. Σ is a covariance matrix. That is, equation (13) is the probability density of a normal distribution having the conversion result of the function φ as an average value. That is, the hypothesis x _{t + 1,0} generated by Equation (12)

The covariance matrix is Gaussian noise with Σ.

ここで、各φ_ｍ（ｍは０以上Ｍ未満の整数）には、例えば、画像の移動、変形、輝度の変化等の各変換機能が与えられる。 An example of how to define the function φ will be described. The function φ is a function expressed by combining operations such as image movement and deformation. That is, the function φ can be expressed as a composite function of M (M is a natural number) functions φ _{0 to} φ _{M−1 as in} the following equation.

Here, each φ _m (m is an integer of 0 or more and less than M) is given, for example, each conversion function such as image movement, deformation, and luminance change.

For example, when the function φ _m is a function expressing the movement of the image, the image is moved by a motion vector v according to a two-dimensional normal distribution having an average of 0 and a covariance matrix Σ _v as shown in the following equation.

When the function φ _m is a function expressing affine transformation of an image, the image is deformed by an affine parameter according to _a 6-dimensional normal distribution with _a mean of 0 and a covariance matrix Σa as in the following equation.

Further, when the function φ _m is a function expressing the luminance offset of the image, the image is generated by the luminance offset according to the one-dimensional normal distribution of mean 0 and variance matrix σ _ω ² as in the following equation.

As described above, the state transition unit 50 uses the probability density function P that easily generates the hypothesis x _{t, c + 1 in} which the value of the total variation is small, from the hypothesis group {x _{t, c} ⁽ⁿ⁾ } to the hypothesis group {x _{t, c + 1} ⁽ⁿ⁾ } is predicted. Further, the state transition means 50 uses a probability density function P that formulates a process in which an image moves, deforms, blurs and blurs, changes in luminance, and noise is added, to a hypothesis group {x _{t , C} ⁽ⁿ⁾ } predict a hypothesis group {x _{t + 1,0} ⁽ⁿ⁾ }.

(Observed value predicting means 60)
Next, the observation value predicting means 60 shown in FIG. 1 will be described in detail. The observation value predicting means 60 inputs the prediction hypothesis group {x ⁽ⁿ⁾ } (the hypothesis group {x _{t, c + 1} ⁽ⁿ⁾ } or the hypothesis group {x _{t + 1,0} ⁽ⁿ⁾ }) from the state transition means 50, An observation predicted value z ⁽ⁿ⁾ that is a low-resolution image is predicted from a hypothesis x ⁽ⁿ⁾ that is a high-resolution image, and the observation predicted value z ⁽ⁿ⁾ is output to the likelihood calculating means 70. The observation value predicting means 60 is equipped with processing means for generating a low resolution image from a high resolution image (hereinafter referred to as observation). The observation value predicting means 60 is, for example, a high resolution image. On the other hand, by simulating the image degradation process due to blurring, sampling, quantization, etc., observation is performed and a low resolution image is generated.

Assuming that the conversion operation by observation is a function h (hereinafter referred to as an observation function), the observation value predicting means 60 performs observation on each hypothesis image x ⁽ⁿ⁾ , and the result is an observation prediction value z ⁽ⁿ⁾ = h. It is generated as (x ⁽ⁿ⁾ ).

The observation function h may be a combined function of a plurality of factors (blur, blur, sampling, quantization, etc.). For example, when the observation function h is composed of J factors (J is a natural number), and the j-th factor (j is an integer greater than or equal to 0 and less than J) is the function h _j, it is expressed by the following equation. .

For example, when the function h _j is a function representing blur or blur, it is assumed that a convolution operation (operator *) with the spread function s (definition area is defined as region C) of the point is performed according to the following formula: Is done.

とする。ここで、Σ_ｑ ^−１は２行２列の共分散行列である。また、定義域Ｃは、半径Ｒの円内（円周を含む）として定義される。

ここで、｜｜ｑ｜｜はベクトルｑのノルムを示し、Ｚは整数環を示す。ノルムとしては、例えば、ユークリッドノルム（Ｌ_２ノルム）、マンハッタンノルム（Ｌ_１ノルム）、チェビシェフノルム（Ｌ_∞ノルム）等が用いられる。例えば、半径をＲ＝１、ノルムをチェビシェフノルムとすると、３画素×３画素の矩形の畳み込みカーネルとなる。 As the point spread function s, for example, a Gaussian kernel represented by the following equation is used.

And Here, Σ _q ⁻¹ is a 2 × 2 covariance matrix. The definition area C is defined as a circle (including the circumference) having a radius R.

Here, || q || represents the norm of vector q, and Z represents an integer ring. As the norm, for example, Euclidean norm (L ₂ norm), Manhattan norm (L ₁ norm), Chebyshev norm (L _∞ norm), or the like is used. For example, if the radius is R = 1 and the norm is a Chebyshev norm, a rectangular convolution kernel of 3 pixels × 3 pixels is obtained.

ここで、行列Ｓは標本化格子の標本化間隔及びシアーを決定する行列である。例えば、行列Ｓとして、対角行列Ｓ＝ｄｉａｇ（Ｓ_ｘ，Ｓ_ｙ）を用いると、水平方向に間隔Ｓ_ｘ、垂直方向に間隔Ｓ_ｙの標本化格子となる。一方、ベクトルｕは、標本化格子の標本化位相を決定するベクトルである。このベクトルにより、標本化の開始点（原点）の位置は、画像ｘ_ｉｎ上における画像座標ｕとなる。 Further, when the function h _j is a function representing sampling, for example, the function h _j is calculated by the following expression using a 2 × 2 matrix S and a two-dimensional column vector u.

Here, the matrix S is a matrix that determines the sampling interval and shear of the sampling grid. For example, when a diagonal matrix S = diag (S _x , S _y ) is used as the matrix S, a sampling grid with a spacing S _{x in} the horizontal direction and a spacing S _{y in} the vertical direction is obtained. On the other hand, the vector u is a vector that determines the sampling phase of the sampling grid. This vector, the position of the starting point of sampling (origin) is an image coordinate u on the image x _in.

For example, in the case where the horizontal origin is S _x , the vertical direction is S _y , and the sampling origin on x _in is (u _x , u _y ), the matrix S and the image coordinates u are as follows: It is expressed by the following formula.

ここで、関数Ｑ（ｐ）は、画像座標ｐに最も近い格子点の画像座標を出力する関数である。 In the equation (22), if there is a possibility that Sp + u does not exist on the lattice point, the following equation may be used instead of the equation (22).

Here, the function Q (p) is a function for outputting the image coordinates of the lattice point closest to the image coordinates p.

ここで、関数

は、入出力間の輝度変換または量子化を表す関数である。 Further, when the function h _j is a function representing luminance conversion or quantization of a pixel value, for example, it is calculated by the following expression.

Where the function

Is a function representing luminance conversion or quantization between input and output.

ここで、Ｘ_０及びＸ_１は、それぞれ画素値（輝度値）の最小値（黒レベル）及び最大値（白レベル）である。 For example, gamma correction with a gamma value of γ is expressed by the following equation.

Here, X ₀ and X ₁ are the minimum value (black level) and the maximum value (white level) of the pixel value (luminance value), respectively.

ここで、Ψは量子化ステップ、ωは量子化の際の丸めの方法を決めるオフセットである。例えば、量子化ステップ２にて四捨五入による量子化を行う場合には、Ψ＝２，ω＝１とすればよい。 The quantization is expressed by the following equation.

Here, Ψ is a quantization step, and ω is an offset that determines a rounding method at the time of quantization. For example, when performing quantization by rounding in quantization step 2, ψ = 2 and ω = 1 may be used.

As described above, the observation value predicting means 60 uses the observation function h that simulates the image degradation process such as blurring, sampling, and quantization to convert the hypothesis x ⁽ⁿ⁾ that is a high resolution image into a low resolution image. A certain observed predicted value z ⁽ⁿ⁾ is predicted.

(Likelihood calculation means 70)
Next, the likelihood calculating means 70 shown in FIG. 1 will be described in detail. The likelihood calculating means 70 inputs the observed predicted value z ⁽ⁿ⁾ from the observed value predicting means 60 and at the same time the low resolution image y (the hypothesis group {x _{t, c + 1} ⁽ⁿ⁾ } predicted by the state transition means 50 ⁾ . _{Alternatively,} the low resolution image y) corresponding to the cycle or time (frame) of the hypothesis group {x _{t + 1,0} ⁽ⁿ⁾ } is input. Then, the likelihood calculating means 70 compares the observed predicted value z ⁽ⁿ⁾ and the low resolution image y, quantifies the similarity between images, and calculates the likelihood L ⁽ⁿ⁾ (similarity between images ^). The degree is calculated to be a likelihood L ⁽ⁿ⁾⁾ and output to the observation update means 80.

ここで、σ_Distanceは、距離に対する尤度の下がり具合を決めるための正の定数である。 The likelihood L ⁽ⁿ⁾ may be calculated based on the distance d ⁽ⁿ⁾ between the observed predicted value z ⁽ⁿ⁾ and the low resolution image y. Degree L ⁽ⁿ⁾ is defined by an exponential function of distance d ⁽ⁿ⁾ .

Here, σ _Distance is a positive constant for determining the degree of decrease in the likelihood with respect to the distance.

式（２９）の下付きのｋは、Ｌ_ｋノルムのｋであり、ｋ＝０のときハミング距離、ｋ＝１のときマンハッタン距離、ｋ＝２のときユークリッド距離、ｋ＝∞のときチェビシェフ距離となる。 The distance d ⁽ⁿ⁾ is calculated by, for example, the L _k norm between the observed predicted value z ⁽ⁿ⁾ and the low resolution image y.

The subscript k in the equation (29) is k of L _k norm, Hamming distance when k = 0, Manhattan distance when k = 1, Euclidean distance when k = 2, and Chebyshev distance when k = ∞. It becomes.

ここで、Σ_Mahalanobisは、マハラノビス距離を計算するための共分散行列である。Σ_Mahalanobisを単位行列とすると、尤度Ｌ^（ｎ）は、観測予測値ｚ^（ｎ）と低解像画像ｙとの間のユークリッド距離になる。また、Σ_Mahalanobisを対角行列とし、各対角要素に、対応する画素位置への分散値を設定すると、分散が小さく設定された画素ほど、尤度Ｌ（ｎ）に大きく影響を与えることができる。 The distance d ⁽ⁿ⁾ is calculated from the Mahalanobis distance between the observed predicted value z ⁽ⁿ⁾ and the low resolution image y.

Here, Σ _Mahalanobis is a covariance matrix for calculating the Mahalanobis distance. When Σ _Mahalanobis is a unit matrix, the likelihood L ⁽ⁿ⁾ is the Euclidean distance between the observed predicted value z ⁽ⁿ⁾ and the low resolution image y. In addition, when Σ _Mahalanobis is a diagonal matrix and a variance value at a corresponding pixel position is set for each diagonal element, a pixel whose variance is set to have a smaller influence on the likelihood L (n). it can.

Thus, the likelihood calculating means 70 compares the observed predicted value z ⁽ⁿ⁾ and the low resolution image y, quantifies the similarity between the images, and calculates the likelihood L ⁽ⁿ⁾ . The likelihood L ⁽ⁿ⁾ becomes a larger value as the similarity between images is higher.

(Observation update means 80)
Next, the observation update means 80 shown in FIG. 1 will be described in detail. The observation updating unit 80 inputs the likelihood L ⁽ⁿ⁾ from the likelihood calculating unit 70 and also inputs the prediction hypothesis group {x ⁽ⁿ⁾ } and the weight group {w ⁽ⁿ⁾ } from the state transition unit 50. . The observation updating means 80 updates the weights ^{w (n)} based on the likelihood ^{L (n)} for each hypothesis ^{x (n)} seeking new weight w _new ^(n), the predicted hypotheses ^{{x ( n)} } and the update weight group {w ⁽ⁿ⁾ } are output to the representative value calculation means 30 and the resampling means 90.

The new weight w _new ⁽ⁿ⁾ is obtained by the following equation, for example.

ここで、δはディラックのデルタ関数である。前記式（３２）の標本抽出は、以下の手順で実行される。 (Resampler 90)
Next, the resampling means 90 shown in FIG. 1 will be described in detail. The resampling means 90 receives the hypothesis group {x ⁽ⁿ⁾ } and the weight group {w ⁽ⁿ⁾ } from the initialization means 20 during the initial operation in response to the switching of the switching means 10-2. , The prediction hypothesis group {x ⁽ⁿ⁾ } and the update weight group {w ⁽ⁿ⁾ } are input from the observation update unit 80. Then, the resampling means 90 duplicates or deletes the hypothesis x ⁽ⁿ⁾ based on the weight w ^(n), and creates a new hypothesis group {x _new ⁽ⁿ⁾ } and a new weight group {w _new ^{(n )}} to be resampled output to the latch means 40. Resampling means 90, for example, take the sample from the hypothesis ^{x (n)} the probability density distribution for the D (x), the following equation, to generate a new hypothesis x _new ^(n).

Here, δ is a Dirac delta function. The sampling of the equation (32) is executed according to the following procedure.

First, the resampling means 90 normalizes the weights w ^(k) (k = 0, 1,..., N−1), and the sequence (α _k ) accumulated with respect to k is expressed by the following equation. Define.

Next, the resampling means 90 is the n th (n = 0, 1,..., N _new −1) (N _new is a natural number. Preferably, N _new = N specimens ^{u (n)} of the.), extracts from 0 or 1 less than the uniform distribution U [0, 1). This sampling process is approximately realized by normalizing the pseudo random number to 0 or more and less than 1 based on the range of values that the pseudo random number can take.

Then, the resampling means 90 generates the ^nth new hypothesis x _new ⁽ⁿ⁾ by the following equation.

On the other hand, the resampling means 90 sets a uniform value as a new weight w _new ⁽ⁿ⁾ by the following equation.

Note that the resampling means 90 may output the input hypothesis group {x ⁽ⁿ⁾ } and the weight group {w ⁽ⁿ⁾ } as they are without resampling. Further, resampling means 90 duplicates simply large hypothesis ^{x (n)} of the weight ^{w (n)} generates new hypotheses x _new ^(n), the sum of these weights w _new ⁽ⁿ⁾ May be resampled to be equal (or approximately equal ^{) to} the sum of the weights w ⁽ⁿ⁾ of the original hypothesis x ⁽ⁿ⁾ . Further, the resampling unit 90 may delete the hypothesis x ^{(n) in} which the weight w ⁽ⁿ⁾ is equal to or less than a predetermined threshold. Further, in the configuration of the signal processing device 1 shown in FIG. 1, the resampling means 90 itself may not exist. Further, the resampling means 90 may perform the reconstruction process every time (every process cycle) or once every plural times. Further, reconstruction may be performed without changing the number of hypothesis groups {x ⁽ⁿ⁾ } and weight groups {w ⁽ⁿ⁾ }, or the number may be increased or decreased.

As described above, according to the signal processing device 1 according to the embodiment of the present invention, the state transition means 50 is likely to generate a hypothesis where the total variation value is small, or the probability density function P or the image is moved and deformed. Predict a hypothesis group of the next cycle or the next time (frame) using a probability density function P that formulates a process in which blurring, blurring, luminance changes, noise, and the like are added, and a prediction hypothesis A group {x ⁽ⁿ⁾ } is generated. Further, the observation value predicting means 60 generates an observation predicted value z ⁽ⁿ⁾ by reducing the resolution of the prediction hypothesis group {x ⁽ⁿ⁾ } using the observation function h that simulates the image degradation process, and calculates the likelihood. The means 70 compares the observed predicted value z ⁽ⁿ⁾ and the low resolution image y, and calculates and calculates the likelihood L ⁽ⁿ⁾ having a larger value as the similarity between the images is higher. The new weight w _new ⁽ⁿ⁾ is obtained so that the weight w ⁽ⁿ⁾ increases as the likelihood L ⁽ⁿ⁾ increases, and the representative value calculation means 30 determines that the prediction hypothesis group {x ⁽ⁿ⁾ } and A representative value is calculated based on the update weight group {w ⁽ⁿ⁾ } and output as a super-resolution image s.

In this case, since the state transition means 50 implements a stochastic process such as total variation regularization and does not need to perform processing with a high calculation load due to repeated calculation as in the prior art, speeding up and efficiency improvement are possible. The signal processing apparatus 1 can be applied to a device that requires real-time processing. In addition, since the state transition means 50, the observation value prediction means 60, the likelihood calculation means 70, and the observation update means 80 can perform independent calculations for each hypothesis x ⁽ⁿ⁾ and can perform parallel calculations, It is compatible with computing chips and is suitable for incorporation into equipment. Therefore, when the super-resolution image s is generated by improving the resolution of the low-resolution image y, it is possible to increase the speed and efficiency of the calculation and generate the high-definition super-resolution image s. be able to.

  Further, according to the signal processing device 1 according to the embodiment of the present invention, the single super-resolution method of Patent Document 1 and the multiple-resolution super-resolution method of Patent Document 2 described above can be substituted by this single super-resolution method. can do. For this reason, hardware and software resources can be made more efficient in the processing of television images and the like in which moving images and still images are mixed.

  The present invention has been described with reference to the embodiment. However, the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the technical idea thereof. For example, although the above embodiment has been described with respect to image signals, the present invention is also applicable to other signals such as audio signals. In the case of targeting an audio signal, the signal processing device 1 generates and outputs a high-quality audio signal by the above processing. In the above embodiment, the process of converting the low resolution image y to the super resolution image s has been described. However, the present invention is a process of converting a blurred image into a sharp image. The removal process also has application.

In the above-described embodiment, the super-resolution processing for a two-dimensional image has been described. However, the present invention is also applicable to signals having a number of dimensions other than two-dimensional. In the embodiment, in the vector representation of the low resolution image y, the super resolution image s, the hypothesis x ^(n), etc., the pixel values are arranged in a prescribed order (for example, raster scanning order). Thus, a column vector was constructed. When a signal with an arbitrary number of dimensions other than two dimensions is to be processed, a column vector may be configured by arranging sample value groups constituting a dimensional signal in a prescribed order instead of this configuration order. In the case of a one-dimensional signal, for example, a column vector may be configured while maintaining the signal order. In the case of a three-dimensional signal (moving image), for example, a column vector may be configured by arranging the frames in the order of raster scanning and further arranging them in the time direction. That is, a signal having an arbitrary number of dimensions may be formed by scanning a line in a certain axial direction and arranging the result group in another axial direction and repeating it over all dimensions to form a column vector. .

Further, in the embodiment, the state transition means 50 of the signal processing apparatus 1 predicts hypotheses time t from ^{_{{x t, c (n)}} }, hypotheses in cycle _{c + 1 {x t, c} + 1 (n)} and At this time, using the probability density function P (x _{t, c + 1} | x _{t, c} ) of the equation (11), the state transition of the original hypothesis x _{t, c} is performed so that the value of the total variation of the image becomes small. In this way, the regularization based on the so-called total variation, which generates the hypothesis x _{t, c + 1} of the next cycle, is realized. The formula (11) is an example formula for reducing the total variation value of the image, and the present invention is not limited to this formula, and other formulas may be used. Good. Further, a probability density function that realizes regularization by a method other than total variation, for example, a smoothness (path) method may be used. This probability density function is sufficiently affected by the pixel value by relating the pixel values constituting the low resolution image with a curve when generating a high resolution image from the low resolution image, It is a function that realizes regularization so that pixel values do not vary so much.

  An ordinary computer can be used as the hardware configuration of the signal processing apparatus 1 according to the embodiment of the present invention. The signal processing apparatus 1 includes a computer having a volatile storage medium such as a CPU and a RAM, a non-volatile storage medium such as a ROM, an interface, and the like. Switching means 10-1, 10-2, initialization means 20, representative value calculation means 30, latch means 40, state transition means 50, observation value prediction means 60, likelihood calculation means 70, observation provided in the signal processing apparatus 1 The functions of the updating unit 80 and the resampling unit 90 are realized by causing the CPU to execute a program describing these functions. These programs can also be stored and distributed in a storage medium such as a magnetic disk (floppy (registered trademark) disk, hard disk, etc.), optical disk (CD-ROM, DVD, etc.), semiconductor memory, or the like.

1 Signal Processing Units 10-1, 10-2 Switching Unit 20 Initialization Unit 30 Representative Value Calculation Unit 40 Latching Unit 50 State Transition Unit 60 Observation Value Prediction Unit 70 Likelihood Calculation Unit 80 Observation Update Unit 90 Re-sampling Unit

Claims (7)

In a signal processing device that generates a high resolution signal from a low resolution signal,
Holding means for holding a plurality of candidates for the high resolution signal as a hypothesis group, and holding a weight for the candidate as a weight group;
State transition means for causing a state transition of a hypothesis group held in the holding means and generating a new hypothesis group;
An observation conversion means for converting a new hypothesis group generated by the state transition means into an observation signal having the same resolution as the low resolution signal;
A likelihood calculation means for comparing the observation signal converted by the observation conversion means with the low resolution signal, and calculating the similarity between the signals as a likelihood;
Updating the weight group based on the likelihood computed by the likelihood computing means, and generating a new weight group for the new hypothesis group;
Based on the new hypothesis group generated by the state transition means and the new weight group updated by the observation update means, the representative value of the hypothesis in the new hypothesis group is calculated as the high-resolution signal. Representative value calculating means;
A signal processing apparatus comprising: The signal processing device according to claim 1,
The state transition means changes the signal value of each signal point constituting a hypothesis of the hypothesis group based on a predetermined rule, and generates a new hypothesis group. The signal processing device according to claim 2,
The state transition means generates a new hypothesis group such that the total variation value indicating the sum of absolute values of the gradients of adjacent signal points becomes smaller with respect to the signal value of each signal point constituting the hypothesis of the hypothesis group. A signal processing device characterized by generating The signal processing device according to claim 3.
The state transition means includes a signal value of a target point in a signal constituting a hypothesis of the hypothesis group, and a total variation value calculated from a signal value of a signal point within a predetermined range from the target point. A signal processing apparatus, wherein a state of the hypothesis group is changed using a probability density function defined so as to decrease when a signal value is changed, and a new hypothesis group is generated. In the signal processing device according to any one of claims 1 to 4,
The signal is a still image signal,
A signal processing apparatus characterized in that a new hypothesis group and a new weight group are generated by a predetermined number of iterations by the state transition means, observation conversion means, likelihood calculation means, and observation update means. In the signal processing device according to any one of claims 1 to 4,
The signal is a moving image signal composed of a plurality of frames,
A new hypothesis group and a new weight group are generated for each frame or by a predetermined number of iterations in the frame by the state transition means, observation conversion means, likelihood calculation means, and observation update means. A signal processing device characterized by that.   The signal processing program for functioning a computer as a signal processing apparatus as described in any one of Claim 1-6.

Images