JP2006101032A - Interpolation image generating apparatus, interpolation image generating method, and interpolation image generating program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an interpolation image generating apparatus or the like capable of reducing noise such as "irregular impulse noise" caused by substantially different pixel values intruded into an interpolation frame. <P>SOLUTION: The interpolation image generating apparatus includes: a likelihood calculation section 113 for calculating a likelihood on the basis of a distribution of pixel information inside a coincident region in a first block on a first reference image determined by a first motion vector and a second block on a second reference image; a prior probability calculation section 114 for calculating a prior probability denoting the probability of a spatial distribution in the coincident region and the non-coincident region; a posterior probability calculation section 115 for calculating a posterior probability denoting the likelihood of the division of the coincident region and the non-coincident region on the basis of the likelihood and the prior probability; a discrimination section 116 for discriminating that the coincident region and the non-coincident region are divided by a most probable region division; and a motion compensation section 120 for applying motion compensation only to the coincident region on the basis of the first and second reference images and the first motion vector so as to generate an interpolation image when the discrimination section 116 discriminates the most probable region division. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an interpolation image generation apparatus, an interpolation image generation method, and an interpolation image generation program for generating an interpolation image to be interpolated between a first reference image and a second reference image of a moving image.

  In order to display a moving image smoothly, a frame interpolation method for interpolating an image between frames of the moving image is used. As the frame interpolation method, there are the following methods.

  First, an interpolation frame is divided into interpolation target blocks having a uniform lattice, and a motion vector is obtained by calculating a correlation between two reference frames geometrically symmetrically about the interpolation target block. Then, it is determined whether the correlation between the frames, that is, the absolute value of the difference between the pixel values between the two reference frames is equal to or less than a certain threshold value, and the pixel area within the certain threshold value within the block is regarded as a matching area. A pixel region larger than the threshold is divided into two regions as a mismatch region. Then, the motion vector is assigned to the matching area, and the mismatching area is recursively searched to generate an interpolated frame image (see, for example, Patent Document 1).

  According to the technique disclosed in Patent Document 1, a frame is divided into a matching area and a mismatching area, and motion vectors are obtained by different methods in each area. There is an advantage that block distortion does not occur even if there is a motion of.

JP 2004-104666 A (see FIGS. 15 and 16)

  However, in the technique of Patent Document 1, when dividing into a matching area and a mismatching area, a pixel value difference absolute value between two reference frames is calculated for each pixel, and whether the pixel belongs to the matching area by the difference absolute value. Since it is determined whether it belongs to the non-matching region, the matching determination is performed at a very fine level for the entire screen.

  Actually, even when a moving image is observed at a global level, even a portion that is observed to be very close between frames may be greatly different between frames at a fine pixel level. In such a case, the pixel level match determination result and the global level match determination result do not necessarily match.

  For this reason, when an interpolation frame is generated by performing coincidence determination at the pixel level, when the moving image is observed from the entire screen, it may be observed as an operation different from the actual operation. In other words, block distortion can be reduced by dividing the area in an ideal case where there is little noise in the image. However, in the case of an image where there is a lot of noise, the pixel value that is different from the original value is entered in the interpolation frame due to the noise. There is a problem of being visually recognized as noise such as sesame salt noise.

  The present invention has been made in view of the above, and an interpolation image generation device and an interpolation image generation capable of reducing noise such as sesame salt noise and displaying a moving image smoothly by reducing block distortion A method and an interpolated image generation program are provided.

  In order to solve the above-described problems and achieve the object, the present invention is an interpolation image generation apparatus that generates an interpolation image to be interpolated between a first reference image and a second reference image of a moving image, A motion search means for obtaining a first motion vector from the first reference image and the second reference image, and a region for dividing a block of a predetermined size in the first reference image into a matching region and a mismatching region The coincidence in the first block on the first reference image and the second block on the second reference image determined by the dividing means and the first motion vector obtained by the motion search means A likelihood calculating means for calculating a likelihood based on a distribution of pixel information inside the area; a prior probability calculating means for calculating a prior probability indicating a probability of a spatial distribution of the matched area and the mismatched area; and the likelihood. In calculation means A posteriori probability calculating means for calculating, based on the Bayes' theorem, a posterior probability indicating the likelihood of division of the matched area and the mismatched area from the likelihood calculated by the prior probability calculating means and the prior probability calculated by the prior probability calculating means And, when the posterior probability calculated by the posterior probability calculating unit has a maximum value, the determination of determining that the matching region and the mismatching region divided by the region dividing unit are likely region divisions And the determining unit determines that the matching region and the non-matching region are likely region divisions from the first reference image, the second reference image, and the first motion vector. Motion compensation means for performing motion compensation based only on the matching region and generating the interpolated image.

  The present invention also provides an interpolation image generation method and an interpolation image generation program corresponding to the interpolation image generation apparatus.

  According to the present invention, based on the distribution of pixel information inside the matching region in the first block on the first reference image and the second block on the second reference image defined by the first motion vector. The likelihood is calculated, the prior probability indicating the probability of the spatial distribution of the coincidence region and the disagreement region is calculated, and the likelihood of the region division of the coincidence region and the disparity region is indicated from the calculated likelihood and the calculated prior probability. The posterior probability is calculated based on Bayes' theorem, and when the calculated posterior probability has the maximum value, the divided matching area and the mismatching area are determined to be likely area divisions, and the matching area and the mismatching area are When it is determined that the region is likely to be divided, motion compensation is performed based on only the matching region from the first reference image, the second reference image, and the first motion vector, and an interpolation image is generated. , Painting It is no longer necessary to perform match determination every time, and it is possible to divide the matching area and the non-matching area using a more probable method, reducing noise such as sesame salt noise and smoothing the moving image by reducing block distortion. The effect that it can be displayed is produced.

  Exemplary embodiments of an interpolation image generation apparatus, an interpolation image generation method, and an interpolation image generation program according to the present invention will be described below in detail with reference to the accompanying drawings.

(Embodiment 1)
The interpolated image generating apparatus according to the first embodiment interpolates between a first reference frame p1 (first reference image) and a second reference frame p2 (second reference image) of a moving image. When generating an image of a frame, a block of a predetermined size in the reference frame p1 is divided into a matching area and a mismatching area, and whether or not the area division is a likely area division is determined using the concept of Bayes decision rule. It is formulated as a MAP (maximum a posteriori probability) estimation problem. Here, the coincidence determination problem is a problem in which a block pair determined by a motion vector obtained by motion estimation is divided into a coincidence region and a disagreement region.

  First, the Bayes determination rule used in the coincidence determination process performed by the interpolated image generation apparatus according to the present embodiment will be described.

The Bayesian decision rule is based on a stochastic model of statistical pattern recognition. Generally, c ₁ of m s, c _2, and · · · c _m, the value of n features extracted from the input pattern set (the feature _{vector), x = (x 1,} x 2 ,..., X _n ), the probability (prior probability) p (c _i ) (i = 1,..., M) that a pattern belonging to each class occurs, and under each class c _i , Assume that the occurrence probability p (x | c _i ) (i = 1,..., M) of the pattern x is known. At this time, the class c _k that maximizes the posterior probability under the condition that x has occurred, that is, the following expression p (c _k | x) = maxp (c _i | x)
If it is determined that the pattern x belongs to the class c _k that satisfies the above, the rule that it is known that the overall identification error probability is minimized is the Bayes determination rule.

  In the interpolated image generating apparatus of the present embodiment, this Bayes determination rule is used for matching determination. That is, how likely the matching area and the mismatching area are to the input image is formulated as a posterior probability.

  In the Bayes decision rule, the posterior probability is expressed as a product of likelihood and prior probability. The prior probability is formulated as indicating the probability of the spatial distribution of the coincidence region and the disagreement region. Likelihood assumes that the observed noise follows a Gaussian distribution and models the block difference in the matching region as a Gaussian distribution. In this way, by formulating in consideration of noise, it is possible to improve the robustness against noise and perform region matching determination.

Here, y _a and y _b are pixel values of the block B, and x is a label indicating a matching area and a mismatching area. That is, x = 1 is the label of the matching area, and x = 0 is the label of the mismatching area. The MAP estimation problem is a problem that maximizes the probability (posterior probability) p (x | y) of the coincidence determination x when the image y is given. From the base decision rule, the prior probability p (x) and the likelihood p (y | x) of y when the coincidence determination x is given have the relationship shown in Formula 1.

  The prior probability p (x) is a probability when the local distribution of 0 (mismatch region) and 1 (match region) is repeated alternately, that is, 0 (mismatch region) and 1 (match region) are dispersed. It is determined that the spatial distribution of 1 (coincidence region) is stochastically higher than the spatial distribution of 0 (noncoincidence region), and is formulated as shown in Equation 2.

  Here, β and c are cliques that are two points of adjacent pixels in the block, and Z is a normalization constant. FIG. 24 is an explanatory diagram illustrating an example of a clique that is two points of adjacent pixels in a block.

  The likelihood p (y | x) indicates that, when the other conditions are the same, the larger the value, the more likely the condition is under the condition x. In the present embodiment, the likelihood is represented by a pixel difference. Therefore, when the match determination x is given, it is assumed that the smaller the difference value in the match region, the larger the value of p (y | x), which is more likely.

If the noise observed in the image is white Gaussian noise with variance σ ² and average 0, the likelihood can be modeled as shown in Equation 3.

  In the present embodiment, the likelihood modeling is performed assuming that the noise follows a Gaussian distribution. However, the likelihood modeling may be performed based on a generalized Gaussian distribution or a Poisson distribution.

  When likelihood is modeled as shown in Equation 3, the MAP estimation problem can be formulated as shown in Equation 4.

ここで数４式の対数を取って、数５式のように定式化してもよい。

Here, the logarithm of Formula 4 may be taken and formulated as Formula 5.

  By calculating Equation 4 (or Equation 5) and calculating coincidence determination x, it is possible to perform coincidence determination under the Bayes decision rule. In other words, in the interpolated image generation apparatus of the present embodiment, the prior probability represented by Equation 2 is obtained, the likelihood represented by Equation 3 is obtained, and Equation 4 obtained by multiplying the prior probability and the likelihood is obtained by Equation 4. A match determination x that maximizes the posterior probability shown is obtained.

  Next, an interpolated image generating apparatus according to the present embodiment that generates an interpolated frame by performing matching determination using such a haze determination rule will be described.

  In the present embodiment, the input image signal (moving image signal) is a 60 Hz non-interlaced signal (progressive signal), and the temporal center between two reference frames adjacent to the 60 Hz non-interlaced signal. An example will be described in which an interpolation frame is generated at a position (interpolation frame plane), and the interpolation frame is interpolated between two reference frames to be converted into a 120 Hz non-interlace signal. Further, a temporally preceding reference frame is referred to as a reference frame p1, and a temporally subsequent reference frame is referred to as a reference frame p2. However, the present invention is not limited to the above image signal, and the present invention can be applied to image signals of various frame rates.

  FIG. 1 is a block diagram of a functional configuration of the interpolated image generation apparatus according to the first embodiment. As shown in FIG. 1, the interpolated image generation apparatus 100 according to the present embodiment includes a motion estimation unit 110, a motion compensation unit 120, and a frame memory 130.

  The motion estimation unit 110 divides the input two reference frames p1 and p2 into lattice blocks, divides the divided blocks into matching areas and mismatched areas, and obtains a motion vector in each area. .

  As shown in FIG. 1, the motion estimation unit 110 includes a motion search unit 111, a region division unit 112, a likelihood calculation unit 113, a prior probability calculation unit 114, a posterior probability calculation unit 115, and a determination unit 116. It has.

  The motion search unit 111 receives two reference frames p1 and p2 and performs a process of obtaining a first motion vector for each block of the uniform grid. The motion search unit 111 further performs processing for obtaining the second motion vector based only on the mismatch region when the determination unit 116 described later determines that the matching region and the mismatch region are likely region divisions. .

  The area dividing unit 112 performs a process of dividing the blocks in the reference frames p1 and p2 into a matching area and a mismatching area. The area dividing unit 112 performs a matching determination process on the divided matching area and the mismatching area together with the likelihood calculating section 113, the prior probability calculating section 114, the posterior probability calculating section 115, and the determining section 116.

  The likelihood calculating unit 113 is based on the Gaussian distribution of the difference between the pixel values in the matching region in the block of the reference frame p1 and the block on the reference frame p2 determined by the first motion vector obtained by the motion search unit 111. A process of calculating the likelihood p (y | x) according to Equation 3 is performed. This likelihood is represented by a pixel difference, and when the match determination x is given, the smaller the difference value in the match region, the greater the likelihood p (y | x) value, and the more likely it is Means.

  The prior probability calculation unit 114 performs a process of calculating the prior probability p (x) indicating the probability of the spatial distribution of the coincidence region and the disagreement region according to Equation 2. This prior probability p (x) is a low value in the case of a spatial distribution in which the mismatch region and the match region are dispersed, and the spatial distribution of the match region is higher than the spatial distribution of the mismatch region. Is set by the equation (3).

  The posterior probability calculation unit 115 multiplies the likelihood calculated by the likelihood calculation unit 113 by the prior probability calculated by the prior probability calculation unit 114, and indicates the likelihood of dividing the matched region and the unmatched region. A process of calculating (x | y) according to Equation 4 is performed.

  The determination unit 116 determines whether or not the posterior probability p (x | y) has the maximum value. When the posterior probability p (x | y) has the maximum value, the matching region and the mismatching region divided by the region dividing unit 112 are likely to be divided into regions. The coincidence determination process is performed to determine that

  The motion compensation unit 120 matches the reference frames p1 and p2 and the first motion vector obtained by the motion search unit 111 when the determination unit 116 determines that the matching region and the mismatching region are likely region divisions. Motion compensation is performed based on only the region, and processing for generating an interpolation frame is performed. The frame memory 130 is a storage device that temporarily stores an image of a reference frame.

  Next, an interpolation image generation process by the interpolation image generation apparatus 100 configured as described above will be described. FIG. 2 is a flowchart showing a procedure of the entire process of generating an interpolated image.

  First, the motion search unit 111 divides the interpolation frame into small blocks bl0 having a uniform lattice (step S201), and starts scanning the block (step S202). Then, motion estimation is performed to obtain a motion vector group mvg and a disagreement filter group afg as a matching region in the block (step 203). Here, the motion estimation process will be described later.

  Next, motion compensation processing is performed from the motion vector group mvg, the mismatch filter group afg, the reference frame p1, and the reference frame p2, and an interpolation frame is generated (step S204). Such motion compensation processing will be described later. The processes in steps S203 and S204 are repeated until all the small blocks bl0 are scanned. If all the small blocks bl0 are scanned, the process is completed (step S205).

  Next, the motion estimation process in step S202 will be described. FIG. 3 is a flowchart showing the procedure of the motion estimation process.

  In the motion estimation process, the block having the highest correlation is searched from the search areas of the reference frames p1 and p2, the first motion vector is obtained (motion search process), and the small blocks bl1 and bl2 that are the searched block pairs are searched. A match / mismatch is determined internally to obtain a mismatch filter af (match determination process), and a process of recursively searching for the second motion vector from only the mismatched pixels is performed.

  First, the motion search unit 111 initializes a variable it for iteration processing to 1, starts iteration processing, sets a digital filter having the same size as the small block bl0 as a mismatch filter, and sets the mismatch filter af [ [0] are all substituted with 1 (mismatched pixels) (step S301).

  FIG. 4 is an explanatory diagram showing the state of the mismatch filter af [0] in the first iterative process. As shown in FIG. 4, in step S301, all 1 (mismatched pixels) are set in the mismatch filter af [0]. Note that the mismatch filter af [α] is a mismatch filter at the number of iterations α (ite = α).

  The mismatch filter is a digital filter having the same size as the small block bl0 and having 0 or 1 as a value. FIG. 5 is an explanatory diagram showing the contents of the mismatch filter. As shown in FIG. 5, the mismatch filter is set to 0 for matching pixels and 1 for mismatching pixels.

  Next, a motion search process is performed by the motion search unit 111. FIG. 6 is an explanatory diagram showing an outline of the motion search process. As shown in FIG. 6, the motion search unit 111 geometrically centered on the small block bl0 on the interpolated frame using only a pixel having a mismatch filter [ite-1] value of 1, that is, a mismatch region. The small block bl1 on the symmetric reference frame p1 and the small block bl2 on the reference frame p2 are searched to search for a pair of small blocks having the highest degree of correlation to obtain a motion vector mv [ite] (step S302). .

  FIG. 7 is an explanatory diagram of motion vector estimation processing in the case of the first iteration (ite = 1). At the time of the first iteration, when calculating the degree of correlation, as shown in FIG. 7, the calculation is performed only for the pixels whose value of the mismatch filter af [0] is 1, but at the first iteration, the mismatch filter af [0 ], Since all the pixels in the block are set to 1 (mismatched pixels), the correlation calculation is performed for all the pixels in the block.

  FIG. 8 is an explanatory diagram of motion vector estimation processing in the second and subsequent iterations (ite> 1). In the second and subsequent iterations, during the correlation calculation, as shown in FIG. 8, the correlation calculation is performed only for the pixel whose value of the mismatch filter af [ite−1] is 1.

  Next, the motion search unit 111 obtains the small block bl1 on the reference frame p1 and the small block bl2 on the reference frame p2 determined by the motion vector mv [It] (step S303).

  Next, the area dividing unit 112 performs matching determination from the small block bl1 and the small block bl2, and obtains a mismatch filter af [ite] (step S304). The coincidence determination process will be described later.

  Next, the region dividing unit 112 obtains a logical product of the mismatch filter af [ite] and the mismatch filter af [ite-1], and sets the obtained logical product in the mismatch filter af [ite] (step S305). As a result, the value of the mismatch filter obtained in the match determination process is reflected in the mismatch filter af [ite].

  Then, the motion search unit 111 increments the variable it by 1 (step S306), repeatedly executes the motion search process and the match determination process from steps S302 to S306, and the variable it exceeds an arbitrary integer n. At this point (step S307), the repetition of the processing from step S302 to S306 is terminated. As a result, the motion vector group mvg (motion vector mv [i] (i = 1... N)) and the mismatch filter group afg (mismatch filter af [i] (i = 1... N)) are output. The

  Next, the matching determination process in step S304 will be described. FIG. 9 is a flowchart showing the procedure of the coincidence determination process.

  In the present embodiment, the matching is determined by solving the MAP estimation problem hierarchically. First, the area dividing unit 112 receives the small block bl1 and the small block bl2, generates the block layer bl_lay1 from the small block bl1, and generates the block layer bl_lay2 from the small block bl2 (step S901). Such hierarchy generation processing will be described later.

  Next, the region dividing unit 112 starts a loop that recursively descends the hierarchy with the loop variable as i, and sets 0 as the initial value of the loop variable i (step S902).

  Next, the area dividing unit 112 generates a mismatch filter af having the same size as the block layer bl_lay1 [i + 1], sets all the values of the generated mismatch filter af to 1 (mismatched pixels) (step S903), Is started (step S904). Here, the scanning index is (x, y).

  Next, a combination search loop for the search area is started (step S905). FIG. 10 is an explanatory diagram illustrating an example of a search area pattern indicating all combinations of search areas. In FIG. 10, the shaded area indicates the search area.

  Then, the area dividing unit 112 selects one arbitrary search area pattern that has not been selected from the search area patterns shown in FIG. 10 and generates a combination block (step S906). Then, the combination block is pasted at the position (2x, 2y) in the mismatch filter af to generate the mismatch filter af_tmp (step S907).

  Next, the prior probability calculation unit 114 calculates the prior probability p (x) using the mismatch filter af_tmp (step S908). Here, the calculation process of the prior probability p (x) will be described later.

  Then, the likelihood calculating unit 113 calculates the likelihood p (y | x) using the mismatch filter af_tmp, the block layer bl_lay1 [i + 1], and the block layer bl_lay2 [i + 1] (step S909). Here, the process of calculating the likelihood p (y | x) will be described later.

  Next, the posterior probability calculation unit 115 calculates the posterior probability p (x | y) from the product of the prior probability p (x) and the likelihood p (y | x) (step S910).

  Next, the determination unit 116 sets the mismatch filter af_tmp that gives the maximum value of the posterior probability p as the mismatch filter af_max (step S911). Specifically, the determination unit 116 determines the maximum value of the posterior probability p (x | y) obtained from the mismatch filter af_tmp generated according to the search area pattern selected in step S906 in the combination loop up to the previous time. Thus, if the posterior probability p (x | y) obtained from the mismatch filter af_tmp generated according to the search area pattern selected in the current combination loop is large and is large, The mismatch filter af_tmp giving the probability p (x | y) is set as the mismatch filter af_max.

  Then, it is determined whether or not all combinations of search areas have been searched, that is, whether or not all search area patterns shown in FIG. 10 have been selected (step S912). (S912: No), the processing from step S905 to S911 is repeatedly executed.

  On the other hand, if it is determined in step S912 that all combinations of search areas have been searched (step S912: Yes), the current mismatch filter af_max is set as the mismatch filter af (step S913).

  Then, the processes from step S904 to S913 are repeatedly executed until the entire block is scanned (step S914: No). When the entire block is scanned (step S914: Yes), the loop variable i is updated to i = i + 1. (Step S915). Then, the processing from step S902 to S915 is repeated until the loop variable i becomes greater than or equal to the number n of layers (step S916: No), and when the loop variable i becomes greater than or equal to the number n of layers (step S916: Yes), the processing is performed. finish. Thereby, the mismatch filter af which gives the maximum of the posterior probability p (x | y) is output.

  Next, the hierarchy generation process in step S901 will be described. FIG. 11 is a flowchart illustrating the procedure of the hierarchy generation process. Here, a process of generating a hierarchical block layer lbl_lay sampled in the spatial direction from the input small block lbl will be described. Such a hierarchy generation process is executed for each of the small block bl1 and the small block bl2 in step S901 in the coincidence determination process of FIG. 9, and the block layer of the small block bl1 and the block layer of the small block bl2 are generated. become. Here, a process of generating a block layer lbl_lay from the small block lbl by inputting the small block lbl will be generally described.

  First, the area dividing unit 112 calculates the number of samplings n in the spatial direction from the vertical size and horizontal size of the small block lbl (step S1101). For example, in the case of a 16 × 16 size block, if 1 / 2-scale sampling is repeatedly applied four times in both the vertical and horizontal directions (n = 4), a 1 × 1 minimum block is obtained. In the case of 32 × 16, when 1 / 2-scale sampling is applied four times in both the vertical and horizontal directions (n = 4), the minimum block is 2 × 1.

  Next, the area dividing unit 112 secures an n + 1 arrangement of the n-th block block lbl_lay (step S1102). This arrangement is used as an area for storing a small block of each layer. Then, the region dividing unit 112 stores the small block lbl in the block layer lbl_lay [n] (step S1103).

  Next, the region dividing unit 112 starts a hierarchical loop with the initial value of the loop variable i as n (step S1104). Then, the block layer lbl_lay [i] is sampled once in the spatial direction and stored in lbl_lay [i−1] (step S1105). Here, arbitrary sampling may be used for sampling in the spatial direction. For example, sampling with an average of four surrounding points as sampling points can be used. When this sampling is performed, the scale becomes 1/2.

  Next, the area dividing unit 112 updates the loop variable i to i−1 (step S1106). Then, it is determined whether or not the loop variable i is smaller than 0 (step S1107). If it is equal to or larger than 0 (step S1107: No), the processing from step S1104 to S1106 is repeatedly executed. On the other hand, when the loop variable i is smaller than 0 (step S1107: Yes), the hierarchy generation process is terminated. Thereby, the block layer lbl_lay is generated and output. As a result of the processing in step S901, the block layer bl_lay1 is generated from the small block bl1, and the block layer bl_lay2 is generated from the small block bl2.

  Next, the calculation process of the prior probability p (x) by the prior probability calculation unit 114 in step S908 of the coincidence determination process of FIG. 9 will be described. FIG. 12 is a flowchart illustrating a procedure of a prior probability calculation process performed by the prior probability calculation unit 114.

  Prior probability calculation section 114 first inputs mismatch filter af_tmp, initializes all variables a, n, and d to 0 (step S1201), and scans cliques that are two points of adjacent pixels in the block (step S1201). S1202).

  Next, it is checked whether or not the values of the mismatch filter af_tmp in the target clique are all 0 (matching pixels) (step S1203). If the values of the mismatch filter af_tmp in the target clique are all 0 (matching pixel) (step S1203: Yes), the variable a is incremented by 1 (step S1205).

  On the other hand, if any value of the mismatch filter af_tmp in the target clique is not 0 (match pixel) in step S1203 (step S1203: No), all the values of the mismatch filter af_tmp in the target clique are 1 (mismatch pixels). ) Is checked (step S1204). If the values of the mismatch filter af_tmp in the target clique are all 1 (mismatch pixels) (step S1204: Yes), the variable n is incremented by 1 (step S1206).

  On the other hand, if any value of the mismatch filter af_tmp in the target clique is not 1 (mismatch pixel) in step S1204 (step S1204: No), the variable d is increased by 1 (step S1207).

  Then, it is determined whether or not all cliques have been scanned (step S1208). If not all have been scanned (step S1208: No), the processing from step S1202 to S1207 is repeatedly executed. On the other hand, when all the cliques have been scanned (step S1208: Yes), sum represented by Equation 6 is calculated (step S1209).

  Next, a priori probability p (x) is calculated by Equation 7 using the sum calculated in step 1209 (step S1210).

  These Formula 6 and Formula 7 are calculation formulas for specifically obtaining the prior probabilities formulated by Formula 2. Thereby, the prior probability is calculated and output.

  Next, the likelihood p (y | x) calculation process by the likelihood calculation unit 113 in step S909 of the coincidence determination process in FIG. 9 will be described. FIG. 13 is a flowchart illustrating a procedure of likelihood calculation processing by the likelihood calculation unit 113.

  First, the likelihood calculating unit 113 inputs the blocks bl_lay1 and bl_lay2 and the mismatch filter af_tmp, and calculates a difference block db1 between the blocks bl_lay1 and bl_lay2 (step S1301). Then, the sum sum of (the square of the difference block dbl) / 2 is obtained only for the pixels (matching pixels) for which the mismatch filter af_tmp is 0 (step S1302). Then, the likelihood p (y | x) is calculated by Equation 8 (step S1303) and output.

  Next, the motion compensation process by the motion compensation unit 120 in step S204 of FIG. 2 will be described. FIG. 14 is a flowchart showing a procedure of motion compensation processing by the motion compensation unit 120.

  In the motion compensation process, the motion vector group mvg (motion vector mv [i] (i = 1... N)) and mismatch filter group afg (mismatch filter af [i] (i = 1. N)), motion compensation is performed from the reference frame p1 and the reference frame p2, and an interpolation frame is generated.

  First, the motion compensation unit 120 sets a variable “ite” for iterative processing to an initial value 1 (step S1401). Then, an image block pb1 on the reference frame p1 determined by the point-symmetrical position of the motion vector mv [ite] of the motion vector group mvg around the small block bl0 on the interpolation frame is obtained (step S1402), and the motion vector mv [ite] is obtained. ] Determines the image block pb2 on the reference frame p2 determined by (step S1403). Then, the motion compensation unit 120 obtains the position of the small block bl0 on the interpolation frame (step S1404).

  Then, the motion compensation unit 120 starts scanning the pixels in the block (step S1405), and obtains an average value of the pixel at the target pixel position of the image block pb1 and the pixel of the image block pb2 (step S1406).

  Next, the motion compensation unit 120 checks whether the value of the mismatch filter [ite] is 0, that is, whether it is a matching pixel (step S1407). If the value of the mismatch filter [ite] is 0, that is, if it is a matching pixel (step S1407: Yes), the average pixel is copied to the target pixel position on the interpolation frame (step S1408). On the other hand, if the value of the mismatch filter [ite] is 1 in step S1407, that is, if it is a mismatched pixel (step S1407: No), the average pixel is not copied.

  The processes in steps S1407 and S1408 are repeatedly executed until all the pixels in the block are scanned (step S1409). As a result, the motion compensation process is performed only on the matching region, and an interpolation frame is generated. On the other hand, in the case of the mismatch region, the average pixel interpolation frame is not copied to the small block bl0. However, after the motion compensation process is completed, the motion estimation process (FIG. 3) is performed recursively. 2 motion vectors).

  When the scanning of all the pixels in the block is completed (step S1409: Yes), the motion compensation unit 120 increments the variable it for the iterative process by 1 (step S1410), and the variable it is n (arbitrary integer). It is determined whether or not the threshold is exceeded (step S1411). Then, the processing from step S1402 to S1410 is repeatedly executed until the variable it exceeds n.

  The motion estimation process in step S203 in FIG. 2 and the motion compensation process described above are repeatedly executed until all the small blocks 0 divided in step S201 in FIG. 2 have been scanned. A frame will be generated.

  Next, the result of coincidence determination by the interpolated image generation apparatus 100 according to the present embodiment is compared with the result of coincidence determination by a conventional method. FIG. 15 is an explanatory diagram showing a state in which the matching area is changed in the block. As shown in FIG. 15, the coincidence area was changed while scanning along the x-axis and the y-axis, and the posterior probabilities at that time were tabulated. FIG. 16 is an explanatory diagram illustrating an example of two blocks of a noise-free image in which the left half region is a matching region. Therefore, the posterior probability distribution is ideally maximized when the left half is the matching region.

  First, in the image shown in FIG. 16, matching determination by the conventional method, that is, simply taking an absolute value difference between pixels having the same relative position in the block, and a matching area if the absolute value difference value is smaller than a threshold value (for example, 10). The coincidence determination is performed to generate an interpolation frame. FIG. 17 is an explanatory diagram showing an interpolated frame image generated using a conventional matching determination method. As shown in FIG. 17, in the case of an image having no noise at all, an interpolation image is normally generated even by the conventional matching determination method.

  FIG. 18 is an explanatory diagram showing the distribution of posterior probabilities obtained by matching determination by the interpolated image generation apparatus 100 according to the present embodiment for the two blocks shown in FIG. As shown in FIG. 18, the posterior probability distribution shows that the posterior probability p (x | y) is maximized when the left half is the coincidence region, and is operating accurately.

  FIG. 19 is an explanatory diagram illustrating an example of two blocks of an image having noise in which the left half region is a matching region. The image shown in FIG. 19 is subjected to matching determination by a conventional method to generate an interpolation frame. FIG. 20 is an explanatory diagram showing an image of an interpolation frame generated using a conventional matching determination method. In the conventional method, when the absolute value difference value between the pixels is smaller than a certain threshold value, the matching determination as the matching region is performed, so that it is not possible to deal with the noise component, and as shown in FIG. It can be seen that sesame salt noise is generated in the interpolation frame.

  FIG. 21 is an explanatory diagram showing distribution of posterior probabilities obtained by matching determination by the interpolated image generating apparatus 100 according to the present embodiment for the two blocks shown in FIG. As shown in FIG. 21, the posterior probability distribution has the maximum posterior probability p (x | y) when the left half is the coincidence region, and operates accurately even for images containing noise. I understand that.

  As described above, in the interpolated image generating apparatus according to the first embodiment, the coincidence determination problem is formulated as a MAP estimation problem using Bayesian decision theory, and it is not necessary to perform the coincidence determination for each pixel, and the matching is performed with a more probable method. Since the area and the non-matching area can be divided, noise such as sesame salt noise can be reduced and the robustness can be improved while the block distortion is reduced and the moving image can be displayed smoothly.

(Embodiment 2)
In the interpolated image generation apparatus 100 according to the first embodiment, first, the small blocks input in the match determination process are hierarchized by the hierarchy generation process (FIG. 11), and then the match determination and the posterior probability are calculated for each hierarchy block. However, the interpolated image generation apparatus according to the second embodiment performs hierarchization while performing matching determination processing.

  The functional configuration of the interpolated image generating apparatus according to the present embodiment is the same as that of the interpolated image apparatus of the first embodiment. Also, the overall process of interpolation image generation, the motion estimation process, and the motion compensation process are the same as the overall process (FIG. 2), the motion estimation process (FIG. 3), and the motion compensation process (FIG. 14) of the first embodiment, respectively. Done. In the interpolated image generation apparatus according to the present embodiment, the matching determination process in step 304 of the motion estimation process shown in FIG.

  FIG. 22 is a flowchart showing a procedure of matching determination processing according to the second embodiment. First, the region dividing unit 112 inputs the small block bl1 and the small block bl2, starts a loop that recursively descends the hierarchy, setting the loop variable as i, and sets 1 as the initial value of the loop variable. (Step S2201). Then, the area dividing unit 112 prepares a mismatch filter af having the same size as the small block bl1, and sets all 1 to the mismatch filter af (step S2202).

Next, the area dividing unit 112 starts scanning within the block, and sets the scanning index to (x, y) (step S2203). Here, the range of the scanning index (x, y) is (0, 0) <= (x, y) <(2 ^(i-1) , 2 ^(i-1) ). Then, the area dividing unit 112 starts a combination search loop (step S2204). Here, the combination of search areas is the same as the search area pattern shown in FIG.

Next, the region dividing unit 112 selects one search region pattern from FIG. 10 to generate a combination block (step S2205), and scale-converts this combination block to 2 ^(ni-1) times vertically and horizontally (step S2205). Step S2206). Then, the size is calculated by Equation 9 (step S2207).

  Next, the region dividing unit 112 pastes the combination block at the position (x * size, y * size) in the mismatch filter af to generate a mismatch filter af_tmp (step S2208).

  Next, the prior probability calculation unit 114 calculates the prior probability p (x) using the mismatch filter af_tmp (step S2209). Here, the calculation process of the prior probability p (x) is performed in the same manner as the prior probability calculation process (FIG. 12) of the first embodiment.

  The likelihood calculating unit 113 calculates likelihood p (y | x) using the mismatch filter af_tmp, the block layer bl_lay1 [i + 1], and the block layer bl_lay2 [i + 1] (step S2210). Here, the likelihood p (y | x) calculation process is performed in the same manner as the likelihood calculation process (FIG. 13) of the first embodiment.

  Next, the posterior probability calculation unit 115 calculates the posterior probability p (x | y) from the product of the prior probability p (x) and the likelihood p (y | x) (step S2211).

  Next, the determination unit 116 sets af_tmp that gives the maximum value of the posterior probability p as the mismatch filter af_max (step S2212). Specifically, as in the first embodiment, the determination unit 116 determines the posterior probability p obtained from the mismatch filter af_tmp generated according to the search area pattern selected in step S2205 in the previous combination loop. It is determined whether or not the posterior probability p (x | y) obtained from the mismatch filter af_tmp generated according to the search area pattern selected in the current combination loop is larger than the maximum value of (x | y). And the mismatch filter af_tmp giving the current prior probability p (x | y) as the mismatch filter af_max.

  Then, it is determined whether or not all combinations in the search area have been searched (step S2213). If all combinations have not been searched (step S2213: No), the processes from step S2204 to S2212 are repeatedly executed. .

  On the other hand, if it is determined in step S2213 that all combinations of search areas have been searched (step S2213: Yes), the determination unit 116 sets the mismatch filter af_max as the mismatch filter af (step S2214).

  Then, the processes from step S2203 to S2214 are repeatedly executed until the entire block is scanned (step S2215: No). When the entire block is scanned (step S2215: Yes), the loop variable i is updated to i = i + 1. (Step S2216). Then, the processing from step S2201 to S2216 is repeated until the loop variable i becomes greater than or equal to the number of layers n (step S2217: No), and when the loop variable i becomes greater than or equal to the number of layers n (step S2217: Yes), the processing is performed. finish. Thereby, the mismatch filter af is output.

  As described above, in the interpolated image generating apparatus according to the second embodiment, since the small blocks are hierarchized while performing the coincidence determination process, there is no need for a separate layer generation process in advance, and the coincidence determination process is quickly performed. It can be carried out.

  Further, in the interpolated image generation apparatus according to the second embodiment, the coincidence determination problem is formulated as a MAP estimation problem using Bayesian decision theory, so that it is not necessary to perform the coincidence determination for each pixel, and a more probable method is used for the coincidence region. The non-coincidence area can be divided, so that noise such as sesame salt noise can be reduced and the robustness can be improved while the block distortion is reduced and the moving image can be displayed smoothly.

(Embodiment 3)
In the interpolated image generation apparatus according to the first and second embodiments, the hierarchical mismatch filter is used in the match determination process. However, the interpolated image generation apparatus according to the third embodiment randomly applies the mismatch filter, and statistical physics. This is performed by using a metropolis algorithm in academia or the like to perform a match determination process.

  The functional configuration of the interpolated image generating apparatus according to the present embodiment is the same as that of the interpolated image apparatus of the first embodiment. Further, the entire process of generating an interpolated image is performed in the same manner as the entire process of the first embodiment (FIG. 2). In the interpolated image generation apparatus according to the present embodiment, the method for calculating the maximum value of the posterior probability p (x | y) is different from that in the first embodiment in the matching determination process in step 304 of the overall process shown in FIG. .

  In the metropolis algorithm (see A. Murat Tekalp, “Digital Video Processing”, Prentice Hall, 1995, p31) used in the coincidence determination processing in this embodiment, the posterior probabilities are obtained for a random mismatch filter. And whether to adopt the posterior probability according to the annealing temperature is determined. The annealing temperature is set according to the annealing schedule. If the annealing temperature is high, it may be adopted even if the posterior probability is low, so that the local maximum can be escaped. When the annealing temperature is lowered, the possibility of adopting a low posterior probability is reduced and gradually converges to the global maximum.

  FIG. 23 is a flowchart illustrating a procedure of matching determination processing according to the third embodiment. The area dividing unit 112 receives the small block bl1 and the small block bl2, and performs a match determination from the small block bl1 and the small block bl2 to obtain a mismatch filter af (step S2301). Then, the annealing temperature T is initialized with the initial temperature T0, the number of iterations M at a certain temperature is initialized with M0, and the mismatch filter af_max that gives the maximum value of the posterior probability is set as the mismatch filter af (step S2302).

  Next, the prior probability calculation unit 114 calculates the prior probability p (x) using the mismatch filter af (step S2303). Here, the calculation process of the prior probability p (x) is performed in the same manner as the prior probability calculation process (FIG. 12) of the first embodiment.

  Then, the likelihood calculating unit 113 calculates the likelihood p (y | x) using the mismatch filter af, the small block bl1, and the small block bl2 (step S2304). Here, the likelihood p (y | x) calculation process is performed in the same manner as the likelihood calculation process (FIG. 13) of the first embodiment.

  Next, the posterior probability calculation unit 115 calculates CurP which is the current posterior probability p (x | y) from the product of the prior probability p (x) and the likelihood p (y | x), and the maximum posterior probability p CurP is set to MaxP which is (x | y) (step S2305).

  Next, the region dividing unit 112 generates a mismatch filter af_new as a perturbation solution by adding a perturbation to the mismatch filter af (step S2306). Here, as the perturbation, for example, 4 pixels may be selected at random, and the value of the selected 4 pixels may be inverted.

  Then, the prior probability calculation unit 114 calculates the prior probability p (x) using the generated mismatch filter af_new (step S2307). Further, the likelihood calculating unit 113 calculates the likelihood p (y | x) using the mismatch filter af_new, the small block bl1, and the small block bl2 (step S2308). Then, the posterior probability calculation unit 115 calculates NewP, which is the posterior probability p (x | y) of the next state, from the product of the prior probability p (x) and the likelihood p (y | x) (step S2309). Then, the determination unit 116 obtains a difference ΔCost between NewP, which is the posterior probability p (x | y) of the next state, and CurP, which is the current posterior probability p (x | y) (step S2310), and the difference ΔCost is 0. It is determined whether it is larger (step S2311).

  When the difference ΔCost between the posterior probability NewP of the next state and the current posterior probability CurP is greater than 0 (step S2311: Yes), the determination unit 116 adds the posterior probability NewP of the next state to the current posterior probability CurP. And the mismatch filter af_new is set in the mismatch filter af (step S2312). Then, the determination unit 116 determines whether or not the posterior probability NewP of the next state is larger than the maximum posterior probability MaxP (step S2313).

  When the posterior probability NewP of the next state is larger than the maximum posterior probability MaxP (step S2313: Yes), the determination unit 116 sets the posterior probability NewP of the next state as the maximum posterior probability MaxP, The mismatch filter af_new is set to the mismatch filter af_max that gives the maximum value of the posterior probability (step S2314).

  On the other hand, in step S2313, when the posterior probability NewP of the next state is equal to or less than the maximum posterior probability MaxP (step S2313: No), the mismatch filter af_max that gives the maximum value of the maximum posterior probability MaxP and the posterior probability is changed. Not.

  In step S2311, when the difference ΔCost between the posterior probability NewP of the next state and the current posterior probability CurP is equal to or smaller than 0 (step S2311: No), the determination unit 116 determines that the current posterior probability CurP and the mismatch filter af are Without change, a uniform random number from 0 to 1 is generated, and it is checked whether this uniform random number is smaller than exp (ΔCost / T) (step S2315). When the uniform random number is smaller than exp (ΔCost / T) (step S2315: Yes), the determination unit 116 sets the posterior probability NewP of the next state to the current posterior probability CurP, and sets the mismatch filter af. A mismatch filter af_new is set (step S2316).

  On the other hand, if the uniform random number is not less than exp (ΔCost / T) in step S2315 (step S2315: No), the current posterior probability CurP and the mismatch filter af are not changed.

  Next, the iteration number M is decremented by 1, and it is checked whether or not M has become 0 (step S2317). If the number of iterations M is not 0 (step S2317: No), the processes from step S2306 to S2316 are repeatedly executed. On the other hand, when the number of iterations M becomes 0 in step S2317 (step S2317: Yes), the determination unit 116 adds the number of iterations M to the time Time, multiplies the temperature T by the decrease rate α, and further repeats. The number of times M is multiplied by the rate of increase β to update the time Time, the temperature T, and the number of iterations M (step S2318). Here, the decrease rate α of the temperature T is a value between 0.8 and 1.0. Further, the increase rate β of M is set to a value of 1.0 or more.

  Then, the determination unit 116 determines whether or not the time Time has exceeded the maximum time TimeMax (step S2319), and while not exceeding the maximum time TimeMax (step S2319: No), the processing from step S2306 to S2318 is performed. Repeatedly. On the other hand, if the time Time exceeds the maximum time TimeMax in step S2319 (step S2319: Yes), the process ends. By such processing, the mismatch filter af_max having the maximum value of the posterior probability p (x | y) is output.

  As described above, in the interpolated image generating apparatus according to the present embodiment, the mismatch filter is randomly given and the determination process is performed by using the metropolis algorithm in statistical physics or the like. Since the area and the non-matching area can be divided, noise such as sesame salt noise can be reduced and the robustness can be improved while the block distortion is reduced and the moving image can be displayed smoothly.

  The interpolated image generation apparatus according to the first to third embodiments includes a control device such as a CPU, a storage device such as a ROM (Read Only Memory) and a RAM, an external storage device such as an HDD and a CD drive device, a display device, and the like. The display device and an input device such as a keyboard and a mouse are provided, and a hardware configuration using a normal computer is employed.

  The interpolation image generation program executed by the interpolation image generation apparatus according to the first to third embodiments is a file in an installable format or an executable format, and is a CD-ROM, flexible disk (FD), CD-R, DVD ( The program is recorded on a computer-readable recording medium such as Digital Versatile Disk).

  The interpolation image generation program executed by the interpolation image generation apparatus according to the first to third embodiments is stored on a computer connected to a network such as the Internet and is provided by being downloaded via the network. You may do it. Further, the interpolation image generation program executed by the apparatus of this embodiment may be provided or distributed via a network such as the Internet.

  Further, the interpolation image generation program according to the first to third embodiments may be provided by being incorporated in advance in a ROM or the like.

  The interpolation image generation program executed by the interpolation image generation apparatus according to the first to third embodiments includes the above-described units (motion search unit 111, region division unit 112, likelihood calculation unit 113, prior probability calculation unit 114, posterior probability). The module configuration includes a calculation unit 114, a determination unit 116, and a motion compensation unit 120). As actual hardware, a CPU (processor) reads out and executes an interpolated image generation program from the storage medium, and executes the above-described units. Are loaded on the main storage device, and the motion search unit 111, the region division unit 112, the likelihood calculation unit 113, the prior probability calculation unit 114, the posterior probability calculation unit 114, the determination unit 116, and the motion compensation unit 120 are loaded on the main storage device. To be generated.

1 is a block diagram illustrating a functional configuration of an interpolated image generation apparatus according to a first embodiment; It is a flowchart which shows the procedure of the whole process of interpolation image generation. It is a flowchart which shows the procedure of a motion estimation process. It is explanatory drawing which shows the state of the mismatch filter af [0] of the first iteration process. It is explanatory drawing which shows the content of a mismatch filter. It is explanatory drawing which shows the outline | summary of a motion search process. It is explanatory drawing of the motion vector estimation process in the case of the first repetition (ite = 1). It is explanatory drawing of the motion vector estimation process in the case of the repetition after the 2nd time (ite> 1). It is a flowchart which shows the procedure of a coincidence determination process. It is explanatory drawing which shows the example of a search area | region pattern. It is a flowchart which shows the procedure of a hierarchy production | generation process. It is a flowchart which shows the procedure of the prior probability calculation process by the prior probability calculation part. It is a flowchart which shows the procedure of the likelihood calculation process by the likelihood calculation part 113. 5 is a flowchart illustrating a procedure of motion compensation processing by a motion compensation unit 120. It is explanatory drawing which shows the state which changed the coincidence area within the block. It is explanatory drawing which shows the example of two blocks of the image without a noise whose left half area | region becomes a coincidence area. It is explanatory drawing which shows the image of the interpolation frame produced | generated using the conventional matching determination method. It is explanatory drawing which shows distribution of the posterior probability calculated | required by the matching determination by the interpolation image generation apparatus 100 concerning this Embodiment with respect to two blocks shown in FIG. It is explanatory drawing which shows the example of two blocks of the image in which the noise whose left half area | region becomes a coincidence area exists. It is explanatory drawing which shows the image of the interpolation frame produced | generated using the conventional matching determination method. It is explanatory drawing which shows distribution of the posterior probability calculated | required by the matching determination by the interpolation image generation apparatus 100 concerning this Embodiment with respect to two blocks shown in FIG. 10 is a flowchart illustrating a procedure of matching determination processing according to the second embodiment. 12 is a flowchart illustrating a procedure of matching determination processing according to the third embodiment. It is explanatory drawing which shows the clique which is two points in a block.

Explanation of symbols

110 motion estimation unit 111 motion search unit 112 region dividing unit 113 likelihood calculation unit 114 prior probability calculation unit 115 posterior probability calculation unit 116 determination unit 120 motion compensation unit 130 frame memory

Claims (10)

An interpolation image generation device that generates an interpolation image to be interpolated between a first reference image and a second reference image of a moving image,
Motion search means for obtaining a first motion vector from the first reference image and the second reference image;
Area dividing means for dividing a block of a predetermined size in the first reference image into a matching area and a mismatching area;
The first block on the first reference image and the second block on the second reference image defined by the first motion vector obtained by the motion search means are located inside the matching area in the second block on the second reference image. Likelihood calculating means for calculating likelihood based on distribution of pixel information;
A prior probability calculating means for calculating a prior probability indicating a spatial distribution probability of the matching region and the mismatch region;
Based on the likelihood calculated by the likelihood calculating means and the prior probability calculated by the prior probability calculating means, a posterior probability indicating the likelihood of division of the matched region and the unmatched region is calculated based on Bayes' theorem. A posteriori probability calculating means;
Determining means for determining that the matching area and the mismatching area divided by the area dividing means are likely area divisions when the posterior probability calculated by the posterior probability calculating means has a maximum value; ,
When the determination unit determines that the matching area and the mismatching area are likely area divisions, the matching area is calculated from the first reference image, the second reference image, and the first motion vector. Motion compensation means for performing motion compensation based only on the generated image and generating the interpolated image;
An interpolated image generating apparatus comprising:   The motion search means obtains a second motion vector based only on the mismatch area when the determination means determines that the matching area and the mismatch area are likely area divisions. The interpolated image generation apparatus according to claim 1.   The likelihood calculating means is based on a distribution according to a Gaussian distribution of pixel differences of the matching region in the first block on the first reference image and the second block on the second reference image. The interpolation image generation device according to claim 1, wherein likelihood is calculated.   The likelihood calculating means has a distribution according to a generalized Gaussian distribution of pixel differences of the matching regions in the first block on the first reference image and the second block on the second reference image. 4. The interpolated image generation apparatus according to claim 3, wherein a likelihood based on the calculated likelihood is calculated.   The said prior probability calculation means calculates the said prior probability which shows a high probability, so that the spatial distribution of the said coincidence area | region and the said mismatch area | region is concentrated. The interpolated image generation device described in 1.   The said prior probability calculation means calculates the said prior probability which shows the probability that the spatial distribution of the said coincidence area is higher than the spatial distribution of the said disagreement area | region. Interpolated image generation device.   7. The area dividing unit divides a block of a predetermined size in the first reference image into a matching area and a mismatch area in a hierarchical structure. Interpolated image generation device.   The interpolation according to claim 1, wherein the region dividing unit divides a block of a predetermined size in the first reference image into a matching region and a mismatching region at random. Image generation device. An interpolation image generation method for generating an interpolation image to be interpolated between a first reference image and a second reference image of a moving image,
A motion search step for obtaining a first motion vector from the first reference image and the second reference image;
An area dividing step of dividing a block of a predetermined size in the first reference image into a matching area and a mismatching area;
The first block on the first reference image and the second block on the second reference image defined by the first motion vector obtained by the motion search step are located inside the matching area in the second block on the second reference image. A likelihood calculating step for calculating a likelihood based on the distribution of pixel information;
A prior probability calculating step of calculating a prior probability indicating a probability of spatial distribution of the matched region and the mismatched region;
Based on the likelihood calculated by the likelihood calculating step and the prior probability calculated by the prior probability calculating step, a posterior probability indicating the likelihood of dividing the matched region and the unmatched region is calculated based on Bayes' theorem. A posteriori probability calculation step;
A determination step of determining, when the posterior probability calculated by the posterior probability calculation step has a maximum value, the matching region and the non-matching region divided by the region dividing step are likely region divisions; ,
When it is determined by the determining step that the matching region and the mismatching region are likely region divisions, the first reference image, the second reference image, and the first motion vector A motion compensation step for performing motion compensation based only on the matching region and generating the interpolated image;
A method for generating an interpolated image, comprising: An interpolation image generation program for generating an interpolation image to be interpolated between a first reference image and a second reference image of a moving image,
A motion search procedure for obtaining a first motion vector from the first reference image and the second reference image;
An area dividing procedure for dividing a block of a predetermined size in the first reference image into a matching area and a mismatching area;
The first block on the first reference image and the second block on the second reference image defined by the first motion vector obtained by the motion search procedure are located inside the matching area in the second block on the second reference image. A likelihood calculation procedure for calculating a likelihood based on the distribution of pixel information;
A prior probability calculating procedure for calculating a prior probability indicating a probability of a spatial distribution of the matching region and the mismatch region;
Based on the likelihood calculated by the likelihood calculation procedure and the prior probability calculated by the prior probability calculation procedure, a posterior probability indicating the likelihood of division of the matched region and the unmatched region is calculated based on Bayes' theorem. A posteriori probability calculation procedure;
A determination procedure for determining, when the posterior probability calculated by the posterior probability calculation procedure has a maximum value, that the matching region and the mismatching region divided by the region division procedure are likely region divisions; ,
When it is determined by the determination procedure that the matching region and the mismatching region are likely region divisions, the first reference image, the second reference image, and the first motion vector A motion compensation procedure for performing motion compensation based only on the matching region and generating the interpolated image;
Interpolated image generation program for causing a computer to execute.

Images