JP2015177341A - Frame interpolation device and frame interpolation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a frame interpolation device or the like for generating an interpolation frame of a natural image.SOLUTION: The frame interpolation device includes: a motion vector interpolation part for allocating an interpolated motion vector which is calculated on the basis of a motion vector indicating a motion of an image between frames and a time position of an interpolation frame to be inserted between two frames, to the interpolation frame for each unit region; a motion compensated image generation part for generating a forward motion compensated image and a backward motion compensated image on the basis of the interpolated motion vector; and an interpolation frame generation part for generating the interpolation frame by mutually averaging corresponding regions in the forward motion compensated image and the backward motion compensated image with weights being different in a normal region to which one or a pair of interpolated motion vectors are allocated for each unit region, and an abnormal region formed from at least either a collision region or a perforated region.

Description

  Embodiments described herein relate generally to a frame interpolation apparatus and a frame interpolation method.

  A technique for smoothing an image by inserting an interpolation frame between frames is known. An apparatus for generating an interpolation frame usually generates an interpolation frame based on a motion vector obtained by searching for a video.

JP 2010-166386 A JP-A-9-214899 JP 2005-204066 A

  In many cases, a hidden surface area is generated in an image in which a moving object is captured. The “hidden area” is an area where the object and the background behind the object overlap, and the background behind the object is temporarily invisible. In the hidden surface area, image information such as the background behind the object is lost, and therefore, a correct motion vector is often not obtained. When a correct motion vector cannot be obtained, the interpolation frame generated by the device that generates the interpolation frame becomes an unnatural image.

  The problem to be solved by the present invention is to provide a frame interpolation apparatus and a frame interpolation method for generating an interpolation frame of a natural image.

  To achieve the above object, the frame interpolation device according to the embodiment is based on a motion vector indicating the motion of an image between two frames and a time position of an interpolation frame inserted between the two frames. A motion vector interpolator that calculates an interpolated motion vector indicating an image motion between the two frames and the interpolated frame, and assigns the calculated interpolated motion vector to the interpolated frame for each unit region; A forward motion compensated image generated based on image information of a frame on the forward direction side of the frame and the interpolated motion vector, image information of a frame on the reverse side of the two frames, and A backward motion compensated image generated based on the interpolated motion vector, a motion compensated image generating unit for generating, and one or a pair of the unit regions for each unit region At least one of a normal area to which an inter-motion vector is assigned, a collision area to which a plurality or a plurality of pairs of interpolation motion vectors are assigned for each unit area, and a perforated area to which no interpolation motion vector is assigned An interpolation frame generation unit that generates the interpolation frame by averaging the corresponding areas of the forward motion compensation image and the backward motion compensation image with different weights. .

It is a block diagram of the frame interpolation apparatus of an embodiment. It is a flowchart which shows the motion search process of embodiment. It is a figure which shows a mode that a reference | standard frame is divided | segmented into a some block. It is a figure which shows a mode that the similar part of a search object block is searched from a reference frame. It is a figure which shows an example of an input frame (a reference frame and a reference | standard frame). FIG. 6 is a partially enlarged view of the input frame shown in FIG. 5. It is a figure which shows a mode that the motion vector was allocated to the reference | standard frame. It is a functional block diagram which shows the function of the motion compensation part with which a frame interpolation apparatus is provided. It is a figure which shows a mode that three interpolation frames were inserted between two input frames. It is a figure which shows the relationship between a motion vector and an interpolation motion vector. It is a flowchart which shows the motion compensation process of embodiment. It is a flowchart which shows the motion vector interpolation process of embodiment. It is a figure which shows a mode that an interpolation motion vector is calculated based on the motion vector allocated to the reference | standard frame. It is a figure which shows a perforated area | region and a collision area | region. It is a figure which shows a mode that the interpolation motion vector was allocated to the interpolation frame. It is a flowchart which shows the motion compensation image generation process of embodiment. It is a figure which shows a mode that a reverse direction motion compensation image is produced | generated. It is a figure which shows an example of a reverse direction motion compensation image. It is a figure which shows a mode that a forward direction motion compensation image is produced | generated. It is a figure which shows an example of a forward direction motion compensation image. It is a flowchart which shows the interpolation frame production | generation process of embodiment. It is a figure which shows an example of the interpolation frame produced | generated when the weighting coefficient used for the average of a non-normal area | region is not shifted. It is a figure which shows an example of a neighboring pixel. It is a figure which shows the relationship between a certainty factor and a weighting coefficient. It is a figure which shows an example of the interpolation frame produced | generated when the weighting coefficient used for the average of a non-normal area | region is shifted. It is a figure which shows the relationship between the ratio of a neighboring pixel, and reliability.

  Hereinafter, embodiments for carrying out the invention will be described with reference to the drawings. In the drawings, the same or equivalent parts are denoted by the same reference numerals.

  FIG. 1 is a block diagram of a frame interpolation device 100 of the present embodiment. The frame interpolation apparatus 100 generates a frame (hereinafter referred to as “interpolation frame”) to be inserted between a plurality of frames (hereinafter referred to as “input frame”) constituting the video. The frame interpolation apparatus 100 includes a control unit 110, a storage unit 120, an input unit 130, a motion search unit 140, a motion compensation unit 150, and an output unit 160.

  The control unit 110 includes a processing device such as a processor. The control unit 110 controls each unit of the frame interpolation device 100 by operating according to a program stored in a ROM (Read Only Memory) or a RAM (Random Access Memory) (not shown).

  The storage unit 120 includes a storage device capable of reading and writing data such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a semiconductor memory, and a hard disk. The storage unit 120 includes various storage areas such as a frame memory area 121, a motion vector storage area 122, and an interpolation frame storage area 123. The frame memory area 121 stores a video signal acquired by the input unit 130. In addition, the motion vector storage area 122 stores search results generated by the motion search unit 140. The interpolation frame storage area 123 stores the interpolation frame generated by the motion compensation unit 150.

  The input unit 130 includes an input interface such as a serial interface or a parallel interface. The input unit 130 stores the input video signal in the frame memory area 121. It is assumed that the video signal is composed of input frames at intervals of 0.1 seconds, for example. In the following description, in order to facilitate understanding, frame numbers are assigned to input frames in order of time, such as F [0], F [1], F [2],. And

  The motion search unit 140 performs a motion search of the input frame. As a search method, for example, a block matching method or a gradient method can be used. The motion search unit 140 stores search results such as motion vectors and evaluation values in the motion vector storage area 122.

  The motion compensation unit 150 operates in accordance with a program stored in a ROM or RAM (not shown), thereby realizing various operations including “motion compensation processing”. In the motion compensation process, an interpolation frame is generated based on the search result of the motion search unit 140. The motion compensation unit 150 stores the generated interpolation frame in the interpolation frame storage area 123. Note that the motion compensation unit 150 may realize its function with a single processor, or with the cooperation of a plurality of processors.

  The output unit 160 includes an output interface such as a serial interface or a parallel interface. The output unit 160 outputs the interpolation frame stored in the interpolation frame storage area 123.

Next, the operation of the frame interpolation apparatus 100 will be described.
The operation of the frame interpolation apparatus 100 is divided into a “motion search process” executed by the motion search unit 140 and a “motion compensation process” executed by the motion compensation unit 150. First, the motion search process will be described.

  The motion search unit 140 starts processing when the control unit 110 is instructed to start motion search processing. The motion search unit 140 searches for image motion between two input frames. In the following description, it is assumed that the frame numbers of two input frames to be searched are specified from the control unit 110 at the same time as the motion search processing start command. At this time, the frame numbers specified by the control unit 110 are F [n−1] and F [n].

  FIG. 2 is a flowchart showing the operation of the motion search unit 140. The motion search unit 140 acquires two designated input frames from the frame memory area 121 (S101). In the following description, the frame F [n] is referred to as a reference frame, and the frame F [n−1] is referred to as a reference frame.

  The motion search unit 140 divides the reference frame F [n] into a plurality of blocks (S102). The block size is arbitrary. FIG. 3 shows an example in which the reference frame F [n] is divided into block sizes of 8 × 8 pixels.

  The motion search unit 140 selects one block that has not been searched yet from the reference frame F [n] as a search target block (S103).

  Subsequently, the motion search unit 140 searches for a portion similar to the search target block from the reference frame F [n−1] (S104). The search method may be a known search method such as a block matching method or a gradient method, or may be a search method that has been independently improved by a device manufacturer. After the search is completed, the motion search unit 140 acquires evaluation values of the search target block and the similar part. The evaluation value indicates the degree of matching between the search target block and the similar part. Note that the search range of the motion search unit 140 does not necessarily have to be the entire reference frame F [n−1]. The search range may be a certain range of the reference frame F [n−1], for example, a preset range centered on coordinates corresponding to the search target block. FIG. 4 shows an example in which the search range is 64 × 64 pixels.

  Here, the operation of step S104 will be described using a specific example. FIG. 5 is an image of x × y pixels with wavy lines drawn in the background. In FIG. 5, in addition to the wavy lines in the background, a letter “T” that moves from the right to the left in the drawing in front of the wavy lines is drawn. The letter “T” moves at a rate of 8 pixels per 0.1 second. FIG. 6 is an enlarged view of broken line portions indicated by blocks a and b in FIG. 5 in order to make the operation of S104 easy to see. The black portion at the center is the vertical bar portion of the letter “T”. In this example, the width of the vertical bar is 10 pixels.

  First, attention is focused on the block a of the reference frame F [n] shown in FIG. Part of the wavy line that is the background is drawn in the block a. The image of the block a completely matches the image of the same coordinate in the reference frame F [n−1], that is, the image of the block e. Therefore, the motion search unit 140 determines a similar part of the block a as an image of the block e.

  Next, attention is focused on block b of reference frame F [n]. A part of the letter “T” moving from right to left is drawn in the block b. The image of the block b completely coincides with the image of the reference frame F [n−1] in the portion (block f) moved about 8 pixels in the plus direction of the X axis. Therefore, the motion search unit 140 determines a similar part of the block b as an image of the block f.

  Subsequently, attention is focused on the block c of the reference frame F [n]. In the block c, a background partially hidden by a letter “T” is drawn. The background portion of the block c is the hidden surface of the letter “T” in the reference frame F [n−1]. Therefore, the image information of the background portion of the block c does not exist in the reference frame F [n−1]. However, the block c partially matches the image of the block g of the reference frame F [n−1]. Therefore, the motion search unit 140 determines a similar part of the block c as an image of the block g.

  Finally, attention is focused on the block d of the reference frame F [n]. Part of the wavy line that is the background is drawn in block d. In the image of the block g of the reference frame F [n−1] at the same coordinates as the block d, a part of the background is hidden by the letter “T”, and the image of the block d completely matches. Absent. However, most of the images of the block d and the block g are the same (background portion other than the letter “T”). Therefore, the motion search unit 140 determines a similar part of the block d as an image of the block g. In the example of FIG. 6, the background is only a simple wavy line, but in the case of a background with a more complicated pattern, the evaluation value regarding the relationship between the block d and the block g is like the relationship between the block c and the block g. In addition, it is higher than the case where most of the images do not match due to the hidden background image.

  In the example of FIG. 6, the image boundary of the similar portion of the reference frame F [n−1] (the image boundary of the block e to the block g) matches the block boundary of the reference frame F [n]. The image boundary of the part does not necessarily coincide with the block boundary.

  The motion search unit 140 generates a motion vector of the search target block based on the search result of S104 (S105). FIG. 7 represents the image of FIG. 6 in a one-dimensional format. Specifically, FIG. 7 is a diagram in which the base frame F [n] and the reference frame F [n−1] illustrated in FIG. 6 are cut along the A-A ′ line and the B-B ′ line, respectively. The thick line portion in the drawing is a “T” character and a wavy line portion. As shown in FIG. 7, a motion vector toward the block e is generated for the block a, and a motion vector toward the block f is generated for the block b. For blocks c and d, a motion vector directed to block g is generated.

  The motion vector expression format is not limited to a specific expression format, and various expression formats can be used. For example, the motion vector may be expressed in a coordinate format. In the example of FIG. 7, since the similar parts of the blocks a and d are at the same coordinate position in the reference frame F [n−1], the motion vector is expressed as (0, 0). Further, since the similar part of the blocks b and c is at a position moved by about 8 pixels in the positive direction of the X axis, the motion vector is expressed as (+8, 0). The motion search unit 140 stores the generated motion vector in the RAM together with the evaluation value calculated in S104.

  Subsequently, the motion search unit 140 determines whether or not the search for all the blocks of the reference frame F [n] has been completed (S106). When the search for all the blocks is not completed (S106: No), the motion search unit 140 repeats the processes from S103 to S106 until the search for all the blocks is completed. When the search for all the blocks is completed (S106: Yes), the motion search unit 140 proceeds to S107.

  The motion search unit 140 stores the motion vectors of all the blocks in the motion vector storage area 122 in association with the frame number of the reference frame F [n] and the evaluation value (S107), and ends the process.

  Next, the motion compensation process executed by the motion compensation unit 150 will be described. FIG. 8 is a block diagram of the motion compensation unit 150. The motion compensation unit 150 functions as a motion vector interpolation unit 151, a motion compensated image generation unit 152, and an interpolation frame generation unit 153 by executing motion compensation processing. In the following description, in order to facilitate understanding, it is assumed that the frame interval of the video into which the frame interpolation device 100 inserts the interpolation frame is 0.1 seconds by slow reproduction, for example. Further, for example, as shown in FIG. 9, the frame interpolation apparatus 100 inserts three interpolation frames at equal intervals (that is, at intervals of 0.025 seconds) between two input frames.

  The motion compensation unit 150 starts processing when the control unit 110 instructs to start motion compensation processing. The motion compensation unit 150 generates an interpolation frame to be inserted between the two input frames based on the motion vector generated by the motion search process. In the following description, two input frames F [n] and F [n−1] to be inserted into the interpolation frame and the interpolation frame to be inserted from the control unit 110 simultaneously with the motion search process start command. Let I [n] [m] be specified. m is an integer of 1 or more. In the case of FIG. 9, m = 1-3. Furthermore, it is assumed that the insertion position of the interpolation frame I [n] [m] is designated from the control unit 110. The insertion position is specified as a relative time position T with respect to two input frames, for example. FIG. 91 is a diagram illustrating a relationship between a motion vector and an interpolation motion vector. The traveling direction of time is “forward direction”, and the opposite direction is “reverse direction”. Hereinafter, the interpolation frame I [n] [1] will be described as an example. As shown in FIG. 10, the insertion position of the interpolation frame I [n] [1] is assumed to be a position advanced by 0.025 seconds in the forward direction from the input frame F [n−1]. In this case, the time position T is 0.25 (= 0.025 seconds / 0.1 seconds).

  FIG. 11 is a flowchart for explaining the operation of the motion compensation unit 150. First, the motion compensation unit 150 performs a motion vector interpolation process (S210). The motion vector interpolation process is executed by the motion vector interpolation unit 151. The motion vector interpolation unit 151 assigns an interpolation motion vector to the interpolation frame I [n] [1] based on the motion vector generated by the motion search process. The interpolated motion vector is a motion vector indicating the motion of the image between the two input frames F [n], F [n−1] and the interpolated frame I [n] [1].

  FIG. 12 is a flowchart for explaining the operation of the motion vector interpolation unit 151. The motion vector interpolation unit 151 acquires a motion vector between the input frames F [n] and F [n−1] from the motion vector storage area 122. Then, an interpolation motion vector is calculated for each block based on the acquired motion vector and the time position T of the interpolation frame I [n] [1] (S211). The interpolated motion vector is composed of a backward motion vector MVa and a forward motion vector MVb. The backward motion vector MVa is a motion vector between the input frame F [n-1] and the interpolation frame I [n] [1]. The forward motion vector MVb is a motion vector between the input frame F [n] and the interpolation frame I [n] [1].

Specifically, the motion vector interpolation unit 151 calculates a backward motion vector MVa and a forward motion vector MVb based on the following equations (1) and (2). In the following equation, MV is a motion vector between the input frames F [n] and F [n−1], and T is a relative value for the two input frames of the interpolation frame I [n] [1]. Time position.
MVa = MV × T (1)
MVb = −MV × (1-T) (2)

  Here, the backward motion vector MVa and the forward motion vector MVb will be described using a specific example. FIG. 13 is a diagram illustrating how an interpolation motion vector is calculated based on a motion vector between input frames F [n] and F [n−1]. As shown in FIG. 7, when the motion vector MV of the block b is (+8, 0), the motion vector interpolation unit 151 calculates the backward motion vector MVa (+2, 0) based on the equation (1). To do. In addition, the motion vector interpolation unit 151 calculates a forward motion vector MVb (−6, 0) based on Expression (2). Since the motion vector MV of the block c is (+8, 0), the motion vector interpolation unit 151 calculates the backward motion vector MVa (+2, 0) and the forward motion vector MVb (−6, 0). . Since the motion vectors MV of the blocks a and d are both (0, 0), the motion vector interpolation unit 151 calculates both the backward motion vector MVa and the forward motion vector MVb as (0, 0).

  The motion vector interpolation unit 151 assigns the interpolation motion vector calculated in S211 to the interpolation frame I [n] [1] (S212). At this time, the motion vector interpolation unit 151 assigns an interpolation motion vector to the interpolation frame I [n] [1] for each unit region. The unit region may be composed of a plurality of pixels or may be composed of one pixel. In the following description, the unit area is assumed to be one pixel.

  FIG. 14 is a diagram illustrating a state in which the interpolation motion vector is assigned to the interpolation frame I [n] [1] for each pixel. In the interpolation frame I [n] [1], a pixel to which a plurality of pairs of interpolation motion vectors are assigned (a pixel in the Q region) and a pixel to which no interpolation motion vector is assigned (a pixel in the P region) ). In the following description, a pixel to which a plurality of pairs of interpolation motion vectors are assigned is referred to as a “collision pixel”, and a pixel to which no interpolation motion vector is assigned is referred to as a “hole pixel”. An area composed of collision pixels is called a “collision area”, and an area composed of perforated pixels is called a “perforated area”. Furthermore, the collision area and the perforated area are collectively referred to as “non-normal area”. The non-normal area may be composed of both the collision area and the perforated area, or may be composed of either one of the collision area or the perforated area.

  The motion vector interpolation unit 151 selects one pair of interpolated motion vectors from among a plurality of pairs of interpolated motion vectors assigned to each collision pixel (S213). At this time, the motion vector interpolation unit 151 selects an interpolation motion vector based on the evaluation value calculated by the motion search unit 140. Specifically, the motion vector interpolation unit 151 selects an interpolation motion vector generated based on the motion vector MV having the largest evaluation value as the interpolation motion vector of the collision pixel.

  In the case of FIG. 14, interpolation motion vectors of “interpolation motion vector toward block c and block g” and “interpolation motion vector toward block d and block g” are assigned to each pixel in the collision region Q. In the example shown in FIG. 6, the evaluation value of the motion vector from the block d to the block g is larger than the evaluation value of the motion vector from the block c to the block g. Therefore, the motion vector interpolation unit 151 selects “interpolated motion vectors toward the blocks d and g” as the interpolation motion vectors of the pixels in the collision area Q.

  The motion vector interpolation unit 151 assigns a pair of interpolation motion vectors to each holed pixel (S214). At this time, the motion vector interpolation unit 151 directly uses the interpolated motion vector assigned to the plurality of pixels in the preset range around the perforated pixel as an interpolated motion vector of the perforated pixel. For example, the motion vector interpolation unit 151 acquires an interpolation motion vector assigned to each of a plurality of pixels in a range of 63 × 63 pixels centered on a perforated pixel. At this time, when an interpolation motion vector is not assigned to a pixel within the range, the pixel is ignored and an interpolation motion vector is acquired only from the pixel to which the interpolation motion vector is assigned. Then, the motion vector interpolation unit 151 extracts an interpolation motion vector having the same value from the plurality of pairs of acquired interpolation motion vectors, and sets the extracted interpolation motion vector as an interpolation motion vector of a perforated pixel.

  In the case of FIG. 14, no interpolated motion vector is assigned to the pixels in the perforated region P. In general, the image has the largest number of background images. Therefore, also in the case of FIG. 14, it is considered that the number of background pixels is the largest around the pixels in the perforated region P. In this case, since the background portion is not moving, there are the largest number of pixels of the motion vector (0, 0) indicating no motion around the pixels in the perforated region P. From the motion vector (0, 0), a forward interpolation motion vector MVb (0, 0) and a backward interpolation motion vector MVa (0, 0) are generated as interpolation motion vectors. The motion vector interpolation unit 151 assigns the forward interpolation motion vector MVb (0, 0) and the backward interpolation motion vector MVa (0, 0) to each pixel in the perforated region P. FIG. 15 is a diagram illustrating a state in which an interpolation motion vector is assigned to an interpolation frame by the motion vector interpolation process.

  When the interpolation processing of the interpolation motion vector in S210 is completed, the motion compensation unit 150 executes a motion compensated image generation process (S220). The motion vector interpolation process is executed by the motion compensated image generation unit 152. The motion compensated image generation unit 152 generates a motion compensated image based on the interpolated motion vector generated by the motion vector interpolation process. The “motion compensated image” is an image used in the interpolation frame generation process, and includes a forward direction motion compensated image and a backward direction motion compensated image.

  FIG. 16 is a flowchart for explaining the operation of the motion compensated image generation unit 152. The motion compensated image generation unit 152 performs the reverse direction based on the image information of the input frame F [n−1] on the reverse direction side when viewed from the interpolation frame I [n] [1] and the reverse interpolation motion vector MVa. A motion compensated image is generated (S221). The motion compensated image generation unit 152 generates a backward motion compensated image by pasting the pixel ahead of the arrow of the backward interpolation motion vector MVa to the original coordinates of the arrow. FIG. 17 shows how the backward motion compensation image is generated. In this case, the generated backward motion compensated image is, for example, an image as shown in FIG. It can be seen that the width of the vertical bar of “T”, which was 10 pixels, widens toward the collision area Q.

  The motion compensation image generation unit 152 performs forward motion compensation based on the image information of the input frame F [n] on the forward direction side when viewed from the interpolation frame I [n] [1] and the forward interpolation motion vector MVb. An image is generated (S222). The motion compensated image generation unit 152 generates a forward motion compensated image by pasting the pixel ahead of the arrow of the forward interpolation motion vector MVb to the original coordinates of the arrow. FIG. 19 shows how a forward motion compensation image is generated. In this case, the generated forward motion compensated image is, for example, an image as shown in FIG. It can be seen that the width of the vertical bar of “T”, which was 10 pixels, widens toward the perforated region P.

  When the generation of the motion compensated image in S220 is completed, the motion compensation unit 150 executes an interpolation frame generation process (S230). The interpolation frame generation process is executed by the interpolation frame generation unit 153 shown in FIG. The interpolation frame generation unit 153 generates an interpolation frame I [n] [1] to be inserted between the two input frames by averaging the two motion compensation images generated by the motion compensation image generation processing. .

  FIG. 21 is a flowchart for explaining the operation of the interpolation frame generation unit 153. The interpolation frame generation unit 153 first calculates a reference weight coefficient Wr (S231). The reference weighting factor Wr is a weighting factor used when averaging the normal areas in the two motion compensation images. The “normal area” is composed of normal pixels to which only one pair of interpolation motion vectors is assigned. The reference weight coefficient Wr is a numerical value between 0 and 1. When the weight of the backward motion compensated image is increased, the reference weight coefficient Wr is a value close to 0, and when the weight of the forward motion compensated image is increased, the reference weight coefficient Wr is a value close to 1. The interpolation frame generation unit 153 calculates the reference weight coefficient Wr based on the insertion position of the interpolation frame I [n] [1]. Specifically, the interpolation frame generation unit 153 calculates the reference weight coefficient Wr based on the time position T of the interpolation frame I [n] [1]. In the present embodiment, since the time position T is a numerical value between 0 and 1, the interpolation frame generation unit 153 directly calculates the time position T as the reference weight coefficient Wr.

  FIG. 22 is an image obtained by averaging the backward motion compensated image shown in FIG. 18 and the forward motion compensated image shown in FIG. 20 using the reference weight coefficient Wr. When all regions are averaged using the reference weighting coefficient Wr, an afterimage that should not exist may appear darkly in the non-normal region (the perforated region P and the collision region Q). Therefore, in S232 to S237, the interpolation frame generation unit 153 averages the images using different weighting factors in the normal region and the non-normal region.

  First, the interpolation frame generation unit 153 selects one pixel that has not yet been assigned a pixel value from among a plurality of pixels constituting the interpolation frame I [n] [1] (S232).

  The interpolation frame generation unit 153 determines whether or not the pixel selected in S232 (hereinafter referred to as “selected pixel”) is a normal pixel (S233). When the selected pixel is a normal pixel (S233: Yes), the interpolation frame generation unit 153 proceeds to S236. If the selected pixel is not a normal pixel (S233: No), that is, if the selected pixel is a collision pixel or a holed pixel, the interpolation frame generation unit 153 proceeds to S234.

  When the selected pixel is not a normal pixel (S233: No), the interpolation frame generation unit 153 calculates a certainty factor indicating a high possibility that the selected pixel is a hidden surface area (S234). In the hidden surface area, the object and the background or object behind the object overlap, and the background or object behind the object is temporarily invisible. In the example of FIG. 6, the “T” character portion of the base frame F [n] and the reference frame F [n−1] is the hidden surface area.

Specifically, the interpolation frame generation unit 153 calculates the certainty factor Ap or the certainty factor Aq based on the following formulas (3) and (4). The certainty factor Ap is the certainty factor when the selected pixel is a perforated pixel, and the certainty factor Aq is the certainty factor when the selected pixel is a collision pixel.
Ap = Rp (3)
Aq = Rq (4)

  At this time, Rp is the ratio of perforated pixels in the neighboring pixels of the selected pixel, and Rq is the ratio of collision pixels in the neighboring pixels of the selected pixel. The neighboring pixels are located in a preset range that is determined based on the selected pixel. FIG. 23 is a diagram illustrating an example of neighboring pixels. The neighboring pixels are 24 pixels in the square range of 5 × 5 pixels, for example, with the selected pixel as the center. In the case of FIG. 23, 21 pixels out of 24 pixels around the selected pixel 1 are perforated pixels, so Rp is calculated as 0.875 (= 21/24). In addition, since 18 of the 24 pixels around the selected pixel 2 are collision pixels, Rq is calculated as 0.75 (= 18/24).

  Subsequently, the interpolation frame generation unit 153 calculates a correction weighting coefficient based on the calculated certainty factor Ap or certainty factor Aq (S235). The correction weighting coefficient is used when averaging the non-normal regions in the two motion compensation images. The interpolation frame generation unit 153 calculates a correction weighting factor based on the reference weighting factor Wr. Specifically, the interpolated frame generation unit 153 shifts the weight to either one of the forward motion compensation image and the backward motion compensation image based on the value of the reference weight coefficient Wr, thereby obtaining the correction weight coefficient. calculate.

  Generally, the perforated area P is considered as an area hidden by an object whose background moves. Therefore, when generating an image of the perforated region P, it is considered that a more natural image can be generated by generating an image based on the reverse image F [n−1]. On the other hand, the collision area Q is considered to be an area where the background hidden by the moving object appears. Therefore, when an image of the collision area Q is generated, it is considered that a more natural image can be generated by generating an image based on the forward image F [n]. Therefore, when the selected pixel is a perforated pixel, the interpolation frame generation unit 153 shifts the weight toward the backward motion compensated image with the reference weight coefficient Wr as a base point, while the selected pixel is a collision pixel. If it is, the weight is shifted toward the forward motion compensated image with the reference weight coefficient Wr as a base point.

In this case, the interpolation frame generation unit 153 linearly changes the value toward 0 or 1 with the reference weight coefficient Wr as a base point in order to smooth the change in the afterimage that appears. FIG. 24 is a diagram illustrating a relationship between the certainty factor and the weighting factor. More specifically, the interpolation frame generation unit 153 calculates the correction weighting coefficient Wp or the correction weighting coefficient Wq based on the following formulas (5) and (6). Wp is a correction weighting coefficient used when averaging the perforated areas, and Wq is a correction weighting coefficient used when averaging the collision areas.
Wp = Wr × (1-Ap) (5)
Wq = Wr × (1-Aq) + Aq (6)

The interpolation frame generation unit 153 calculates the pixel value V of the selected pixel of the interpolation frame I [n] [1] by performing weighted averaging of pixels at the same coordinates in the backward motion compensation image and the forward motion compensation image. (S236). Specifically, the interpolation frame generation unit 153 calculates the pixel value V of the selected pixel of the interpolation frame I [n] [1] based on the following formulas (7) to (9). Expression (7) is used when the selected pixel is a holed pixel, and Expression (8) is used when the selected pixel is a collision pixel. Equation (9) is used when the selected pixel is a normal pixel.
V = Va × (1−Wp) + Vb × Wp (7)
V = Va × (1−Wq) + Vb × Wq (8)
V = Va * (1-Wr) + Vb * Wr (9)

  Note that Va is the pixel value of the selected pixel in the backward motion compensated image, and Vb is the pixel value of the selected pixel of the forward motion compensated image. When the calculation of the pixel value V is completed, the interpolation frame generation unit 153 assigns the pixel value V to the selected pixel of the interpolation frame.

  Subsequently, the interpolation frame generation unit 153 determines whether all the pixels are averaged (S237). When all the pixels are not averaged (S237: No), the interpolation frame generation unit 153 returns to S232, and repeats S232 to S237 until all the pixels are averaged. When all the pixels are averaged (S237: Yes), the interpolation frame generation unit 153 proceeds to S238.

  When all the pixels are averaged (S237: Yes), the interpolation frame generation unit 153 stores the interpolation frame I [n] [1] to which the pixel value is assigned in the interpolation frame storage area 123 (S238). In the example shown in FIGS. 18 and 20, the image stored in the interpolation frame storage area 123 is the image shown in FIG. Compared with the image when the correction weighting coefficient shown in FIG. 22 is not shifted, the afterimage of the non-normal area (the perforated area P and the collision area Q) becomes lighter.

  When the storage of the interpolation frame is completed, the motion compensation unit 150 ends the motion compensation process. The control unit 110 transmits the interpolation frame I [n] [1] stored in the interpolation frame storage area 123 to an external device as needed.

  According to this embodiment, images are averaged using different weights in the normal area and the non-normal area, so that an unnatural afterimage generated in the hidden surface area, in particular, near the boundary between the moving object and the background. The resulting unnatural afterimage can be thinned. In addition, since the frame interpolation device 100 shifts the weight used when averaging the perforated regions to the backward motion compensated image side, a natural image in the hidden surface region in which the background is hidden. Can be generated. In addition, since the frame interpolation apparatus 100 shifts the weight used when averaging the collision areas to the forward motion compensated image side, a natural image is generated in the area where the background appears in the hidden surface area. it can.

  In addition, the frame interpolation device 100 calculates a correction weighting coefficient based on the ratio of perforated pixels or collision pixels in the neighboring pixels of the selected pixel. More specifically, the correction weighting coefficient is calculated based on the certainty factor Ap calculated based on the ratio of the perforated pixels in the neighboring pixels or the certainty factor Aq calculated based on the ratio of the collision pixels in the neighboring pixels. doing. Depending on the image, such as an image in which the movement of the object or background is not completely parallel, small perforated areas and collision areas may be scattered in the image. If the selected pixel is not a pixel in the hidden surface area but a holed pixel or a collision pixel scattered in the image, the certainty factor, that is, the ratio is low, and as a result, the correction weighting factor is a value close to the reference weighting factor. It is thought that it becomes. Therefore, when the selected pixel is a holed pixel or a collision pixel scattered in the image, the frame interpolation device 100 can average the selected pixels of the two motion compensated images with a weighting factor close to the reference weighting factor. It is less likely that pixels having values significantly different from the surrounding pixel values occur suddenly in the frame. As a result, the frame interpolation device 100 can generate a more natural interpolation frame.

  The above-described embodiment is an example, and various changes and applications are possible. For example, in the above-described embodiment, the ratio Rp of the perforated pixels occupying the neighboring pixels is directly acquired as the certainty factor Ap, but the value of the certainty factor Ap is not necessarily a value that matches the ratio Rp. For example, when the ratio Rp is larger than the preset value s, the interpolation frame generation unit 153 may regard the selected pixel as a hidden surface area pixel and set the certainty factor Ap to 1. FIG. 26 is a diagram illustrating an example of the relationship between the ratio of neighboring pixels and the certainty factor. Then, the interpolation frame generation unit 153 calculates the correction weighting coefficient Wp based on Expression (5). Thereby, when there is a high possibility that the selected pixel is a pixel in the hidden surface area, the pixel in the reverse direction compensation image can be used as the pixel in the interpolation frame as it is, so that the interpolation frame generation unit 153 can further reduce the afterimage. As a result, the interpolation frame generation unit 153 can generate a more natural interpolation frame.

  In addition, when the ratio Rp is smaller than the preset value d, the frame interpolation device 100 regards the selected pixel as not a hidden area pixel but a perforated pixel scattered in the image. The degree Ap may be 0. Then, the interpolation frame generation unit 153 calculates the correction weighting coefficient Wp based on Expression (5). As a result, when there is a high possibility that the selected pixel is not a pixel in the hidden surface area but a perforated pixel scattered in the image, the pixel value calculated using the reference weight coefficient Wr is used as the selected pixel of the interpolation frame. Since the value can be a value, the pixel value of the selected pixel can be a value closer to the surrounding pixel values. As a result, the interpolation frame generation unit 153 can generate a more natural interpolation frame.

  In the above-described embodiment, the ratio Rq of the collision pixels occupying the neighboring pixels is directly acquired as the certainty factor Aq. However, the value of the certainty factor Aq is not necessarily a value that matches the ratio Rq. For example, if the ratio Rq is larger than the preset value s, the frame interpolation apparatus 100 may regard the selected pixel as a hidden surface area pixel and set the certainty factor Ap to 1. Then, the interpolation frame generation unit 153 calculates the correction weighting coefficient Wq based on Expression (6). Thereby, when there is a high possibility that the selected pixel is a pixel of the hidden surface area, the pixel of the forward compensation image can be used as it is as the pixel of the interpolation frame, so that the interpolation frame generation unit 153 further increases the afterimage generated in the interpolation frame. Can be thin. As a result, the interpolation frame generation unit 153 can generate a more natural interpolation frame.

  Further, when the ratio Rq is smaller than the preset value d, the frame interpolation apparatus 100 regards the selected pixel as a collision pixel scattered in the image, not a pixel in the hidden surface area, and the certainty factor. Aq may be 0. Then, the interpolation frame generation unit 153 calculates the correction weighting coefficient Wq based on Expression (6). As a result, when there is a high possibility that the selected pixel is not a pixel in the hidden surface area but a collision pixel scattered in the image, the pixel value calculated using the reference weight coefficient Wr is used as the value of the selected pixel in the interpolation frame. Therefore, the pixel value of the selected pixel can be made closer to the peripheral pixel values. As a result, the interpolation frame generation unit 153 can generate a more natural interpolation frame.

  In the above-described embodiment, the neighboring pixels are pixels in a 5 × 5 pixel square range centered on the selected pixel. However, the neighboring pixel range is in a 5 × 5 pixel square range. It is not limited. Further, the range of neighboring pixels is not limited to a square range, and the range of neighboring pixels may be a rectangular range such as a rectangle or rhombus centered on the selected pixel. Further, the range of neighboring pixels is not limited to a rectangular range, and may be, for example, a circular or elliptical range. Further, the position of the selected pixel may not be the center of the range. For example, the selected pixel may be located at the end of the range.

  In the above-described embodiment, the interpolation frame generation unit 153 shifts the weight to the backward motion compensation image side when the selected pixel is a perforated pixel, but the shift direction is not limited to the backward direction side. . The interpolation frame generation unit 153 may appropriately shift the weight to the forward motion compensated image side according to the nature of the video.

  In the above-described embodiment, the interpolation frame generation unit 153 shifts the weight to the forward motion compensation image side when the selected pixel is a collision pixel, but the shift direction is not limited to the forward direction side. The interpolated frame generation unit 153 may appropriately shift the weight toward the backward motion compensated image according to the nature of the video.

  In the above-described embodiment, the input frames used for generating the interpolation frame are the input frames F [n−1] and F [n] immediately before and after the interpolation frame. The frame is not limited to the input frame immediately before and immediately after. The input frame may be an input frame located at a position two frames or more away from the interpolation frame.

  Further, in the above-described embodiment, the interpolation frame generation unit 153 calculates the reference weight coefficient Wr based on the time position T of the interpolation frame. However, the interpolation frame generation unit 153 does not use the time position T, and the reference weight coefficient Wr may be calculated. For example, the interpolation frame generation unit 153 may set the reference weight coefficient Wr uniformly to 0.5, and simply average the normal areas uniformly.

  In the above-described embodiment, the unit area has been described as one pixel area. However, the unit area may be an area composed of a plurality of pixels. In this case, the perforated pixel, the collision pixel, and the non-normal pixel can be rephrased as the perforated unit area, the collision unit area, and the non-normal unit area, respectively. The perforated unit area is a concept including a perforated pixel, and the collision unit area is a concept including a collision pixel. Further, the unusual unit area is a concept including unusual pixels.

  In the above-described embodiment, the motion compensation unit 150 executes the interpolation frame generation process after the generation of the motion compensated image (reverse direction motion compensated image and forward direction motion compensated image) is completed in the motion compensated image generation process. However, the motion compensation unit 150 may perform the interpolation frame generation process before the generation of the motion compensated image is completed. The motion compensation unit 150 repeats generation of a certain range of motion compensated images (for example, one block of motion compensated images) and generation of a certain range of interpolated frames (for example, one block of interpolated frames), thereby interpolating frames. May be generated. The motion compensation unit 150 may generate an interpolation frame by repeating the generation of a motion compensated image for one pixel and the generation of an interpolation frame for one pixel.

  In the above-described embodiment, the interpolated motion vector is described as being composed of a pair of interpolated motion vectors of the backward interpolated motion vector MVa and the forward interpolated motion vector MVb. The interpolation motion vector may not be configured. The interpolation motion vector may be composed of any one of the backward interpolation motion vector MVa and the forward interpolation motion vector MVb. In this case, the interpolated motion vector is determined based on the information on one of the interpolated motion vector MVa and the forward interpolated motion vector MVb and the information on the motion vector MV at any time. You may specify.

  In the above-described embodiment, the video in which the frame interpolation device 100 inserts the interpolation frame has been described as a slow playback video. However, the video in which the frame interpolation device 100 inserts the interpolation frame is not limited to the slow playback video. For example, the video into which the frame interpolation device 100 inserts the interpolation frame may be a normal playback speed video.

  In the above-described embodiment, the frame interpolation device 100 is configured to output the generated interpolation frame to an external device. However, the frame interpolation device 100 has a reproduction function, and generates the generated interpolation frame and the input frame. It may be configured to output the video generated based on the display device. In this case, the frame interpolation device 100 may be configured to output a video signal from the output unit 160, or may be configured to include a display unit that displays video and output video to the display unit. . Of course, the frame interpolation device 100 may be a device that does not have a video reproduction function and only generates an interpolation frame.

  The frame interpolation device 100 can be regarded as a finished product such as a television, a recorder, a personal computer, a fixed telephone, a mobile phone, a smartphone, a tablet terminal, a PDA (Personal Digital Assistant), a game machine, and the like. Further, the frame interpolation apparatus 100 can be regarded as a component mounted on a finished product, for example, a semiconductor, a semiconductor substrate, or the like.

  The frame interpolation apparatus 100 according to the present embodiment may be realized by a dedicated system or an ordinary computer system. For example, a program for executing the above-described operation is stored and distributed in a computer-readable recording medium such as an optical disk, a semiconductor memory, a magnetic tape, or a flexible disk, the program is installed in the computer, and the above-described processing is performed. The frame interpolation apparatus 100 may be configured by executing the above. Further, the program may be stored in a disk device provided in a server device on a network such as the Internet so that it can be downloaded to a computer. Further, the above-described functions may be realized by cooperation between an OS (Operating System) and application software. In this case, a part other than the OS may be stored and distributed in a medium, or a part other than the OS may be stored in a server device and downloaded to a computer.

  Although the embodiment of the present invention has been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. The novel embodiment can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 100 ... Frame interpolation apparatus 110 ... Control part 120 ... Storage part 121 ... Frame memory area 122 ... Motion vector storage area 123 ... Interpolation frame storage area 130 ... Input part 140 ... Motion search part 150 ... Motion compensation part 151 ... Motion vector interpolation part 152 ... Motion compensated image generation unit 153 ... Interpolation frame generation unit 160 ... Output unit

Claims (5)

The motion of the image between the two frames and the interpolated frame based on the motion vector indicating the motion of the image between the two frames and the time position of the interpolated frame inserted between the two frames A motion vector interpolation unit that calculates an interpolated motion vector indicating the calculated motion vector and assigns the calculated interpolated motion vector to the interpolated frame for each unit region;
A forward motion compensated image generated based on image information of a frame on the forward direction side of the two frames and the interpolated motion vector, and a frame on the reverse side of the two frames. A motion compensation image generation unit that generates a backward motion compensation image generated based on the image information and the interpolation motion vector;
A normal region in which one or a pair of the interpolated motion vectors is assigned to each unit region, a collision region in which a plurality of or a plurality of pairs of the interpolated motion vectors are assigned to each unit region, and the interpolated motion vectors. The interpolation is performed by averaging the corresponding regions of the forward motion compensation image and the backward motion compensation image with different weights from the non-normal region composed of at least one of the perforated regions not assigned. An interpolation frame generation unit for generating a frame,
Frame interpolation device. The interpolation frame generation unit
Calculated by shifting the weight to either one of the forward motion compensated image and the backward motion compensated image based on the value of the reference weight coefficient that is a weight coefficient used when averaging the normal region A weighted average of the non-normal region of the forward motion compensated image and the non-normal region of the reverse motion compensated image based on the correction weighting factor to be
The frame interpolation apparatus according to claim 1. The non-normal area includes the perforated area,
The interpolation frame generation unit is configured to generate the forward motion compensation image based on the correction weighting factor calculated by shifting a weight to the backward motion compensation image side based on a value of the reference weighting factor. A weighted average of the perforated region and the perforated region of the reverse motion compensated image;
The frame interpolation apparatus according to claim 2. The non-normal area includes the collision area,
The interpolation frame generation unit is configured to generate the forward motion compensated image based on the correction weight coefficient calculated by shifting a weight toward the forward motion compensated image from the reference weight coefficient value. A weighted average of the collision area and the collision area of the reverse motion compensated image;
The frame interpolation device according to claim 2 or 3. Interpolated motion indicating the motion of the image between the two frames and the interpolated frame based on the motion vector indicating the motion of the image between the frames and the time position of the interpolated frame inserted between the two frames A motion vector interpolation step of calculating a vector and assigning the calculated interpolation motion vector to the interpolation frame for each unit region;
A forward motion compensated image generated based on image information of a frame on the forward direction side of the two frames and the interpolated motion vector, and a frame on the reverse side of the two frames. A motion compensation image generating step for generating a backward motion compensation image generated based on the image information and the interpolated motion vector;
A normal region to which one or a pair of the interpolated motion vectors are assigned per unit region, a collision region to which a plurality of or a plurality of pairs of the interpolated motion vectors are assigned to the unit region, and the interpolated motion By averaging the corresponding regions of the forward motion compensation image and the backward motion compensation image with different weights in the non-normal region composed of at least one of the perforated regions to which no vector is assigned. An interpolation frame generation step for generating the interpolation frame,
Frame interpolation method.

Images