WO2013073369A1 - Dispositif de traitement d'images, procédé de traitement d'images, programme de traitement d'images, dispositif d'affichage d'images et procédé d'affichage d'images - Google Patents

Dispositif de traitement d'images, procédé de traitement d'images, programme de traitement d'images, dispositif d'affichage d'images et procédé d'affichage d'images Download PDF

Info

Publication number
WO2013073369A1
WO2013073369A1 PCT/JP2012/078157 JP2012078157W WO2013073369A1 WO 2013073369 A1 WO2013073369 A1 WO 2013073369A1 JP 2012078157 W JP2012078157 W JP 2012078157W WO 2013073369 A1 WO2013073369 A1 WO 2013073369A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
vector
integrated
viewpoint
time
Prior art date
Application number
PCT/JP2012/078157
Other languages
English (en)
Japanese (ja)
Inventor
俊明 久保
督 那須
直之 藤山
良樹 小野
知篤 堀部
Original Assignee
三菱電機株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 三菱電機株式会社 filed Critical 三菱電機株式会社
Priority to JP2013544205A priority Critical patent/JP5627798B2/ja
Publication of WO2013073369A1 publication Critical patent/WO2013073369A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/97—Determining parameters from multiple pictures
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/128—Adjusting depth or disparity
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G06T2207/10021—Stereoscopic video; Stereoscopic image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20228—Disparity calculation for image-based rendering
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N2013/0074—Stereoscopic image analysis
    • H04N2013/0081—Depth or disparity estimation from stereoscopic image signals

Definitions

  • the present invention relates to an image processing device, an image processing method, an image processing program, an image display device, and an image display method.
  • a stereoscopic image display technique using binocular parallax as an image display technique for an observer to obtain a pseudo depth feeling.
  • an image is viewed by the observer by dividing the image viewed with the left eye and the image viewed with the right eye into a left eye and a right eye of the observer in a three-dimensional space. I feel a solid.
  • the left-eye image and the right-eye image are switched alternately in time and displayed on the display, and at the same time, the left and right images are synchronized with the timing of the image switching.
  • Left and right eyes by using a method to separate the left and right fields of view temporally using glasses that control the amount of light transmitted through the lens, and a barrier or lens that limits the display angle of the image on the front of the display.
  • the observer focuses on the display surface while adjusting the convergence angle of the eyes to the position of the protruding object.
  • This inconsistency has a problem that it induces eye fatigue in the observer when the pop-out amount is too large.
  • the depth sensation that induces eye fatigue varies depending on the distance between the viewer and the display surface of the display and the characteristics of the viewer.
  • the convergence angle represents the angle formed by the left eye line of sight and the right eye line of sight
  • the sense of depth represents the amount of protrusion or retraction of the object expressed by binocular parallax.
  • a hold-type display such as a liquid crystal display
  • the same image continues to be displayed for one frame period, and when an object in the image moves, human eyes follow the moving object continuously, Since the movement of the object is a discontinuous movement in units of one frame, there is a problem that the edge portion looks blurred.
  • Patent Document 1 discloses a technique for generating an intermediate image corresponding to an arbitrary viewpoint between two images from two images having different viewpoints.
  • an interpolated frame is generated from the previous frame of the video signal and the current frame, the interpolated frame is interpolated between the previous frame and the current frame, and the frame rate of the video signal is calculated.
  • a technique for converting is disclosed.
  • An object of the present invention is to provide an image processing device, an image processing method, an image processing program, an image display device, and an image display method capable of generating an image of an arbitrary viewpoint at an arbitrary time from a plurality of input images.
  • An image processing apparatus includes: Receiving the image P (T1, V1) of the first viewpoint V1 at the first time T1 and the image P (T2, V1) of the viewpoint V1 at the second time T2 different from the first time T1; A motion estimation unit for calculating a first motion vector from the image P (T1, V1) to the image P (T2, V1); The image P (T1, V1) and the image P (T1, V2) of the second viewpoint V2 different from the first viewpoint V1 at the time T1 are received, and the image P (T1, V1) is received from the image P (T1, V1).
  • a parallax estimation unit that calculates a first parallax vector to P (T1, V2);
  • An integration unit that calculates an integrated vector in which the first motion vector and the first disparity vector are integrated from the first motion vector and the first disparity vector;
  • an image display device includes: The above image processing apparatus; An image display unit for displaying the image P (t, v) generated by the image generation unit of the image processing device; It is characterized by providing.
  • the image processing method includes: Receiving the image P (T1, V1) of the first viewpoint V1 at the first time T1 and the image P (T2, V1) of the viewpoint V1 at the second time T2 different from the first time T1; A motion estimation step of calculating a first motion vector from the image P (T1, V1) to the image P (T2, V1); The image P (T1, V1) and the image P (T1, V2) of the second viewpoint V2 different from the first viewpoint V1 at the time T1 are received, and the image P (T1, V1) is received from the image P (T1, V1).
  • the image display method includes: The above image processing method; An image display step for displaying the image P (t, v) generated by the image generation step of the image processing method; It is characterized by having.
  • An image processing program is Receiving the image P (T1, V1) of the first viewpoint V1 at the first time T1 and the image P (T2, V1) of the viewpoint V1 at the second time T2 different from the first time T1; A motion estimation step of calculating a first motion vector from the image P (T1, V1) to the image P (T2, V1); The image P (T1, V1) and the image P (T1, V2) of the second viewpoint V2 different from the first viewpoint V1 at the time T1 are received, and the image P (T1, V1) is received from the image P (T1, V1).
  • an image at an arbitrary viewpoint at an arbitrary time can be generated from a plurality of input images.
  • FIG. 1 It is a figure which shows the structure of the image display apparatus which concerns on embodiment. It is a figure for demonstrating operation
  • (A) And (b) is a figure for demonstrating the operation
  • FIG. 1 is a diagram showing a configuration of an image display apparatus 100 according to an embodiment of the present invention.
  • the image display device 100 includes an image processing unit 10 and an image display unit 20.
  • the image processing unit 10 generates an image of an arbitrary viewpoint at an arbitrary time from a plurality of input images, and the image display unit 20 displays the image generated by the image processing unit 10.
  • the image processing unit 10 includes a motion estimation unit 11, a parallax estimation unit 12, a parallax / motion integration unit (hereinafter referred to as “integration unit”) 13, and an image generation unit 14.
  • the image processing unit 10 includes, as input images, an image P (T1, V1) of the first viewpoint V1 at the first time T1 and a first viewpoint V1 at a second time T2 different from the first time T1.
  • Image P (T2, V1) and an image P (T1, V2) of the second viewpoint V2 different from the first viewpoint V1 at the first time T1 are input.
  • the image P (T1, V1) of the viewpoint V1 at the time T1 is, for example, an image captured from the viewpoint V1 at the time T1.
  • the motion estimation unit 11 receives the image P (T1, V1) of the viewpoint V1 at the time T1 and the image P (T2, V1) of the viewpoint V1 at the time T2, and the image P (T2) from the image P (T1, V1). , V1) is calculated as the first motion vector MV1. Specifically, the motion estimation unit 11 refers to the image P (T1, V1) and the image P (T2, V1), and for each pixel, the image P (T1, V1) to the image P (T2, V1). The first motion vector MV1 is calculated and output to the integration unit 13.
  • the parallax estimation unit 12 receives the image P (T1, V1) of the viewpoint V1 at the time T1 and the image P (T1, V2) of the viewpoint V2 at the time T1, and the image P (T1) from the image P (T1, V1). , V2) is calculated as the first parallax vector DV1. Specifically, the parallax estimation unit 12 refers to the image P (T1, V1) and the image P (T1, V2), and the image P (T1, V1) to the image P (T1, V2) for each pixel. The first parallax vector DV1 is calculated and output to the integration unit 13.
  • the integration unit 13 uses the first motion vector MV1 calculated by the motion estimation unit 11 and the first disparity vector DV1 calculated by the disparity estimation unit 12, and the first motion vector MV1 and the first disparity vector. An integrated vector obtained by integrating DV1 is calculated.
  • the image generation unit 14 includes at least one of the image P (T1, V1), the image P (T2, V1), and the image P (T1, V2), and the integration vector calculated by the integration unit 13.
  • An image P (t, v) at an arbitrary viewpoint v at an arbitrary time t is generated.
  • the time t and the viewpoint v are specified by, for example, an external module or apparatus of the image processing unit 10 or a user.
  • the range of the time t is, for example, T1 ⁇ t ⁇ T2, but is not limited to this range.
  • the range of the viewpoint v is, for example, V1 ⁇ v ⁇ V2, but is not limited to this range.
  • the integration unit 13 uses the first motion vector MV1 and the first parallax vector DV1 as the integration vector to generate a second signal from the image P (T2, V1) to the image P (t, v).
  • An integrated vector TV2 and a third integrated vector TV3 from the image P (T1, V2) to the image P (t, v) are calculated.
  • the image generation unit 14 generates an image P (t, v) from the image P (T2, V1), the image P (T1, V2), the second integrated vector TV2, and the third integrated vector TV3. .
  • the integration unit 13 includes a motion conversion unit 15, a parallax conversion unit 16, a vector addition unit 17, a first vector subtraction unit 18, and a second vector subtraction unit 19. Prepare.
  • the motion conversion unit 15 converts the first motion vector MV1 for each pixel from the image P (T1, V1) to the second motion vector MV2 to the position (or image) of the viewpoint V1 at an arbitrary time t, The result is output to the vector adder 17.
  • the disparity conversion unit 16 converts the first disparity vector DV1 from the image P (T1, V1) to the second disparity vector DV2 from the image P (T1, V1) to the position (or image) at an arbitrary viewpoint v for each pixel, The result is output to the vector adder 17.
  • the vector addition unit 17 adds the second motion vector MV2 obtained by the motion conversion unit 15 and the second parallax vector DV2 obtained by the parallax conversion unit 16 for each pixel, and performs first integration.
  • the vector TV1 is calculated and output to the first vector subtracting unit 18 and the second vector subtracting unit 19.
  • the second motion vector MV2 is a vector from the image P (T1, V1) of the viewpoint V1 at the time T1 to the position of the viewpoint V1 at the time t
  • the second disparity vector DV2 is the image P (of the viewpoint V1 at the time T1.
  • the first integrated vector TV1 is the position of the viewpoint v at time t from the image P (T1, V1) of the viewpoint V1 at time T1 (or Corresponds to a vector to (image).
  • the first vector subtracting unit 18 subtracts the first motion vector MV1 from the first integrated vector TV1 obtained by the vector adding unit 17 for each pixel to calculate a second integrated vector TV2 to obtain an image.
  • the first motion vector MV1 is a vector from the image P (T1, V1) of the viewpoint V1 at the time T1 to the image P (T2, V1) of the viewpoint V1 at the time T2, and the first integrated vector TV1 is at the time T1. Since it is a vector from the image P (T1, V1) of the viewpoint V1 to the position of the viewpoint v at time t, the second integrated vector TV2 is from the image P (T2, V1) of the viewpoint V1 at time t2 at time t2. This corresponds to a vector to the position (or image) of the viewpoint v.
  • the second vector subtraction unit 19 subtracts the first parallax vector DV1 from the first integrated vector TV1 obtained by the vector addition unit 17 for each pixel to calculate a third integrated vector TV3, and the image Output to the generation unit 14.
  • the first disparity vector DV1 is a vector from the image P (T1, V1) of the viewpoint V1 at time T1 to the image P (T1, V2) of the viewpoint V2 at time T1, and the first integrated vector TV1 is at time T1. Since it is a vector from the image P (T1, V1) of the viewpoint V1 to the position of the viewpoint v at the time t, the third integrated vector TV3 is from the image P (T1, V2) of the viewpoint V2 at the time t1 at the time t. This corresponds to a vector to the position (or image) of the viewpoint v.
  • the image generation unit 14 includes an image P (T2, V1), an image P (T1, V2), a second integrated vector TV2 calculated by the first vector subtraction unit 18, and a second vector subtraction unit 19 From the third integrated vector TV3 calculated by the above, an output image P (t, v) of the viewpoint v at time t is generated and output to the image display unit 20.
  • the image display unit 20 displays the output image P (t, v) of the viewpoint v at the time t generated by the image generation unit 14.
  • FIG. 2 is a diagram for explaining the operation of the motion estimation unit 11.
  • the motion estimation unit 11 obtains the first motion vector MV1 from the image P (T1, V1) of the viewpoint V1 at the time T1 to the image P (T2, V1) of the viewpoint V1 at the time T2. calculate.
  • the motion estimation unit 11 obtains the first motion vector MV1 by, for example, a block matching method. Specifically, the motion estimation unit 11 extracts a target image block including the target pixel from the image P (T1, V1), and estimates to which region of the image P (T2, V1) the target image block has moved. Then, the relative position of the estimated region with respect to the target image block is detected as the first motion vector MV1 for the target pixel.
  • FIG. 3 is a block diagram illustrating a configuration example of the motion estimation unit 11.
  • the motion estimation unit 11 includes an attention image block cutout unit 11a, a search image block cutout unit 11b, and a motion vector determination unit 11c.
  • the target image block cutout unit 11a when obtaining the first motion vector MV1 for the target pixel at the position p (i, j) of the image P (T1, V1), the target image block cutout unit 11a A peripheral region of the target pixel is cut out as a block from the image P (T1, V1).
  • the target image block cutout unit 11a has a rectangular region (block) D2B centered on the target pixel and having a width (horizontal size) of (2 ⁇ BM + 1) and a height (vertical size) of (2 ⁇ BN + 1). Cut out (i, j).
  • i represents a horizontal position in the image
  • j represents a vertical position in the image.
  • the search image block cutout unit 11b has a plurality of the same size as the rectangular region D2B (i, j) centered on each position included in the search range of the motion vector corresponding to the target pixel from the image P (T2, V1). Cut out rectangular areas (blocks).
  • the search range is a set S (i, j) of coordinates represented by the following formula (1).
  • S (i, j) ⁇ (i + k, j + l) ⁇ (1)
  • k ⁇ SH,..., ⁇ 1, 0, +1,..., SH (SH is a predetermined integer of 1 or more)
  • l ⁇ SV,. ,..., SV (SV is a predetermined integer of 1 or more).
  • the search range defined in this way is a rectangular area having a horizontal size of 2SH + 1 and a vertical size of 2SV + 1.
  • the search image block cutout unit 11b has the same size as the rectangular area D2B (i, j) centered on each position (i + k, j + l) included in the coordinate set S (i, j) (2 ⁇ SH + 1) ⁇ (2 ⁇ SV + 1) rectangular regions D1B (i + k, j + l) are cut out.
  • the motion vector determination unit 11c is configured between the block D2B (i, j) input from the target image block cutout unit 11a and the block D1B (i + k, j + l) input from the search image block cutout unit 11b.
  • the difference absolute value sum SAD (i + k, j + l) which is the sum of the absolute values of the differences between all the pixels in the block, that is, (2 ⁇ BM + 1) ⁇ (2 ⁇ BN + 1) pixels corresponding to each other, is obtained. .
  • This difference absolute value sum SAD (i + k, j + l) is expressed by the following equation (2).
  • D1 (i + k + r, j + l + s) represents the pixel value of the position (i + k + r, j + l + s) of the image P (T2, V1)
  • D2 (i + r, j + s) represents the image P (T1, V1). Represents the pixel value at position (i + r, j + s).
  • This first motion vector MV1 may be represented as MV1 (i, j) for the purpose of distinguishing it from motion vectors for other pixels.
  • the motion vector determination unit 11c performs the above motion vector detection on all the pixels of the image P (T1, V1), and detects the first motion vector MV1 for each pixel.
  • the processing method of the motion estimation unit 11 is not limited to the above method, and other motion vector calculation methods such as a method of obtaining a motion vector using a phase correlation function may be employed.
  • FIG. 5 is a diagram for explaining the operation of the motion conversion unit 15.
  • the motion conversion unit 15 converts the first motion vector MV1 from the image P (T1, V1) of the viewpoint V1 at the time T1 to the position of the viewpoint V1 at the time t, for each pixel.
  • the motion vector MV2 MV1 ⁇ (t ⁇ T1) / (T2 ⁇ T1) (3)
  • FIG. 6 is a diagram for explaining the operation of the parallax estimation unit 12.
  • the parallax estimation unit 12 obtains the first parallax vector DV1 from the image P (T1, V1) of the viewpoint V1 at the time T1 to the image P (T1, V2) of the viewpoint V2 at the time T1. calculate.
  • the configuration of the parallax estimation unit 12 is the same as the configuration of the motion estimation unit 11 illustrated in FIG. 3, for example.
  • the processing method of the parallax estimation unit 12 is not limited to the above method, and other motion vector calculation methods such as a method of obtaining a motion vector using a phase correlation function may be employed.
  • FIG. 7 is a diagram for explaining the operation of the parallax converter 16.
  • the disparity conversion unit 16 converts the first disparity vector DV1 from the image P (T1, V1) of the viewpoint V1 at the time T1 to the position of the viewpoint v at the time T1, for each pixel.
  • the disparity vector DV2. For this conversion, the following equation (4) is used for each pixel.
  • DV2 DV1 ⁇ (v ⁇ V1) / (V2 ⁇ V1) (4)
  • FIG. 8 is a diagram for explaining the operation of the vector addition unit 17.
  • the vector addition unit 17 uses the second motion vector MV2 and the second parallax vector DV2 for each pixel from the image P (T1, V1) of the viewpoint V1 at time T1.
  • a vector to the position of the viewpoint v at time t is calculated as a first integrated vector TV1.
  • the following equation (5) is used for each pixel.
  • TV1 DV2 + MV2 (5)
  • FIG. 9 is a diagram for explaining the operation of the first vector subtraction unit 18.
  • the first vector subtraction unit 18 uses the first integrated vector TV1 and the first motion vector MV1 for each pixel, and uses the image P (T2, T2, T2 at time T2.
  • a vector from V1) to the position of the viewpoint v at time t is calculated as a second integrated vector TV2.
  • the following equation (6) is used for each pixel.
  • TV2 TV1-MV1 (6)
  • FIG. 10 is a diagram for explaining the operation of the second vector subtraction unit 19.
  • the second vector subtracting unit 19 uses the first integrated vector TV1 and the first parallax vector DV1 for each pixel, and the image P (T1, T1) at the time T1.
  • a vector from V2) to the position of the viewpoint v at time t is calculated as a third integrated vector TV3.
  • the following equation (7) is used for each pixel.
  • TV3 TV1-DV1 (7)
  • FIG. 11 is a diagram for explaining the operation of the image generation unit 14.
  • the image generation unit 14 uses the image P (T2, V1), the image P (T1, V2), the second integrated vector TV2, and the third integrated vector TV3 to generate a time t.
  • the image generation unit 14 moves the pixel value of each pixel of the image P (t, v) by an inverse vector ( ⁇ TV2) of the second integrated vector from the pixel P (T2).
  • the image generation unit 14 determines the position pa () of the image P (t, v).
  • the pixel value Dp (x, y) of the pixel at x, y) is converted into the pixel value D1 (x-tv2x, y-tv2y) at the position pb (x-tv2x, y-tv2y) of the image P (T2, V1).
  • D2 (x-tv3x, y-tv3y) at the position pc (x-tv3x, y-tv3y) of the image P (T1, V2) is obtained by the following equation (8).
  • Dp (x, y) ⁇ D1 (x ⁇ tv2x, y ⁇ tv2y) + D2 (x ⁇ tv3x, y ⁇ tv3y) ⁇ / 2 (8)
  • FIG. 12 is a diagram for explaining processing of the image processing unit 10.
  • FIG. 12 focusing on the pixel at the position p (x, y) of the image P (T1, V1), the processing of each unit of the image processing unit 10 will be described.
  • FIG. 12 for convenience of explanation, the position of each image, the rectangular area, and the like are shown superimposed.
  • the motion estimation unit 11 identifies the rectangular region R1 of the image P (T2, V1) corresponding to the rectangular region R0 centered on the pixel at the position p (x, y) of the image P (T1, V1). Then, the motion estimation unit 11 converts the vector from the position p (x, y) to the center position p1 of the rectangular area R1 with respect to the pixel at the position p (x, y) of the image P (T1, V1). Calculated as a motion vector MV1.
  • the motion conversion unit 15 calculates the second motion vector MV2 for the pixel at the position p (x, y) of the image P (T1, V1) from the first motion vector MV1 by the above equation (3). .
  • the parallax estimation unit 12 identifies the rectangular region R2 of the image P (T1, V2) corresponding to the rectangular region R0 centered on the pixel at the position p (x, y) of the image P (T1, V1). Then, the parallax estimation unit 12 converts the vector from the position p (x, y) to the center position p2 of the rectangular region R2 to the first pixel for the pixel at the position p (x, y) of the image P (T1, V1). Calculated as the parallax vector DV1.
  • the parallax conversion unit 16 calculates the second parallax vector DV2 for the pixel at the position p (x, y) of the image P (T1, V1) from the first parallax vector DV1 by the above equation (4). .
  • the vector addition unit 17 adds the second motion vector MV2 and the second parallax vector DV2, and performs first integration for the pixel at the position p (x, y) of the image P (T1, V1).
  • Vector TV1 (tv1x, tv1y) is calculated.
  • the first integrated vector TV1 corresponds to a vector from the position p (x, y) of the image P (T1, V1) to the position p3 corresponding to the target pixel of the output image P (t, v).
  • the coordinates of the position p3 are (x + tv1x, y + tv1y).
  • the first vector subtraction unit 18 subtracts the first motion vector MV1 from the first integrated vector TV1 to obtain the second for the pixel at the position p (x, y) of the image P (T1, V1).
  • the integrated vector TV2 (tv2x, tv2y) is calculated.
  • the second integrated vector TV2 corresponds to a vector from the position p1 of the image P (T2, V1) to the position p3 of the output image P (t, v). Accordingly, the second integrated vector TV2 corresponds to the position p3 (x + tv1x, y + tv1y) of the output image P (t, v).
  • the second vector subtraction unit 19 subtracts the first parallax vector DV1 from the first integrated vector TV1 to obtain a third pixel for the pixel at the position p (x, y) in the image P (T1, V1).
  • the integrated vector TV3 (tv3x, tv3y) is calculated.
  • the third integrated vector TV3 corresponds to a vector from the position p2 of the image P (T1, V2) to the position p3 of the output image P (t, v). Therefore, the third integrated vector TV3 corresponds to the position p3 (x + tv1x, y + tv1y) of the output image P (t, v).
  • the pixel value Dp (x + tv1x, y + tv1y) of the pixel at the position p3 (x + tv1x, y + tv1y) of the output image P (t, v) can be obtained by the following equation (9).
  • the pixel value of the pixel at the position (x, y) of the image P (T2, V1) is represented as D1 (x, y)
  • the pixel value at the position (x, y) of the image P (T1, V2) is represented.
  • the pixel value is represented as D2 (x, y).
  • Dp (x + tv1x, y + tv1y) ⁇ D1 (x + tv1x ⁇ tv2x, y + tv1y ⁇ tv2y) + D2 (x + tv1x ⁇ tv3x, y + tv1y ⁇ tv3y) ⁇ / 2 (9)
  • the image generation unit 14 can obtain the pixel value of each pixel of the image P (t, v) using the above formula (9) or (10).
  • the integration unit 13 calculates the first integrated vector TV1, the second integrated vector TV2, and the third integrated vector TV3 for the pixel at each position (xm, yn) of the image P (T1, V1). .
  • the second integrated vector TV2 and the third integrated vector TV3 for the pixel at the position (xm, yn) of the image P (T1, V1) are the same as those in the output image P (t, v).
  • yn + tv1y) is obtained as a second integrated vector TV2 and a third integrated vector TV3.
  • the image generation unit 14 performs the following first or second processing, for example.
  • the image generation unit 14 obtains each of the white circles of the image P (t, v) from the second integrated vector TV2 and the third integrated vector TV3 corresponding to the positions of the black triangles of the image P (t, v).
  • the second integrated vector TV2 and the third integrated vector TV3 corresponding to the position are calculated by interpolation.
  • the image generation unit 14 uses the second integrated vector TV2 and the third integrated vector TV3 corresponding to the calculated positions of the white circles to calculate the image P (t, v
  • the pixel value Dp (xm, yn) at each position (xm, yn) of the white circle mark of) is calculated.
  • Dp (xm, yn) ⁇ D1 (xm ⁇ tv2x, yn ⁇ tv2y) + D2 (xm ⁇ tv3x, yn ⁇ tv3y) ⁇ / 2 (11)
  • the image generation unit 14 sets the pixel value of the pixel to D1. Used as (xm-tv2x, yn-tv2y).
  • the position (xm ⁇ tv2x, yn) is calculated from each pixel value of the image P (T2, V1).
  • -Tv2y) is interpolated, and the pixel value is used as D1 (xm-tv2x, yn-tv2y).
  • the image generation unit 14 sets the pixel value of the pixel to D2 (xm ⁇ tv3x, yn ⁇ ). tv3y).
  • the position (xm ⁇ tv3x, yn) is calculated from each pixel value of the image P (T1, V2).
  • -Tv3y) is interpolated and used as D2 (xm-tv3x, yn-tv3y).
  • the image generation unit 14 uses the second integrated vector TV2 and the third integrated vector TV3 corresponding to each position of the black triangle mark of the image P (t, v), and calculates the image P ( The pixel value Dp (xm + tv1x, yn + tv1y) at each position (xm + tv1x, yn + tv1y) of the black triangle mark of t, v) is calculated.
  • Dp (xm + tv1x, yn + tv1y) ⁇ D1 (xm + tv1x ⁇ tv2x, yn + tv1y ⁇ tv2y) + D2 (xm + tv1 ⁇ tv3x, yn + tv1y ⁇ tv3y) ⁇ / 2 (12)
  • the image generation unit 14 sets the pixel value of the pixel to D1 when the pixel exists at the position (xm + tv1x ⁇ tv2x, yn + tv1y ⁇ tv2y) in the image P (T2, V1). Used as (xm + tv1x ⁇ tv2x, yn + tv1y ⁇ tv2y).
  • the image generation unit 14 sets the pixel value of the pixel to D2 (xm + tv1x ⁇ tv3x, yn + tv1y ⁇ tv3y).
  • the image generation unit 14 calculates the positions of the white circles of the image P (t, v) from the pixel values Dp (xm + tv1x, yn + tv1y) of the calculated positions of the black triangles of the image P (t, v).
  • a pixel value Dp (xm, yn) of (xm, yn) is calculated by interpolation.
  • each interpolation method is not particularly limited.
  • the image generation unit 14 uses an average interpolation method or a nearest neighbor interpolation method to interpolate an integrated vector or a pixel value. I do.
  • the image generation unit 14 determines the pixel values of the pixels around the position (xm, yn).
  • An average may be obtained as Dp (xm, yn), or a pixel value of a pixel around the position (xm, yn) (for example, the nearest pixel) may be copied as Dp (xm, yn).
  • FIG. 15 is a flowchart showing processing steps of the image display apparatus 100 according to the present embodiment. Hereinafter, the processing of the image display apparatus 100 will be described with reference to FIG.
  • the image display device 100 refers to the image P (T1, V1) of the viewpoint V1 at time T1 and the image P (T2, V1) of the viewpoint V1 at time T2, for each pixel.
  • the first motion vector MV1 from the image P (T1, V1) of the viewpoint V1 at the time T1 to the image P (T2, V1) of the viewpoint V1 at the time T2 is calculated. This operation is executed by the motion estimation unit 11.
  • the image display device 100 refers to the image P (T1, V1) of the viewpoint V1 at time T1 and the image P (T1, V2) of the viewpoint V2 at time T1, for each pixel.
  • a first disparity vector DV1 from the image P (T1, V1) at the viewpoint V1 at the time T1 to the image P (T1, V2) at the viewpoint V2 at the time T1 is calculated. This operation is executed by the parallax estimation unit 12.
  • the image display apparatus 100 integrates the first motion vector MV1 and the first parallax vector DV1 from the first motion vector MV1 and the first parallax vector DV1. Is calculated. This operation is executed by the integration unit 13.
  • the image display device 100 calculates at least one of the image P (T1, V1), the image P (T2, V1), and the image P (T1, V2) and the integration step S3.
  • An image P (t, v) at an arbitrary viewpoint v at an arbitrary time t is generated from the integrated vector thus obtained. This operation is executed by the image generation unit 14.
  • the image display device 100 uses the first motion vector MV1 and the first disparity vector DV1 as an integrated vector from the image P (T2, V1) at the viewpoint V1 at time T2.
  • a second integrated vector TV2 to the position of the viewpoint v at time t and a third integrated vector TV3 to the position of the viewpoint v at time t from the image P (T1, V2) of the viewpoint V2 at time T1 are calculated. .
  • the image display device 100 includes the image P (T2, V1) of the viewpoint V1 at time T2, the image P (T1, V2) of the viewpoint V2 at time T1, the second integrated vector TV2, and From the third integrated vector TV3, an image P (t, v) of the viewpoint v is generated at time t.
  • the image processing unit 10 may generate the image P (t, v) by another combination of the input image and the integrated vector.
  • the image processing unit 10 calculates the first integrated vector TV1 and the second integrated vector TV2 as integrated vectors, and the image P (T1, V1), the image P (T2, V1), and the first integrated vector.
  • the image P (t, v) may be generated from the vector TV1 and the second integrated vector TV2.
  • the first integrated vector TV1 is used instead of the third integrated vector TV3
  • the image P (T1, V1) is used instead of the image P (T1, V2).
  • the image processing unit 10 calculates the first integrated vector TV1 and the third integrated vector TV3 as the integrated vectors, and the image P (T1, V1), the image P (T1, V2), the first The image P (t, v) may be generated from the integrated vector TV1 and the third integrated vector TV3.
  • the first integrated vector TV1 is used instead of the second integrated vector TV2
  • the image P (T1, V1) is used instead of the image P (T2, V1).
  • the image processing unit 10 uses one integrated vector of the integrated vectors TV1 to TV3 to select one of the input images P (T1, V1), P (T2, V1), and P (T1, V2).
  • the output image P (t, v) may be generated from one image.
  • the image processing unit 10 may calculate the first integrated vector TV1 as the integrated vector, and generate the image P (t, v) from the image P (T1, V1) and the first integrated vector TV1. .
  • the image processing unit 10 determines the position ( The pixel value Dp (x, y) of x, y) may be calculated by the following equation (13).
  • Dp (x, y) D (x ⁇ tv1x, y ⁇ tv1y) (13)
  • the image processing unit 10 may generate each output pixel P (t, v) by moving each pixel of the image P (T1, V1) according to the first integrated vector TV1 corresponding to the pixel. For example, referring to FIG. 14, the pixel at each position (xm, yn) of the image P (T1, V1) represented by a white circle is represented by a black triangle according to the corresponding first integrated vector TV1. Move to position. Then, an output image P (t, v) is generated based on the pixel group at the position of the black triangle mark.
  • the pixel value D (xm, yn) at each position (xm, yn) of the image P (T1, V1) is used as the pixel value at each position (xm + tv1x, yn + tv1y) of the output image P (t, v).
  • Dp (xm + tv1x, yn + tv1y) the pixel value Dp (xm, yn) at each position (xm, yn) of the output image P (t, v) is obtained by interpolation from the obtained pixel value Dp (xm + tv1x, yn + tv1y).
  • the configuration is not limited to the configuration for processing for each pixel.
  • the image processing unit 10 may process each block including a plurality of pixels, or may process each specific object (for example, a three-dimensional object or a moving object) in the image.
  • the image processing unit 10 generates an image at an arbitrary time and viewpoint from a plurality of input images, and its usage is not particularly limited.
  • frame frequency conversion is performed in order to make the movement of a display object smooth, and this is used when changing the viewpoint in order to adjust the pop-out amount.
  • a control unit determines a time t from the frame frequency of the input image and the target frame frequency (that is, the frame frequency of the output image), and the time t is subjected to image processing. To the unit 10.
  • control unit determines viewpoints vl and vr corresponding to the specified pop-out amount based on the adjustment value of the pop-out amount specified by the user, and outputs the viewpoints vl and vr to the image processing unit 10.
  • the image processing unit 10 generates an output image (t, vl) and an output image (t, vr) based on the time t and the viewpoints vl, vr from the control unit, and outputs them to the image display unit 20.
  • the image display unit 20 displays an image (t, vl) and an image (t, vr). As a result, an image with the frame frequency converted and the pop-out amount adjusted is displayed.
  • the image processing unit 10 may be used when generating a multi-view video from a 2-view video for a multi-view display device and converting the frame frequency.
  • the multi-view display device is specifically a device that displays images of three or more viewpoints, for example, a so-called multi-view stereoscopic display.
  • the image processing unit receives the input images P (T1, V1), P (T2, V1), and P (T1, V2), and receives the image P (T1, V1).
  • a first motion vector from the image P (T1, V1) to the image P (T1, V2) is calculated
  • a first disparity vector from the image P (T1, V1) to the image P (T1, V2) is calculated
  • the first motion vector And the first disparity vector an integrated vector in which the first motion vector and the first disparity vector are integrated is calculated, and at any time t from at least one of the input images and the integrated vector.
  • An image P (t, v) at an arbitrary viewpoint v is generated.
  • an image of an arbitrary viewpoint at an arbitrary time can be generated from a plurality of input images. Further, according to the present embodiment, an image of an arbitrary viewpoint at an arbitrary time can be generated from a plurality of input images with a small amount of calculation or a small circuit scale. Specifically, in the present embodiment, an image at an arbitrary time and viewpoint can be generated by one motion estimation process and one parallax estimation process, thereby reducing the amount of calculation, and the electronic circuit The circuit scale can be reduced.
  • the image processing unit calculates two integrated vectors, and generates an output image by averaging the two input images using the two integrated vectors. According to this aspect, it is possible to obtain a better output image as compared with the configuration using one integrated vector. For example, in a configuration using one integrated vector, when motion and parallax are estimated in units of blocks, even if there are two types of motion in one block, only one motion is estimated. For this reason, an erroneous motion is estimated for some of the pixels in the block. On the other hand, according to the configuration in which the output image is generated by averaging the two input images using the two integrated vectors, the adverse effect on the output image due to the erroneous motion estimation is not conspicuous by averaging. Can be.
  • a hold-type display such as a liquid crystal display has a problem that the edge portion looks blurred because the same image is continuously displayed for one frame period.
  • This problem can be solved by a frame frequency conversion technique that increases the number of display images and smoothes the movement of the display object.
  • a motion estimation technique for estimating temporal movement of an object from images P (T1) and P (T2) obtained by imaging at times T1 and T2 is used.
  • the number of display images is increased by generating and displaying the interpolated image P (t) at a desired time t between the two frames of the images P (T1) and P (T2).
  • an intermediate image P (t, V1) is generated from the image P (T1, V1) and the image P (T2, V1) using a motion estimation technique
  • An intermediate image P (t, V2) is generated from the (T1, V2) and the image P (T2, V2) using a motion estimation technique.
  • an output image P (t, v) is generated from the intermediate image P (t, V1) and the intermediate image P (t, V2) using a parallax estimation technique.
  • the parallax estimation process and the motion estimation process require a large amount of calculation.
  • the image processing method according to the comparative example requires two motion estimation processes and one parallax estimation process, so the calculation amount is enormous. Therefore, the circuit scale becomes enormous in the electronic circuit.
  • an output image can be generated by one parallax estimation process and one motion estimation process, and the amount of calculation can be reduced as compared with the image processing method according to the comparative example.
  • the circuit scale can be reduced.
  • the image display device or the image processing unit according to the embodiment described above may be realized only by hardware resources such as an electronic circuit, or may be realized by cooperation of hardware resources and software.
  • the image display device or the image processing unit is realized, for example, by a computer program being executed by a computer, and more specifically, ROM (Read Only Memory). This is realized by reading a computer program recorded on a recording medium such as a main storage device and executing it by a central processing unit (CPU).
  • the computer program may be provided by being recorded on a computer-readable recording medium such as an optical disk, or may be provided via a communication line such as the Internet.
  • the present invention is not limited to the above-described embodiment, and can be implemented in various modes without departing from the gist of the present invention.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
  • Image Analysis (AREA)

Abstract

L'invention concerne la génération d'une image d'un point de vue souhaité à un moment souhaité à partir d'une pluralité d'images d'entrée. Une unité de traitement d'images (10) comprend : une unité d'estimation de mouvement (11) pour recevoir une image (P; T1, V1) à partir d'un point de vue (V1) à un moment (T1), et une image (P; T2, V1) à partir d'un point de vue (V1) à un moment (T2), et calculer un premier vecteur de mouvement de l'image (P; T1, V1) à l'image (P; T2, V1); une unité d'estimation de parallaxe (12) pour recevoir une image (P; T1, V1) et une image (P; T1, V2) d'un point de vue (V2) au moment (T1), et calculer un premier vecteur de parallaxe de l'image (P; T1, V1) à l'image (P; T1, V2); une unité d'intégration (13) pour utiliser le premier vecteur de mouvement et le premier vecteur de parallaxe, et calculer un vecteur intégré à partir de ceux-ci dans lequel le premier vecteur de mouvement et le premier vecteur de parallaxe sont intégrés; et une unité de génération d'images (14) pour générer une image (P; t, v) d'un point de vue souhaité (v) à un moment souhaité (t) en fonction du vecteur intégré et d'au moins une des images ((P; T1, V1); (P; T2, V1); (P; T1, V2)).
PCT/JP2012/078157 2011-11-17 2012-10-31 Dispositif de traitement d'images, procédé de traitement d'images, programme de traitement d'images, dispositif d'affichage d'images et procédé d'affichage d'images WO2013073369A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2013544205A JP5627798B2 (ja) 2011-11-17 2012-10-31 画像処理装置、画像処理方法、画像処理プログラム、画像表示装置、および画像表示方法

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2011251364 2011-11-17
JP2011-251364 2011-11-17

Publications (1)

Publication Number Publication Date
WO2013073369A1 true WO2013073369A1 (fr) 2013-05-23

Family

ID=48429437

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2012/078157 WO2013073369A1 (fr) 2011-11-17 2012-10-31 Dispositif de traitement d'images, procédé de traitement d'images, programme de traitement d'images, dispositif d'affichage d'images et procédé d'affichage d'images

Country Status (2)

Country Link
JP (1) JP5627798B2 (fr)
WO (1) WO2013073369A1 (fr)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH10191393A (ja) * 1996-12-24 1998-07-21 Sharp Corp 多視点画像符号化装置
JP2009003507A (ja) * 2007-06-19 2009-01-08 Victor Co Of Japan Ltd 画像処理方法、画像処理装置及び画像処理プログラム
JP2011223493A (ja) * 2010-04-14 2011-11-04 Canon Inc 画像処理装置および画像処理方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH10191393A (ja) * 1996-12-24 1998-07-21 Sharp Corp 多視点画像符号化装置
JP2009003507A (ja) * 2007-06-19 2009-01-08 Victor Co Of Japan Ltd 画像処理方法、画像処理装置及び画像処理プログラム
JP2011223493A (ja) * 2010-04-14 2011-11-04 Canon Inc 画像処理装置および画像処理方法

Also Published As

Publication number Publication date
JP5627798B2 (ja) 2014-11-19
JPWO2013073369A1 (ja) 2015-04-02

Similar Documents

Publication Publication Date Title
US9270970B2 (en) Device apparatus and method for 3D image interpolation based on a degree of similarity between a motion vector and a range motion vector
KR100720722B1 (ko) 중간영상 생성방법 및 이 방법이 적용되는 입체영상디스플레이장치
US8908011B2 (en) Three-dimensional video creating device and three-dimensional video creating method
JP5287702B2 (ja) 画像処理装置および方法、並びにプログラム
US8803947B2 (en) Apparatus and method for generating extrapolated view
JP5387905B2 (ja) 画像処理装置および方法、並びにプログラム
JP5879713B2 (ja) 画像処理装置、および画像処理方法、並びにプログラム
KR20080076628A (ko) 영상의 입체감 향상을 위한 입체영상 표시장치 및 그 방법
CN109191506B (zh) 深度图的处理方法、系统及计算机可读存储介质
JP5755571B2 (ja) 仮想視点画像生成装置、仮想視点画像生成方法、制御プログラム、記録媒体、および立体表示装置
US20120056871A1 (en) Three-dimensional imaging system and method
JP2012204852A (ja) 画像処理装置および方法、並びにプログラム
JP5521608B2 (ja) 画像処理装置、および画像処理方法、並びにプログラム
JP6033625B2 (ja) 多視点画像生成装置、画像生成方法、表示装置、プログラム、及び、記録媒体
JP5304758B2 (ja) 多視点画像作成装置及び多視点画像作成方法並びに多視点画像表示システム
JP5627498B2 (ja) 立体画像生成装置及び方法
US9113140B2 (en) Stereoscopic image processing device and method for generating interpolated frame with parallax and motion vector
JP5627798B2 (ja) 画像処理装置、画像処理方法、画像処理プログラム、画像表示装置、および画像表示方法
JP5488212B2 (ja) 画像処理装置、画像処理方法および画像表示装置
US9330487B2 (en) Apparatus and method for processing 3D images through adjustment of depth and viewing angle
JP5838775B2 (ja) 画像処理方法、画像処理システムおよび画像処理プログラム
JP2011188108A (ja) 画像処理装置、立体画像表示装置および画像処理方法
KR101907127B1 (ko) 입체 비디오 주밍 및 비디오 내의 전경 및 배경 검출
JP5431393B2 (ja) 立体画像生成装置及び方法
JP2013214916A (ja) 画像処理装置、画像処理方法及び画像表示装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12849618

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2013544205

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12849618

Country of ref document: EP

Kind code of ref document: A1