JP6242104B2 - Image processing apparatus, image processing method and program - Google Patents

Description

  In particular, the present invention relates to an image processing apparatus, an image processing method, and a program that are suitably used to process a stereoscopic video.

  In recent years, with advances in digital technology, equipment capable of capturing and editing stereoscopic video is becoming widespread. For example, three-dimensional video photography, which conventionally required a large-scale device, has become possible with a hand-held camcorder in recent years. Thus, with the widespread use of devices that can easily capture 3D images, the number of 3D images is also rapidly increasing.

  Most of the current stereoscopic images are images that make the viewer perceive a stereoscopic effect using parallax. Here, parallax is the difference between an image viewed with the right eye and an image viewed with the left eye. According to this principle, if the image for the right eye and the image for the left eye have parallax, the brain processes the difference between the respective images, and the image seen by the retina of the right eye and the retina of the left eye The three-dimensional effect is perceived by fusing it with the The device for displaying stereoscopic images separates the image of the right eye and the image of the left eye spatially so that the image for the right eye is viewed with the right eye and the image for the left eye is viewed with the left eye Or the image of the right eye and the image of the left eye are separated and displayed temporally.

  However, if there is a difference (vertical parallax) in the vertical position between the image viewed with the right eye and the image viewed with the left eye for the same object, the brain displays the image for the right eye and the left It is difficult to fuse with the image for the eye. Therefore, the stereoscopic video in which the vertical parallax is present gives the viewer a sense of discomfort or fatigue.

  Therefore, techniques for reducing the vertical parallax have been proposed. In the technology described in Patent Document 1, first, main objects included in the image for the left eye and the image for the right eye are specified. Then, the position of at least one of the main objects in the image for the left eye or in the image for the right eye is changed to reduce the vertical parallax so that the difference between the positions of the main objects becomes a predetermined value. . Further, in the technique described in Patent Document 2, in the left and right images, the difference in the vertical direction of the feature points of the object is calculated, and the image for the right eye is vertically translated by the difference to perform vertical parallax. To reduce. Furthermore, in the technology described in Patent Document 3, trimming is performed on the left-eye video and the right-eye video after applying affine transformation to reduce vertical parallax.

JP 2011-81630 A Patent No. 4763827 gazette Patent No. 4225768

  The technique described in Patent Document 1 does not reduce the vertical parallax when the vertical parallax is less than a predetermined value regardless of the projection amount. However, if the amount of pop-out is large, the influence on three-dimensional sense fusion in the brain is also large, and even if the vertical parallax is small, the influence on three-dimensional sense fusion is large. Do. Furthermore, when the allowable amount of vertical parallax is reduced, the number of objects to be processed increases, and the processing load also increases.

  Further, in the technology described in Patent Document 2, since the vertical parallax is reduced using only the image for the right eye, the target of the image for the right eye is largely moved, and the image becomes unnatural. There is a problem that causes discomfort. Furthermore, in the technique described in Patent Document 3, there is a problem that the angle of view changes to cause a sense of discomfort because trimming is performed.

  SUMMARY OF THE INVENTION In view of the above-described problems, the present invention has an object of generating a stereoscopic image in which discomfort and fatigue are reduced by preventing the processing load from increasing.

The image processing apparatus according to the present invention includes: a determining means for determining the positional information of the object that has issued the right eye image and the left eye image or et extracted, the first in the right eye image of the object extracted from the right eye image The difference in the vertical direction between the position and the position of the second object in the left-eye image which is the second object extracted from the left-eye image and corresponds to the first object is the difference of the first object The first difference is smaller than a threshold determined based on a difference between a position in the right-eye image and a position in the left-eye image of the second object in the horizontal direction. And image processing means for changing the position of at least one of the object and the second object .

  According to the present invention, it is possible to generate a stereoscopic image with reduced discomfort and fatigue, without increasing the processing load.

It is a block diagram showing an example of functional composition of an image processing device concerning a 1st embodiment of the present invention. It is a figure for demonstrating the vertical parallax in a right-eye image and a left-eye image. It is a figure for demonstrating the procedure which extracts positional information from a target object. It is a figure which shows an example of the perpendicular | vertical parallax in one target object. It is a figure for demonstrating the procedure which moves an object so that vertical parallax may be eliminated. In the first embodiment of the present invention, it is a flowchart showing an example of a processing procedure for moving the object to eliminate the vertical parallax. It is a block diagram showing an example of functional composition of an image processing device concerning a 2nd embodiment of the present invention. In the 2nd embodiment of the present invention, it is a flow chart which shows an example of the processing procedure which moves a subject so that vertical parallax may be eliminated.

Hereinafter, an embodiment of the present invention will be described with reference to the attached drawings.
First Embodiment
FIG. 1 is a block diagram showing an example of the functional configuration of an image processing apparatus 10 according to the present embodiment.
In FIG. 1, the object extraction unit 11 extracts an object from the input stereoscopic video, and outputs the position information to the video processing unit 12. The image processing unit 12 moves the position of the object in the stereoscopic image in order to reduce the vertical parallax of the extracted object.

  Here, the style of the stereoscopic video to be input may be any mode as long as the corresponding right-eye video and left-eye video can be simultaneously input. For example, when using a mode in which there is a time difference between the corresponding right-eye video and left-eye video, as in the frame sequential system, a frame buffer is installed at the front stage. Then, after the corresponding right-eye video and left-eye video are synchronized to this frame buffer, a stereoscopic video is output to the object extraction unit 11 and the video processing unit 12.

  Next, the vertical parallax will be described with reference to FIGS. 2 (a) and 2 (b). In FIG. 2, two trees 22 and 23 and a human 24 exist in the right-eye image 20 and the left-eye image 21 respectively, and the right-eye image 20 and the left-eye image 21 are the same target. But the vertical position is different. For example, when the lower end of the trunk of the tree 22 at the left end is compared between the right eye image 20 and the left eye image 21, the position is different by the difference d 1. Similarly, when the position of the waist is compared between the right-eye image 20 and the left-eye image 21, the central human 24 is different in position by a difference d2. Furthermore, in the tree 23 at the right end, the position of the lower end of the trunk is different between the right-eye image 20 and the left-eye image 21 by the difference d3. Thus, the difference between the vertical positions of the same object is referred to as vertical parallax and described below.

  There are a plurality of known methods for extracting an object from a video, but in the present embodiment, the background subtraction method is used. In the present embodiment, it is only necessary to identify the contour of the object, and further, the range of the luminance of the stereoscopic video is unknown, and it is necessary to apply a method having resistance to changes in luminance. Apply. The object may be extracted from the stereoscopic video by another method.

  The object extraction unit 11 calculates position coordinates of an object to generate position information. In the present embodiment, position coordinates of the object are defined as intersection coordinates of a diagonal of a rectangle circumscribing the object. The position coordinates may be defined by another method.

  The position information generated and output by the object extraction unit 11 includes at least an identifier for identifying the object and position coordinates of the object. Here, the identifier may be in any form as long as it can uniquely identify the object. In the present embodiment, a positive integer starting from 0 is used.

  Subsequently, the operation of the object extraction unit 11 will be described in detail with reference to FIG. First, an object is extracted from each of the right-eye image 20 and the left-eye image 21, and a unique identifier is added to the extracted object. For example, as shown in FIG. 3B, the object extraction unit 11 sets two trees present in the right eye image 20 and the left eye image 21 shown in FIG. Extract.

  Next, a rectangle circumscribing the extracted object is calculated. At this time, when the extracted object is in contact with the frame line of the screen, the circumscribed rectangle is set to the outside of the screen frame. For example, as shown in FIG. 3C, the object extraction unit 11 calculates a rectangle 32 circumscribed in contact with the tree at the left end which is the object 31.

  Next, with respect to the circumscribed rectangle, the intersection point of the diagonal line is calculated to be position coordinates. As shown in FIG. 3D, the intersection of the diagonals of the circumscribed rectangle 32 is determined, and the coordinates thereof are set as position coordinates 33.

  Next, position information is generated from the identifier added to the object and the vertical coordinate value of the position coordinate, and is output to the video processing unit 12. As shown in FIG. 3E, position information is generated and output for all the objects 31.

  Next, the relationship between the position coordinates calculated by the object extraction unit 11 and the vertical parallax will be described with reference to FIGS. 4 (a) and 4 (b). In the present embodiment, as shown in FIG. 4B, the coordinate value in the vertical direction of the position coordinate 33 calculated based on the right eye image 20 and the vertical direction of the position coordinate 33 calculated based on the left eye image 21 The difference from the coordinate value is taken, and the absolute value is taken as the vertical disparity DG (n). Here, n is an identifier for identifying the object 31, and the vertical parallax DG (n) is calculated for each object 31. The coordinate values in the vertical direction of the object 31 and the identifier for identifying the object 31 are both present in the position information.

  Subsequently, an operation in which the video processing unit 12 moves the object and reduces the vertical parallax of the object will be described with reference to FIG. Here, a tree existing at the left end of the screen will be described as an example of the object 31. Also, let the identifier attached to the leftmost tree be '0'.

  As shown in FIG. 5A, the position coordinates 33 of the tree at the left end are different in position between the right-eye image 20 and the left-eye image 21. The tree at the left end of the right-eye image 20 is positioned in the lower direction of the screen relative to the left-eye image 21. The vertical parallax, which is the difference between the coordinate values in the vertical direction of each position coordinate 33, is DG (0).

  In order to reduce the vertical parallax DG (0), the video processing unit 12 moves the tree at the left end on the right-eye video 20 upward as shown in FIG. 5B, and conversely, on the left-eye video 21. Move the tree at the left end of the down. Movement directions 43 shown in FIG. 5B indicate the movement directions of the trees at the left end. Here, when the movement amounts of the tree at the left end on the right-eye image 20 and the tree at the left end on the left-eye image 21 are the same, as shown in FIG. 5C, the vertical parallax DG (0) The vertical parallax DG (0) becomes 0 by moving by 1/2 of.

  FIG. 5D shows a state in which movement has been completed for two trees and humans present in the right-eye image 20 and the left-eye image 21. As shown in FIG. 5D, in the case of two trees and humans, the difference between the coordinate values in the vertical direction of the position coordinates is 0, and the vertical parallax is eliminated.

Subsequently, the detailed operation of the video processing unit 12 will be described with reference to the flowchart of FIG.
FIG. 6 is a flowchart showing an example of a detailed processing procedure by the video processing unit 12. Hereinafter, in the description of FIG. 6, an identifier for identifying an object is n, and the number of objects present in a frame of a target image is m. In the present embodiment, since a positive integer starting from 0 is used as the identifier n for identifying the object, the range of the identifier n is 0 ≦ n ≦ m−1. Further, among position coordinates of the object of the identifier n, coordinate values in the vertical direction (y axis) are defined as follows.
Coordinate values in the vertical direction (y-axis) on the right-eye image 20: Gry (n)
Coordinate values in the vertical direction (y-axis) on the left-eye image 21: Gly (n)
I assume. The coordinate values Gry (n) and Gly (n) are calculated by the object extraction unit 11 and included in the position information.

  First, in step S601 in FIG. 6, 0 is substituted as the start value of the identifier n. Then, each operation from step S602 to step S610 is repeated until the identifier n becomes m-1 which is the end value.

In step S602, the horizontal parallax Py (n) of the object with the identifier n is calculated from the position coordinates of the right-eye video 20 and the left-eye video 21. Then, in step S603, the vertical parallax DG (n) of the object of the identifier nn is calculated by the following equation.
DG (n) = | Gly (n) -Gry (n) |
Here, the operator '||' is an operator that calculates an absolute value.

  Next, in step S604, the vertical parallax DG (n) and the allowable amount of vertical parallax are compared. Here, the allowable amount of vertical parallax depends on the horizontal parallax Py (n). As a result of this comparison, when the function for obtaining the allowable amount of vertical parallax is f (Py (n)), when the vertical parallax DG (n) is less than the function f (Py (n)), the vertical parallax DG Do not reduce (n). Therefore, the process for reducing the vertical parallax DG (n) is not performed, and the process is started from step S602 for the next identifier (n + 1). On the other hand, if the vertical parallax DG (n) is equal to or larger than the allowable amount (the function f (Py (n)) or more) as a result of the comparison in step S604, the vertical parallax DG (n) included in the object will be described later. It reduces by each processing of S605 to step S610.

  The allowable amount of vertical parallax with respect to horizontal parallax depends on the display device, the installation environment of the display device, or the image processing method of the display device, and is not unique. Therefore, the vertical parallax is changed in advance for each of a plurality of horizontal parallaxes, and the vertical parallax which is a limit at which stereoscopic fusion can not be performed when the vertical parallax is further increased is obtained. Then, a value obtained by multiplying the limit vertical parallax by a coefficient of 1 or less is taken as the allowable amount of vertical parallax. Thus, a function f is defined that represents the relationship between horizontal parallax and vertical parallax tolerance. According to this function, the larger the horizontal parallax, the smaller the allowable amount of vertical parallax.

  In the present embodiment, the allowable amount of vertical parallax with respect to horizontal parallax is defined by a function, but the method of definition is not limited to a function. For example, the comparison table may be used to compare in step S604.

  In step S605, it is determined whether the object of the identifier n is in contact with the upper side or the lower side of the screen. When the object is in contact with the upper side or the lower side of the screen, the shape may be interrupted at the upper side or the lower side of the screen, and the shape of the object may be different between the right eye image 20 and the left eye image 21. At this time, when moving and aligning the position in the vertical direction, the shape of the object is different between the right-eye image 20 and the left-eye image 21, so that stereoscopic fusion is difficult and confusion in the brain is caused. Therefore, the operation to reduce the vertical parallax is changed depending on whether the object is in contact with the upper side or the lower side of the screen.

  First, as a result of the determination in step S605, if neither the upper side nor the lower side of the screen is in contact with the right eye image 20 and the left eye image 21 for the object of the identifier n, the process proceeds to step S606. On the other hand, although the object of the identifier n on the right-eye image 20 is not in contact with the upper side or the lower side of the screen, when the object of the identifier n in the left-eye image 21 is in contact with the upper side or the lower side of the screen It progresses to step S607.

  Furthermore, the object of the identifier n on the right-eye image 20 is in contact with the upper side or the lower side of the screen, but the object of the identifier n on the left-eye image 21 is not in contact with the upper side or the lower side of the screen Proceed to step S608. When both the right-eye image 20 and the left-eye image 21 touch the upper side or the lower side of the screen with respect to the object of the identifier n, the process of reducing the vertical parallax DG (n) is not performed. Therefore, the process starts from step S602 for the next identifier (n + 1).

  In step S606, a movement vector is calculated in the case where neither the upper side nor the lower side of the screen is in contact with the right-eye image 20 and the left-eye image 21 for the object of the identifier n. In this case, the objects of both the right-eye image 20 and the left-eye image 21 are moved by the same distance, and the movement destination is the midpoint of the vertical position. The movement vector of the object on the right-eye image 20 calculated in step S606 is {0, (Gly (n) -Gry (n)) / 2}. On the other hand, the movement vector of the object on the left eye image 21 is also {0, (Gry (n) -Gly (n)) / 2}.

  In step S607, although the object of the identifier n on the right eye image 20 is not in contact with the upper side or the lower side of the screen, the object of the identifier n in the left eye image 21 is in contact with the upper side or the lower side of the screen Calculate the movement vector in the case of In this case, only the object on the right-eye image 20 is moved without moving the object on the left-eye image 21, and the movement destination is the position of the object on the left-eye image 21. The movement vector of the object on the right eye image 20 calculated in step S607 is {0, Gly (n) -Gry (n)}. Note that the movement vector of the object on the left eye image 21 is 0.

  In step S608, the object of identifier n on the right-eye image 20 is in contact with the upper or lower side of the screen, but the object of identifier n in the left-eye image 21 is in contact with the upper or lower side of the screen Calculate the movement vector when there is not. In this case, only the object on the left eye image 21 is moved without moving the object on the right eye image 20, and the movement destination is the position of the object on the right eye image 20. The movement vector of the object on the left-eye image 21 calculated in step S608 is {0, Gry (n) -Gly (n)}. Note that the movement vector of the object on the right-eye image 20 is zero.

  In step S609, the object of the identifier n is cut out from the right eye image 20 or the left eye image 21, and the cut out object is moved. At this time, the object is moved based on the movement vector calculated in step S606, step S607, or step S608. In the present embodiment, any known method may be used as a method of cutting out an object and a method of moving it. Then, in step S610, the moving source area of the target object extracted and moved in step S609 is filled with the image of the peripheral area. As described above, the process of FIG. 6 is performed for each frame of the stereoscopic video.

  As described above, according to the present embodiment, a function f representing the relationship between the horizontal parallax and the allowable amount of vertical parallax is defined, and when the vertical parallax is less than the allowable amount given by the function f, the object Did not move the As a result, the criteria for reducing the vertical parallax are different between the object with a large pop-out amount and the other object, so that the processing load does not increase, and discomfort and fatigue can be further reduced.

Second Embodiment
Hereinafter, a second embodiment according to the present invention will be described.
FIG. 7 is a block diagram showing an example of a functional configuration of the image processing apparatus 70 according to the present embodiment. The image processing apparatus 70 according to the present embodiment at least includes an object extraction unit 71, a video processing unit 72, and a motion vector calculation unit 73. In addition, since the target object extraction part 71 is the same as the target object extraction part 11 which concerns on 1st Embodiment, description is abbreviate | omitted.

  The image processing unit 72 moves the object to reduce the vertical parallax of the object. In this embodiment, unlike the first embodiment, not only position information but also motion vector information generated by the motion vector calculation unit 73 is used to reduce the vertical parallax of the object.

  The motion vector calculation unit 73 calculates the motion vector MV (n) of the object based on the input three-dimensional video and the position information obtained by the object extraction unit 71, and the motion vector information is processed by the image processing unit 72. Output to Here, n is an identifier for identifying an object.

Next, the operation of the video processing unit 72 will be described in detail with reference to the flowchart of FIG.
FIG. 8 is a flowchart showing an example of a detailed processing procedure by the video processing unit 72. In addition, since the definition of each variable is the same as that of FIG. 6, description is abbreviate | omitted.

  In FIG. 8, the contents of the processing from step S601 to step S610 are the same as those in FIG. When the movement vector is calculated in step S606, it is next determined in step S801 whether or not there is a scene switch between the immediately preceding frame and the frame. As a result of this determination, when there is a scene switch, there is no correlation between the immediately preceding frame and the frame, so the movement vector calculated in step S606 can be used as it is. Therefore, in this case, the process proceeds to step S609. On the other hand, as a result of the determination in step S801, if there is no scene switching, there is a correlation between the immediately preceding frame and the frame, so the validity of the movement vector calculated in step S606 is determined. A need arises. Therefore, the validity of the movement vector is determined in steps S802 and S803.

First, in step S802, it is assumed that the object has moved, and the vertical component MV ′ y (n) of the motion vector MV ′ (n) of the object of the identifier n between the immediately preceding frame and the frame is Calculate). Here, let Gk (n) be the position coordinate of the object in the immediately preceding frame after correction to reduce the vertical parallax, and Gky (n) be the vertical component of the position coordinate Gk (n). I assume. Further, position coordinates of an object assumed to have moved in the frame are set to Gk + 1 (n). In this case, the vertical component of the position coordinates Gk + 1 (n) is {(Gry (n) + Gly (n)) / 2}. Therefore, the vertical component MV'y (n) of the motion vector MV '(n) is calculated as follows.
MV'y (n) = (Gry (n) + Gly (n)) / 2-Gky (n)

Next, in step S803, the sign of the vertical component MV'y (n) of the motion vector MV '(n) calculated in step S802 and the vertical component MVy of the motion vector MV (n) calculated by the motion vector calculation unit 73. Compare with the sign of (n). In step S803, comparison is performed using a logical expression shown below.
(((MVy (n) ≧ 0) &&(MV'y (n) ≧ 0)) ||
((MVy (n) <0) &&(MV'y (n) <0))
Here, the operator '&&' is an operator that performs a logical product, and the operator '||' is an operator that performs a logical sum.

  As a result of this comparison, when the condition of the above logical expression is satisfied, even if the object is moved to reduce the vertical parallax, the motion vector and the moving direction are the same, and the viewer does not feel discomfort . Therefore, since the movement vector calculated in step S606 can be used as it is, the process proceeds to step S609.

  On the other hand, as a result of comparison in step S803, when the condition of the above logical expression is not satisfied, if the object is moved as it is, it interferes with the motion vector, and the viewer may feel discomfort. Therefore, in step S804, resetting of the movement destination and recalculation of the movement vector are performed. Relocation of the moving destination and recalculation of the motion vector are performed based on the sign of the vertical component MVy (n) of the motion vector MV (n) according to the following criteria.

  (1) If (MVy (n) <0) && (Gly (n) <Gry (n)), the movement destination is the position of the object on the left eye image 21. That is, the movement vector in the right-eye image is {0, (Gly (n) -Gry (n)) / 2}, and the movement vector in the left-eye image is {0, 0}.

  (2) When (MVy (n) <0) && (Gly (n) ≧ Gry (n)), the movement destination is the position of the object on the right eye image 20. That is, the movement vector in the right-eye image is {0, 0}, and the movement vector in the left-eye image is {0, (Gry (n) -Gly (n)) / 2}.

  (3) If (MVy (n) ≧ 0) && (Gly (n) <Gry (n)), the movement destination is the position of the object on the right eye image 20. That is, the movement vector in the right-eye image is {0, 0}, and the movement vector in the left-eye image is {0, (Gry (n) -Gly (n)) / 2}.

  (4) If (MVy (n) ≧ 0) && (Gly (n) ≧ Gry (n)), the movement destination is the position of the object on the left eye image 21. That is, the movement vector in the right-eye image is {0, (Gly (n) -Gry (n)) / 2}, and the movement vector in the left-eye image is {0, 0}.

  In step S609, the object of the identifier n is moved using the movement vector recalculated in step S804 as described above.

  As described above, according to the present embodiment, the movement amount and the movement direction of the object are determined by reflecting the motion vector. In this way, it is possible to further reduce the sense of discomfort and the feeling of fatigue with respect to a stereoscopic image including a moving object.

(Other embodiments)
The present invention is also realized by executing the following processing. That is, software (program) for realizing the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU or the like) of the system or apparatus reads the program. It is a process to execute.

11 object extraction unit 12 video processing unit

Claims (7)

A determination unit that determines position information of an object extracted from the right-eye video and the left-eye video;
The position in the right-eye image of the first object extracted from the right-eye image, and the left-eye image of the second object corresponding to the first object, the second object extracted from the left-eye image The difference between the position in the vertical direction and the position in the vertical direction is a threshold value determined based on the difference in the horizontal direction between the position of the first object in the right eye image and the position of the second object in the left eye image Video processing means for changing the position of at least one of the first object and the second object so that the difference in the vertical direction becomes smaller,
An image processing apparatus comprising: The image processing apparatus further comprises calculation means for calculating a motion vector from the right eye image and the left eye image,
The image processing according to claim 1, wherein the image processing means determines a movement vector for reducing the difference in the vertical direction of the object based on the motion vector calculated by the calculation means. apparatus. The determination means is an image processing apparatus according to claim 1 or 2, characterized in that to determine the intersection of diagonal lines of the coordinates in the rectangle circumscribing the extracted object as the position information. The image processing means determines whether the first object extracted from the right-eye image has reached at least one of the upper side and the lower side of the right-eye image, and the first object extracted from the left-eye image The difference in the vertical direction of at least one of the first object and the second object is reduced according to the determination result as to whether or not the two objects reach at least one of the upper side and the lower side of the left eye image. The image processing apparatus according to any one of claims 1 to 3 , wherein an amount of movement of the image processing apparatus is calculated. The image processing means may be configured such that the first object extracted from the right eye image reaches the upper side or the lower side of the right eye image, and the second object extracted from the left eye image is the upper side of the left eye image and If it is determined not to reach the lower side, it claims 1-4, characterized in that to change the position of the second object of the first object and the second object in order to reduce the difference in the vertical direction The image processing apparatus according to any one of the above. A determination step of determining position information of an object extracted from the right-eye video and the left-eye video;
The position in the right-eye image of the first object extracted from the right-eye image, and the left-eye image of the second object corresponding to the first object, the second object extracted from the left-eye image The difference between the position in the vertical direction and the position in the vertical direction is a threshold value determined based on the difference in the horizontal direction between the position of the first object in the right eye image and the position of the second object in the left eye image And the image processing step of changing the position of at least one of the first object and the second object so as to reduce the difference in the vertical direction,
An image processing method comprising: The program for functioning a computer as each means of the image processing apparatus in any one of Claims 1-5 .

Images