JP2013123102A - Image processing apparatus, imaging apparatus, display apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus, an imaging apparatus, a display apparatus and an image processing method which are capable of reducing discomfort of an object expressed by a stereoscopic image configured by an original image, while suppressing loss of depth perception contained in the original image.SOLUTION: An image processing apparatus 300 includes an image processing section 330 to apply parallax correction processing. In the parallax correction processing, with respect to coordinates having different coordinates in a vertical direction, a left-eye viewpoint image and a right-eye viewpoint image are subjected to a different amount of shift a coordinate in a lateral direction. With respect to coordinates having the same coordinate in the vertical direction, a direction in which a coordinate in the lateral direction is shifted in the left-eye viewpoint image is opposite to a direction in which a coordinate in the lateral direction is shifted in the right-eye viewpoint image.

Description

  The present invention relates to an image processing device, an imaging device, a display device, and an image processing method for processing a plurality of viewpoint images constituting a stereoscopic image.

  In an image captured from an oblique direction with respect to a predetermined projection plane such as a horizontal plane or a vertical plane, the object becomes smaller as it is farther from the imaging means, even if the object is actually the same size, due to the effect of perspective. The image is distorted so that

  On the other hand, a technique for correcting image distortion associated with the perspective method has been proposed (for example, Patent Document 1). With such a technique, an image captured from an oblique direction with respect to the predetermined projection plane is corrected to an image captured from the front with respect to the predetermined projection plane (projection perspective correction processing).

  In recent years, a display device that displays a stereoscopic image composed of a plurality of viewpoint images (for example, a left-eye viewpoint image and a right-eye viewpoint image) is known. A technique for applying projection perspective correction processing to such a stereoscopic image has also been proposed (for example, Patent Document 2).

JP-A-2005-107654 JP 2009-2111718

  Here, when the projection perspective correction process is applied to the stereoscopic image, the uncomfortable feeling of the object expressed by the stereoscopic image formed by the image (original image) captured by the imaging unit is reduced. However, the perspective included in the original image is also lost.

  Therefore, the present invention has been made to solve the above-described problem, and while suppressing the loss of perspective included in the original image, the object expressed by the stereoscopic image composed of the original image is uncomfortable. An object is to provide an image processing device, an imaging device, a display device, and an image processing method that can be reduced.

  The image processing apparatus according to the first feature processes a left-eye viewpoint image and a right-eye viewpoint image that form a stereoscopic image. The image processing apparatus includes an image processing unit that applies a parallax correction process that shifts coordinates in the left-right direction in the left-eye viewpoint image and the right-eye viewpoint image. In the parallax correction process, for the coordinates having different vertical coordinates, the amount of shift of the coordinates in the left-right direction is different in the left-eye viewpoint image and the right-eye viewpoint image, and the coordinates having the same vertical coordinates are The direction of shifting the left-right coordinate in the left-eye viewpoint image is the opposite direction to the direction of shifting the left-right coordinate in the right-eye viewpoint image.

  In the first feature, the image processing apparatus includes a specifying unit that specifies an inclination of an imaging reference plane perpendicular to an imaging direction of an imaging unit that captures the stereoscopic image and an inclination of the display reference plane. In the parallax correction process, the image processing unit shifts coordinates in the left-right direction in the left-eye viewpoint image and the right-eye viewpoint image based on the inclination specified by the specifying unit.

  An imaging device according to a second feature includes an imaging unit that captures the plurality of viewpoint images, and an image processing device according to the first feature.

  A display device according to a third feature includes a display unit that displays the stereoscopic image, and an image processing device according to the first feature.

  The image processing method according to the fourth feature is a method of processing a plurality of viewpoint images constituting a stereoscopic image. The image processing method includes a step of applying a parallax correction process for shifting coordinates in the left-right direction in the left-eye viewpoint image and the right-eye viewpoint image. In the parallax correction process, for the coordinates having different vertical coordinates, the amount of shift of the coordinates in the left-right direction is different in the left-eye viewpoint image and the right-eye viewpoint image, and the coordinates having the same vertical coordinates are The direction of shifting the left-right coordinate in the left-eye viewpoint image is the opposite direction to the direction of shifting the left-right coordinate in the right-eye viewpoint image.

  According to the present invention, an image processing device, an imaging device, and an image processing device that can reduce a sense of incongruity of an object expressed by a stereoscopic image constituted by an original image while suppressing loss of perspective included in the original image. A display device and an image processing method can be provided.

FIG. 1 is a diagram for explaining an application scene of the image processing apparatus according to the first embodiment. FIG. 2 is a diagram for explaining an application scene of the image processing apparatus according to the first embodiment. FIG. 3 is a diagram for explaining an application scene of the image processing apparatus according to the first embodiment. FIG. 4 is a diagram for explaining projection perspective correction processing according to a conventional example. FIG. 5 is a diagram for explaining projection perspective correction processing according to a conventional example. FIG. 6 is a diagram for explaining projection perspective correction processing according to a conventional example. FIG. 7 is a diagram for explaining projection perspective correction processing according to a conventional example. FIG. 8 is a diagram for explaining projection perspective correction processing according to a conventional example. FIG. 9 is a block diagram showing the image processing apparatus 300 according to the first embodiment. FIG. 10 is a diagram for explaining the parallax correction processing according to the first embodiment. FIG. 11 is a diagram for explaining the parallax correction processing according to the first embodiment. FIG. 12 is a diagram for explaining the parallax correction processing according to the first embodiment. FIG. 13 is a diagram for explaining the parallax correction processing according to the first embodiment. FIG. 14 is a diagram for explaining the parallax correction processing according to the first modification. FIG. 15 is a diagram for explaining the parallax correction processing according to the first modification. FIG. 16 is a diagram for explaining an application scene of the image processing apparatus according to the second modification. FIG. 17 is a diagram for explaining the provision of perspective according to the second modification. FIG. 18 is a diagram for explaining the provision of perspective according to the second modification. FIG. 19 is a diagram for explaining an application scene of the image processing device according to the third modification.

  Hereinafter, an image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

  However, it should be noted that the drawings are schematic and ratios of dimensions and the like are different from actual ones. Therefore, specific dimensions and the like should be determined in consideration of the following description. Moreover, it is a matter of course that portions having different dimensional relationships and ratios are included between the drawings.

[Outline of Embodiment]
The image processing apparatus according to the embodiment processes a plurality of viewpoint images constituting a stereoscopic image. The image processing apparatus includes an image processing unit that applies a parallax correction process that shifts coordinates in the left-right direction in the left-eye viewpoint image and the right-eye viewpoint image. In the parallax correction process, for the coordinates having different vertical coordinates, the amount of shift of the coordinates in the left-right direction is different in the left-eye viewpoint image and the right-eye viewpoint image, and the coordinates having the same vertical coordinates are The direction of shifting the left-right coordinate in the left-eye viewpoint image is the opposite direction to the direction of shifting the left-right coordinate in the right-eye viewpoint image.

  In the embodiment, in the parallax correction processing, the image processing unit differs in the amount of shifting the coordinate in the left-right direction in the left-eye viewpoint image and the right-eye viewpoint image with respect to the coordinate having different vertical coordinates, and the vertical coordinate is the same. The left-right coordinate is shifted so that the direction of shifting the left-right coordinate in the left-eye viewpoint image is opposite to the direction of shifting the left-right coordinate in the right-eye viewpoint image. As a result, a sense of incongruity of the object represented by the stereoscopic image composed of the original image (that is, a phenomenon in which the object falls down or rises due to the difference between the inclination of the display reference plane and the imaging reference plane in the stereoscopic image, etc.) ) Is reduced. The difference between the tilt of the display reference plane and the tilt of the imaging reference plane can be rephrased as that the display reference plane and the imaging reference plane are not parallel.

  In the embodiment, the image processing unit applies the parallax correction process without applying the perspective correction process. This suppresses the loss of perspective included in the original image.

[First Embodiment]
(Application scene of image processing device)
In the following, application scenes of the image processing apparatus according to the first embodiment will be described. 1 and 2 are diagrams for explaining an application scene of the image processing apparatus according to the first embodiment.

  As illustrated in FIG. 1, the display device 100 includes a display unit that displays a stereoscopic image composed of a plurality of viewpoint images (in the first embodiment, a left-eye viewpoint image and a right-eye viewpoint image) on a display reference plane S. . The display device 100 is, for example, a projection video display device. However, the display device 100 may be a device such as a plasma display or a liquid crystal television. In this case, the display device 100 is installed such that the video display surface overlaps the display reference surface S. The display reference surface S is a vertical surface such as a wall surface, for example. The display reference plane S may be a horizontal plane such as a floor, or may be a slope having an inclination with respect to the vertical plane and the horizontal plane.

  For example, the display reference plane S is parallel to a plane defined by the X1 axis and the Y1 axis that are orthogonal to each other. An axis perpendicular to the X1 axis and the Y1 axis is referred to as a Z1 axis. The X1 axis is an axis along a line connecting both eyes of an observer who observes a stereoscopic image. Here, the direction along the X1 axis may be considered as the “left-right direction”. Further, the direction along the Y1 axis may be considered as the “vertical direction”.

  In contrast, as illustrated in FIG. 2, the imaging device 200 includes an imaging unit that captures a plurality of viewpoint images. For example, the imaging device 200 mainly captures an image of a vertical plane (for example, an object V extending in the vertical direction on the vertical plane).

  In the first embodiment, the imaging apparatus 200 exemplifies a case of capturing a plurality of viewpoint images from a direction inclined with respect to the vertical direction of the display reference plane S on which a stereoscopic image is to be displayed. In other words, the inclination of the imaging reference plane T perpendicular to the imaging direction (optical axis) of the imaging apparatus 200 is different from the inclination of the display reference plane S.

  For example, the imaging reference plane T is parallel to a plane defined by the X0 axis and the Y0 axis that are orthogonal to each other. An axis perpendicular to the X0 axis and the Y0 axis is referred to as a Z0 axis. The X0 axis is an axis along a line connecting the right eye imaging position where the right eye viewpoint image is to be captured and the left eye imaging position where the left eye viewpoint image is to be imaged. The Z axis is an axis along the imaging direction of the imaging device 200. Here, the direction along the X0 axis may be considered as the “left-right direction”. Further, the direction along the Y0 axis may be considered as the “vertical direction”.

  Here, consider a case where a plurality of viewpoint images are captured from a direction inclined with respect to the vertical direction (horizontal direction) of the vertical plane. More precisely, consider a case where imaging is performed in a state where the imaging reference plane T is not parallel to the display reference plane S. In such a case, as shown in FIG. 3, a vertical line segment (object V) in the real world is recorded as an object R in the right-eye viewpoint image. On the other hand, the vertical line segment (object V) is recorded as the object L in the left eye viewpoint image. FIG. 3 is a diagram in which the right eye viewpoint image and the left eye viewpoint image are overlapped. When the right eye viewpoint image is observed with the right eye and the left eye viewpoint image is observed with the left eye, it can be observed as a stereoscopic image. However, since the parallax differs depending on the part of the object V, the upper part of the object V is relatively compared with the lower part. Is recognized on the back side. That is, the object V, which is originally a vertical line segment, is recognized as being collapsed.

(Projection perspective correction processing according to the conventional example)
Hereinafter, the projection / distance correction processing according to the conventional example will be described. FIGS. 4-8 is a figure for demonstrating the projection perspective correction process which concerns on a prior art example.

  As shown in FIG. 4, the inclination of the imaging reference plane T is different from the inclination of the display reference plane S. In other words, the imaging direction (Z0 axis) of the imaging device 200 is inclined with respect to the vertical direction (Z1 axis) of the display reference plane S. In the plane defined by the Y1 axis and the Z1 axis, the inclination of the Z1 axis with respect to the Z0 axis is θ. Accordingly, the coordinates expressed by the X0 axis, the Y0 axis, and the Z0 axis are subjected to projection conversion after being converted into coordinates expressed by the X1 axis, the Y1 axis, and the Z1 axis.

Here, when the imaging direction (optical axis) of the imaging unit that captures the right-eye viewpoint image and the imaging direction (optical axis) of the imaging unit that captures the left-eye viewpoint image, such coordinate conversion is performed using the right eye. This is performed by rotating the viewpoint image and the left-eye viewpoint image about the X0 axis. Specifically, such coordinate conversion can be performed by the following equation.

Subsequently, on the coordinate space expressed by the X1 axis, the Y1 axis, and the Z1 axis, the imaging reference plane T perpendicular to the Z0 axis and the display reference plane perpendicular to the Z1 axis with respect to the imaging position, that is, the viewpoint (origin). Project onto a plane parallel to S. Such projection conversion can be performed by the following equation.

  However, since the surface to be projected only needs to be parallel to the display reference surface S, the value of z2 is an example, and is not limited to such a value.

  Thus, as shown in FIG. 5, the original rectangular image is converted into a distorted image by coordinate conversion and projection conversion (projection perspective correction processing). Subsequently, as shown in FIG. 6, a rectangular image is cut out as an image to be displayed on the display reference plane S.

When the imaging direction (optical axis) of the imaging unit that captures the right-eye viewpoint image and the imaging direction (optical axis) of the imaging unit that captures the left-eye viewpoint image intersect, the coordinate conversion described above is performed using the right-eye viewpoint image and For each of the left eye viewpoint images, the rotation is performed about the X0 axis and the Y0 axis. Specifically, such coordinate conversion can be performed by the following equation.

  Note that θ1 is the inclination of the Z1 axis with respect to the Z0 axis in the plane defined by the Y1 axis and the Z1 axis. θ2 is the inclination of the Z1 axis with respect to the Z0 axis in the plane defined by the X1 axis and the Z1 axis. Also, θ2 is a rotation angle that is relatively opposite between the coordinate conversion of the right-eye viewpoint image and the coordinate conversion of the left-eye viewpoint image.

Subsequently, in the coordinate space expressed by the X1 axis, the Y1 axis, and the Z1 axis, the imaging reference plane T perpendicular to the Z0 axis is parallel to the display reference plane S perpendicular to the Z1 axis with respect to the viewpoint (origin). Project onto a flat surface. Such projection conversion can be performed by the following equation.

  However, since the surface to be projected only needs to be parallel to the display reference surface S, the value of z2 is an example, and is not limited to such a value.

  By such coordinate conversion and projection conversion (projection perspective correction processing), the right eye viewpoint image is converted as shown in FIG. 7, and the left eye viewpoint image is converted as shown in FIG. Subsequently, although not shown, a rectangular image is cut out as an image to be displayed on the display reference plane S.

  As a result of the projection perspective correction process described above, the sense of incongruity of the object represented by the stereoscopic image formed by the image (original image) captured by the imaging device 200 (that is, the inclination of the display reference plane S and the imaging reference in the stereoscopic image) The phenomenon that the object falls down or rises due to the difference from the inclination of the surface T) is reduced. However, since the shape of the object is also corrected, the perspective included in the original image is lost together.

(Image processing device)
Hereinafter, the image processing apparatus according to the first embodiment will be described. FIG. 9 is a diagram illustrating the image processing apparatus 300 according to the first embodiment. As illustrated in FIG. 9, the image processing apparatus 300 includes an acquisition unit 310, a specification unit 320, and an image processing unit 130.

  The image processing device 300 may be provided in the display device 100 or may be provided in the imaging device 200. The image processing apparatus 300 may be provided in an apparatus such as a TV tuner, a DVD player, or a personal computer.

  The acquisition unit 310 acquires input signals constituting a stereoscopic image (left eye viewpoint image and right eye viewpoint image). For example, the acquisition unit 310 acquires an input signal from a device such as a TV tuner, a DVD player, or a personal computer.

  The specifying unit 320 specifies the tilt of the display reference plane S and the tilt of the imaging reference plane T. Eventually, the sum of the inclination of the display reference plane S and the inclination of the imaging reference plane T only needs to be specified. In other words, the specifying unit 320 has Z1 with respect to the Z0 axis in the plane defined by the Y1 axis and the Z1 axis. The tilt of the Z1 axis with respect to the Z0 axis is specified in the plane defined by the tilt of the axis, the X1 axis, and the Z1 axis.

  In the first embodiment, the parameter indicating the inclination of the imaging reference plane T may be detected by a sensor that detects the inclination of the imaging reference plane T and acquired by the acquisition unit 310. A sensor that detects the inclination of the imaging reference plane T is provided in the imaging apparatus 200, for example. Or the parameter which shows the inclination of the imaging reference plane T is provided to the input signal which comprises a stereo image, and may be acquired by the acquisition part 310 with an input signal. Alternatively, the parameter indicating the inclination of the imaging reference plane T may be manually input by the observer via the user interface and acquired by the acquisition unit 310. Alternatively, the parameter indicating the tilt of the imaging reference plane T is manually input by the observer via the user interface, and then given to the input signal that forms the stereoscopic image, and acquired by the acquisition unit 310 together with the input signal. Also good. Alternatively, the parameter indicating the inclination of the imaging reference plane T may be known. This corresponds to a case where imaging conditions are fixed.

  The parameter indicating the inclination of the display reference plane S may be detected by a sensor that detects the inclination of the display reference plane S and acquired by the acquisition unit 310 from the sensor. A sensor for detecting the inclination of the display reference surface S is provided in the display device 100, for example. Alternatively, the parameter indicating the inclination of the display reference plane S may be manually input by the observer via the user interface and acquired by the acquisition unit 310. Alternatively, the parameter indicating the inclination of the display reference plane S may be known. This corresponds to a case where display conditions are fixed. For example, many personal computer monitors can adjust the angle of the display surface, whereas many televisions have a fixed display surface angle.

  The image processing unit 130 applies parallax correction processing that shifts coordinates in the left-right direction in the left-eye viewpoint image and the right-eye viewpoint image. Specifically, in the parallax correction processing, the image processing unit 130 differs in the amount of shifting the coordinate in the left-right direction in the left-eye viewpoint image and the right-eye viewpoint image with respect to the coordinates in which the vertical direction coordinates are different. Are shifted in the left-right direction so that the direction in which the left-right coordinate is shifted in the left-eye viewpoint image is opposite to the direction in which the left-right coordinate is shifted in the right-eye viewpoint image. Apply parallax correction processing. For example, the image processing unit 130 shifts the coordinates in the left-right direction based on the inclination specified by the specifying unit 320 in the parallax correction processing.

  Specifically, the image processing unit 130 specifies the shift amount Δx (y) of the x coordinate using the method illustrated in FIGS. 4 and 5. Subsequently, the image processing unit 130 adds the shift amount Δx (y) to all the x coordinates. That is, the image processing unit 130 converts the coordinates (x, y) into coordinates (x + Δx (y), y).

  For example, as shown in FIG. 10, the shift amount Δx (y) of the center line in the left-right direction is applied to all x coordinates. As a result, as shown in FIG. 11, the rectangular image is converted into a parallelogram image. Subsequently, although not shown, a rectangular image is cut out as an image to be displayed on the display reference plane S.

  Thus, when the horizontal coordinate (x coordinate) is converted, the perspective correction process is not applied, and loss of perspective included in the original image is suppressed.

  For example, by such parallax correction processing, the right eye viewpoint image is converted as shown in FIG. 12, and the left eye viewpoint image is converted as shown in FIG. As a result, a sense of incongruity of the object expressed by the stereoscopic image formed by the image (original image) captured by the imaging device 200 (that is, the inclination of the display reference plane S and the inclination of the imaging reference plane T in the stereoscopic image). The phenomenon that the object falls down or rises due to the difference is reduced. On the other hand, the perspective included in the original image is maintained.

(Function and effect)
In the first embodiment, in the parallax correction process, the image processing unit 130 differs in the amount of shifting the coordinate in the left-right direction in the left-eye viewpoint image and the right-eye viewpoint image for coordinates having different vertical coordinates, For the coordinates that are the same, the left-right coordinate is shifted so that the direction in which the left-right coordinate is shifted in the left-eye viewpoint image is opposite to the direction in which the left-right coordinate is shifted in the right-eye viewpoint image. As a result, a sense of incongruity of the object expressed by the stereoscopic image formed by the original image (that is, in the stereoscopic image, the object falls down or rises due to the difference between the inclination of the display reference plane S and the inclination of the imaging reference plane T). Phenomenon).

  In the first embodiment, the image processing unit 130 applies the parallax correction process without applying the perspective correction process. This suppresses the loss of perspective included in the original image.

[Modification 1]
Hereinafter, Modification Example 1 of the first embodiment will be described. In the following, differences from the first embodiment will be mainly described.

  In 1st Embodiment, it illustrated about the case where the inclination of the imaging reference plane T is known. On the other hand, in the first modification, a case where the inclination of the imaging reference plane T is unknown and the display reference plane S is vertical will be described.

  The first modification utilizes the fact that the right eye viewpoint image and the left eye viewpoint image have different inclinations when corresponding edges are recorded in the image. If a vertical edge in the real world is recorded at the center of the image, the right eye viewpoint image is recorded with the edge tilted to the right as in the case of the vertical line segment in FIG. Also, the left eye viewpoint image is recorded with the edge tilted to the left. In other words, the edge of the right eye viewpoint image is recorded while being inclined to the right relative to the edge of the left eye viewpoint image. The same applies to the case where an edge is recorded outside the center of the image. For example, the uncorrected (right eye viewpoint) image in FIG. 14 and the uncorrected (left eye viewpoint) image in FIG. 15 correspond to edges (R1 and L1, R2 and L2, R3 and L3, R4) of each object. And L4, R5 and L5, and R6 and L6) have different slopes. Then, comparing the slopes of the corresponding edges (R1 and L1, R2 and L2, R3 and L3, R4 and L4, R5 and L5, R6 and L6), the edge of the right eye viewpoint image is the left eye viewpoint. The image is recorded while being tilted to the right relative to the edge of the image. That is, if the average value of the edge angle of the right-eye viewpoint image before correction and the average value of the edge angle of the left-eye viewpoint image before correction are calculated, the average value of the edge angle of the right-eye viewpoint image is the edge of the left-eye viewpoint image. The angle is relatively inclined to the right with respect to the average value. In addition, when adding the angle of an edge in order to calculate an average value, you may weight by the length of an edge.

  Subsequently, a conversion amount is calculated such that the average value of the edge angle of the right-eye viewpoint image is equal to the average value of the edge angle of the left-eye viewpoint image, and the right-eye viewpoint image and the left-eye viewpoint image are converted into parallelograms. Convert to image. For example, when the average value of the edge angle of the right-eye viewpoint image is ψR and the average value of the edge angle of the left-eye viewpoint image is ψL, the conversion amount is (ψR + ψL) / 2, and the right-eye viewpoint image is Is converted into a parallelogram whose angle is 90− (ψR + ψL) / 2. Further, the left-eye viewpoint image is converted into a parallelogram whose lower right corner angle is 90+ (ψR + ψL) / 2. At this time, the converted right-eye viewpoint image and left-eye viewpoint image are parallelograms having a line-symmetric relationship with each other. As a result, the right-eye viewpoint image is converted as shown in FIG. 14, and the left-eye viewpoint image is converted as shown in FIG. Subsequently, although not shown, a rectangular image is cut out as an image to be displayed on the display reference plane S.

  As a result, a sense of incongruity of the object expressed by the stereoscopic image formed by the image (original image) captured by the imaging device 200 (that is, the inclination of the display reference plane S and the inclination of the imaging reference plane T in the stereoscopic image). The phenomenon that the object falls down or rises due to the difference is reduced. On the other hand, the perspective included in the original image is maintained.

  In addition, since the first modification is a process for removing the relative difference between the average value of the edge inclination of the right-eye viewpoint image and the average value of the edge inclination of the left-eye viewpoint image, non-vertical edges are mixed in the real world. Even if it does, it is converted with high accuracy.

  Note that the average value of the edge angles does not need to be calculated from the angles of all the edges. In order to select only the edges close to vertical in the real world, only the edges whose inclination is equal to or greater than a predetermined angle, for example, 45 degrees or more It may be calculated by selecting only the edges.

  In the first modification, the case where the display reference plane S is vertical is described. However, in the case where the display reference plane S is not vertical, the average value of the edge angle of the right-eye viewpoint image and the edge angle of the left-eye viewpoint image are displayed. The right eye viewpoint image and the left eye viewpoint image may be converted into a parallelogram image so that the average value of the angles has a predetermined angular relationship determined by the angle of the display reference plane S. For example, when the display reference plane S is inclined so as to be slightly above the vertical, the average value of the edge angle of the right eye viewpoint image is relative to the average value of the edge angle of the left eye viewpoint image. Convert to an angle that tilts slightly to the left. Then, the larger the inclination of the display reference plane S that is above the vertical, the larger the average angle of the edge angle of the right-eye viewpoint image becomes so that the angle of tilting to the left with respect to the average value of the edge angle of the left-eye viewpoint image increases. To do.

  Further, it is also possible to specify the inclination of the imaging reference plane T from the relationship between the average value of the edge angle of the right-eye viewpoint image and the average value of the edge angle of the left-eye viewpoint image before correction. Based on the inclination of the surface T, the projection perspective correction processing of the first embodiment may be performed. For example, when the difference between the average value of the edge angle of the right eye viewpoint image and the average value of the edge angle of the left eye viewpoint image is small, it can be determined that the inclination of the imaging reference plane T is small. It can be determined that the inclination of the imaging reference plane T increases as the difference between the average value of the edge angle of the right-eye viewpoint image and the average value of the edge angle of the left-eye viewpoint image increases.

  In the first modification, the conversion amount for converting the right eye viewpoint image and the left eye viewpoint image into a parallelogram may be manually input by the observer via the user interface and acquired by the acquisition unit 310. When the input is completed, the conversion amount may be given to the input signal constituting the stereoscopic image and acquired by the acquisition unit 310 together with the input signal. Alternatively, the conversion amount may be manually input via the user interface while the observer observes the stereoscopic image. When the input is completed, the conversion amount may be given to the input signal constituting the stereoscopic image and acquired by the acquisition unit 310 together with the input signal. Alternatively, an index image (for example, a line segment having an average angle) is displayed on each of the left-eye viewpoint image and the right-eye viewpoint image, and the viewer manually adjusts the conversion amount while checking the index image. You may enter in. When the input is completed, the conversion amount may be given to the input signal constituting the stereoscopic image and acquired by the acquisition unit 310 together with the input signal. Alternatively, the conversion amount may be known. This corresponds to a case where imaging conditions and display conditions are fixed.

[Modification 2]
Hereinafter, Modification Example 2 of the first embodiment will be described. In the following, differences from the first embodiment will be mainly described.

  In the first embodiment, the imaging apparatus 200 captures a plurality of viewpoint images in a state where the imaging reference plane T is not parallel to the display reference plane S. On the other hand, in the second modification, the imaging apparatus 200 captures a plurality of viewpoint images in a state where the imaging reference plane T is parallel to the display reference plane S as illustrated in FIG.

  In such a case, there is no perspective of an object (for example, a high-rise building) included in the viewpoint image. Therefore, the image processing apparatus 300 corrects the viewpoint image so that the size in the left-right direction decreases toward the upper side of the viewpoint image.

  By such correction, the right eye viewpoint image is converted as shown in FIG. 17, and the left eye viewpoint image is converted as shown in FIG. As a result, perspective is given to a plurality of viewpoint images constituting the stereoscopic image.

  Note that the vertical direction of the image may be reduced in consideration of the fact that the vertical size is usually reduced by looking up at the object.

  As a result, there is no sense of incongruity of the object represented by the stereoscopic image (such as the falling of the object or lifting of the object), and a video with a sense of perspective can be obtained.

  In the second modification, the same effects as those in the first embodiment are obtained with respect to the stereoscopic effect and perspective of the object represented by the stereoscopic image. Further, such an image cannot be directly generated even by using computer graphics, and can be obtained by converting the once generated image.

[Modification 3]
Hereinafter, Modification 3 of the first embodiment will be described. In the following, differences from the first embodiment will be mainly described.

  In 1st Embodiment, the imaging device 200 illustrated about the case which mainly images a vertical surface, for example. On the other hand, in Modification 3, for example, as illustrated in FIG. 19, the imaging apparatus 200 mainly captures an image of a horizontal plane (for example, an object V extending in the vertical direction on the horizontal plane). Even in such a case, the first embodiment can be applied. In Modification 3, it should be noted that “vertical direction” means the vertical direction in the image, and does not mean the vertical direction.

  As a result, a sense of incongruity of the object expressed by the stereoscopic image formed by the image (original image) captured by the imaging device 200 (that is, the inclination of the display reference plane S and the inclination of the imaging reference plane T in the stereoscopic image). Phenomena that cause horizontal plane tilt due to differences) are reduced. On the other hand, the perspective included in the original image is maintained.

[Other Embodiments]
Although the present invention has been described with reference to the above-described embodiments, it should not be understood that the descriptions and drawings constituting a part of this disclosure limit the present invention. From this disclosure, various alternative embodiments, examples and operational techniques will be apparent to those skilled in the art.

  Although not particularly mentioned in the embodiment, a display device 100 that displays a plurality of viewpoint images to which the parallax correction processing is applied may be provided. In the parallax correction processing, as in the first embodiment, for the coordinates having different vertical coordinates, the amounts of the left-eye viewpoint image and the right-eye viewpoint image that are shifted in the horizontal direction are different, and the vertical coordinates are the same. In the left-eye viewpoint image, the direction in which the left-right coordinate is shifted is the opposite direction to the direction in which the left-right coordinate is shifted in the right-eye viewpoint image.

[Industrial application fields]
The present invention can be used in the following fields, for example.

In the field of architecture, a stereoscopic image capturing device is used to take a picture of a building, and a stereoscopic image is displayed on a vertical surface or a surface inclined from the vertical.

A captured stereoscopic image is displayed on a portable terminal equipped with a stereoscopic image capturing device, a display device, and an angle detection sensor. At the time of imaging, the inclination of the imaging reference plane T is acquired by a sensor. When displaying, even if the angle of the display device varies, the inclination of the display reference plane S is detected by the sensor in real time, and the stereoscopic image is corrected and displayed.

-In the field of architecture, a three-dimensional image capturing device captures a street from the sky obliquely downward, and displays it on a horizontal plane (floor surface or table surface) or a display device that displays a stereoscopic image installed on a slope.

  DESCRIPTION OF SYMBOLS 100 ... Display apparatus, 200 ... Imaging apparatus, 300 ... Image processing apparatus, 310 ... Acquisition part, 320 ... Identification part, 330 ... Image processing part

Claims (5)

An image processing apparatus that processes a left-eye viewpoint image and a right-eye viewpoint image constituting a stereoscopic image,
An image processing unit that applies a parallax correction process that shifts coordinates in the left-right direction in the left-eye viewpoint image and the right-eye viewpoint image in the parallax correction process; For the coordinates in which the coordinates are shifted in the left-right direction in the viewpoint image and the coordinates in the up-down direction are the same, the direction in which the left-right viewpoint image is shifted in the left-right viewpoint image is the same as that in the right-eye viewpoint image. An image processing apparatus characterized by being in a direction opposite to a direction in which coordinates are shifted. A specifying unit that specifies an inclination of an imaging reference plane perpendicular to an imaging direction of an imaging unit that captures the stereoscopic image and an inclination of the display reference plane;
The image processing unit shifts coordinates in the left-right direction in the left-eye viewpoint image and the right-eye viewpoint image based on the tilt specified by the specifying unit in the parallax correction processing. The image processing apparatus described. An imaging unit that captures the plurality of viewpoint images;
An imaging apparatus comprising the image processing apparatus according to claim 1. A display unit for displaying the stereoscopic image;
A display device comprising the image processing device according to claim 1. An image processing method for processing a plurality of viewpoint images constituting a stereoscopic image,
Applying parallax correction processing for shifting coordinates in the left-right direction in the left-eye viewpoint image and the right-eye viewpoint image,
In the parallax correction process, for the coordinates having different vertical coordinates, the amount of shift of the coordinates in the left-right direction is different in the left-eye viewpoint image and the right-eye viewpoint image, and the coordinates having the same vertical coordinates are An image processing method characterized in that a direction in which the left-right coordinate in the left-eye viewpoint image is shifted is opposite to a direction in which the left-right coordinate is shifted in the right-eye viewpoint image.

Images