WO2012053136A1

WO2012053136A1 - Stereoscopic image processing circuit and stereoscopic image capturing device

Info

Publication number: WO2012053136A1
Application number: PCT/JP2011/004039
Authority: WO
Inventors: 高橋　淳
Original assignee: パナソニック株式会社
Priority date: 2010-10-20
Filing date: 2011-07-14
Publication date: 2012-04-26

Abstract

An image processing circuit (30) captures right-eye captured images and left-eye captured images for configuring stereoscopic images. Operations are repeatedly performed for reading, out of the right-eye captured images, images in a right-eye read region as right-eye images (R1), and for reading, out of the left-eye captured images, images in a left-eye read region as left-eye images (L1). A parallax detection unit (201) detects an image shift amount (D201) in the vertical direction between the right-eye images (R1) and the left-eye images (L1). A read region control unit (301) changes the position of the right-eye read region and/or the position of the left-eye read region such that the image shift amount (D201) in the vertical direction detected by the parallax detection unit (201) becomes smaller.

Description

3D image processing circuit and 3D image capturing apparatus

The present invention relates to a stereoscopic video processing circuit and a stereoscopic video imaging apparatus for processing a right-eye image and a left-eye image for forming a stereoscopic video.

In recent years, research and development on stereoscopic video display systems have been actively carried out against the background of technological innovations such as increasing the screen size, increasing the definition, and improving the richness of color expression. In addition to large-scale systems such as movies and amusement facilities, stereoscopic video playback environments such as home-use television receivers and portable video playback devices are rapidly being established, and stereoscopic video shooting for easy acquisition of stereoscopic video content Attention has also been focused on the device.

The right and left parallax images are required to shoot a stereoscopic image. However, as a technology for making the entire apparatus small and easily obtaining a high-quality parallax image, an optical system having left and right binocular parallax, A detachable lens unit having a shutter for making a light beam incident on one image sensor in time series and a stereoscopic video imaging apparatus using the same have been proposed (for example, Patent Document 1). Patent Document 1 discloses a left and right binocular movable mirror having a reflection function and a left and right binocular for alternately inputting light beams of images having different left and right parallax in time series to one image sensor. Using a lens unit that has two liquid crystal shutters and a photographic optical system that includes a zoom function, the parallax amount of the subject is controlled by driving a movable mirror during the zoom operation of the lens unit. A technique for shooting a high-quality stereoscopic image is disclosed.

JP 2001-218228 A

However, an image shift may occur between the left and right parallax images (the image for the right eye and the image for the left eye) due to an error in the physical mounting accuracy of the movable mirror. In addition, when a moving subject is photographed or a camera shake occurs during operation of the stereoscopic video photographing apparatus, an image shift may occur between the left and right parallax images. In particular, since human eyes are sensitive to vertical image shifts, if vertical image shifts occur between left and right parallax images, there is a high possibility of causing eye strain and motion sickness.

Therefore, an object of the present invention is to provide a stereoscopic video processing circuit and a stereoscopic video imaging apparatus capable of correcting a vertical image shift between a right eye image and a left eye image.

According to one aspect of the present invention, the stereoscopic video processing circuit captures a right-eye captured image and a left-eye captured image for constituting a stereoscopic video, and within the right-eye readout region from the right-eye captured image. The right-eye image and the left-eye image obtained by the imaging processing circuit that repeats the operation of reading the image of the right-eye image as the right-eye image and reading the image in the left-eye readout area as the left-eye image from the left-eye captured image. A stereoscopic video processing circuit for processing an eye image, which is detected by a parallax detection unit that detects a vertical image shift amount between the right eye image and the left eye image, and the parallax detection unit. A readout area control unit that changes at least one of the arrangement of the readout area for the right eye and the arrangement of the readout area for the left eye so that the amount of vertical image shift is small.

The stereoscopic image processing circuit can correct a vertical image shift between the right-eye image and the left-eye image.

The stereoscopic video processing circuit is detected by the parallax detection unit so that a vertical image shift amount between the central portion of the right-eye image and the central portion of the left-eye image is small. An image shift correction unit that processes at least one of the peripheral portion of the right-eye image and the peripheral portion of the left-eye image according to a vertical image shift amount may be further provided.

The above-described stereoscopic video processing circuit can correct the vertical image shift also for the right-eye image and the left-eye image that do not reflect the read area arrangement correction based on the image shift amount.

The stereoscopic video processing circuit further includes an image interpolation unit, the right-eye image and the left-eye image are alternately captured, and the image interpolation unit is based on the right-eye image. And interpolating a new right eye image corresponding to a time when the right eye image is not captured, and a new left corresponding to a time when the left eye image is not captured based on the left eye image. It may interpolate an ophthalmic image.

In the stereoscopic video processing circuit, temporal resolution can be improved by interpolating images (right-eye image and left-eye image) that are not actually captured. In addition, since the amount of signal processing in the imaging processing circuit can be reduced, power consumption can be reduced.

The parallax detection unit includes a right eye corresponding to the same time among the right eye image and the left eye image from the imaging processing circuit and the right eye image and the left eye image interpolated by the image interpolation unit. It is also possible to detect an image shift amount in the vertical direction between the image for use and the image for the left eye.

With this configuration, it is possible to appropriately correct the image shift in the vertical direction even when the subject or the camera moves.

According to another aspect of the present invention, the stereoscopic video imaging apparatus includes the stereoscopic video processing circuit, the imaging processing circuit, and a right eye video and a left eye video of a subject corresponding to the right eye viewpoint and the left eye viewpoint, respectively. And an optical device that alternately images the right-eye video and the left-eye video on the imaging surface of the imaging processing circuit, and the imaging processing circuit uses the optical device to capture the imaging surface. The captured image for the right eye and the captured image for the left eye are acquired by capturing the image formed on the left eye.

In the above-described stereoscopic image capturing device, the vertical image misalignment between the right-eye image and the left-eye image can be corrected, so that it is possible to capture a stereoscopic image that viewers are watching and are not tired.

According to another aspect of the present invention, the stereoscopic video processing circuit includes the right-eye image obtained by the imaging processing circuit that repeats the operation of capturing the right-eye image and the left-eye image for forming the stereoscopic video, and A stereoscopic video processing circuit for processing the left-eye image, a parallax detection unit for detecting a vertical image shift amount between the right-eye image and the left-eye image, and the right-eye image The right-eye image according to the vertical image shift amount detected by the parallax detection unit so that the vertical image shift amount between the central portion of the left-eye image and the central portion of the left-eye image is small. And an image shift correction unit that processes at least one of the peripheral portion of the left eye and the peripheral portion of the left-eye image.

According to another aspect of the present invention, the stereoscopic video imaging apparatus includes the stereoscopic video processing circuit, the imaging processing circuit, and a right eye video and a left eye video of a subject corresponding to the right eye viewpoint and the left eye viewpoint, respectively. And an optical device that alternately images the right-eye video and the left-eye video on the imaging surface of the imaging processing circuit, and the imaging processing circuit uses the optical device to capture the imaging surface. The image for the right eye and the image for the left eye are acquired by imaging the image formed on the left eye.

The optical device includes a shutter unit that alternately passes the right-eye image and the left-eye image, and a photographing lens that forms an image that has passed through the shutter unit on the imaging surface of the imaging processing circuit. In addition, the shutter unit may be detachably connected to the photographing lens.

With this configuration, it is possible to easily switch between 3D video and normal 2D video.

Alternatively, the optical device includes a shutter unit that alternately passes the right-eye image and the left-eye image, and a photographing lens that forms an image that has passed through the shutter unit on the imaging surface of the imaging processing circuit. In addition, the shutter unit may be integrated with the photographing lens.

As described above, it is possible to correct the vertical image shift between the right-eye image and the left-eye image.

The figure which shows the structural example of the stereoscopic video imaging device provided with the stereoscopic video processing circuit. The figure which shows the structural example of an optical device. The figure for demonstrating the captured image imaged by the imaging process circuit. The figure for demonstrating a read-out process. The figure for demonstrating an image interpolation process. The figure for demonstrating the detection process of the image deviation | shift amount. The figure for demonstrating the arrangement | positioning change process of a read-out area | region. The figure for demonstrating an image shift correction process. The figure for demonstrating the operation timing of a three-dimensional video processing circuit. The figure for demonstrating the modification of an image interpolation process. The figure for demonstrating the modification of the operation timing of a three-dimensional video processing circuit. The figure for demonstrating the modification of a read-out area | region.

Hereinafter, embodiments will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals, and description thereof will not be repeated.

(Stereoscopic image capturing device)
FIG. 1 shows a configuration example of a stereoscopic video imaging apparatus. This apparatus includes a stereoscopic video processing circuit 10, an optical device 20, an imaging processing circuit 30, and a storage 40 (for example, an SD card or an HDD).

(Optical device)
The optical device 20 alternately passes the right-eye video RR and the left-eye video LL of the subject corresponding to the right-eye viewpoint and the left-eye viewpoint, respectively, and the right-eye video RR and the left-eye on the imaging surface of the imaging processing circuit 30. The eye images LL are alternately formed.

As shown in FIG. 2, the optical device 20 may include a shutter unit 21 and a photographing lens 22. The shutter unit 21 alternately passes the right-eye video R0 and the left-eye video L0 of the subject corresponding to the right-eye viewpoint and the left-eye viewpoint, respectively. For example, the shutter unit 21 includes a right eye optical system 211R and a left eye optical system 211L, and a liquid crystal shutter 212 arranged in parallel. The right eye optical system 211R is bent so as to make an angle with the optical axis of the right eye image, and the left eye optical system 211L is bent so as to make an angle with the optical axis of the left eye image. The liquid crystal shutter 212 alternately passes the right-eye video and the left-eye video in time series. The photographic lens 22 forms an image that has passed through the shutter unit 21 on the imaging surface of the imaging processing circuit 30. As a result, the right-eye video RR and the left-eye video LL are alternately formed on the imaging surface of the imaging processing circuit 30.

The shutter unit 21 may be detachably connected to the photographing lens 22. For example, the shutter unit 21 may be an attachment that is detachably attached to the photographing lens 22. With this configuration, it is possible to easily switch between a stereoscopic video and a normal two-dimensional video. Alternatively, the shutter unit 21 may be integrated with the photographing lens 22. For example, the shutter unit 21 may constitute a lens unit that can be integrated with the photographing lens 22 and exchanged. With this configuration, it is possible to easily switch between a stereoscopic video and a normal two-dimensional video.

[Imaging processing circuit]
The imaging processing circuit 30 captures a right-eye captured image and a left-eye captured image for forming a stereoscopic image by capturing an image formed on the imaging surface by the optical device 20. Accordingly, as shown in FIG. 3, the left-eye captured image L0 _(n-3) , right eye at times T _(n-3) , T _(n-2) ,..., T _{(n + 2)} , T _{(n + 3)} use the captured image _{R0 (n-2), ...} , the right-eye captured image _{R0 (n + 2),} left-eye captured image _{L0 (n + 3)} are respectively imaged. That is, the right-eye captured image R0 and the left-eye captured image L0 are alternately captured. Note that “right-eye captured image R0” is a generic name for right-eye captured images (..., R0 _(n−2) , R0 _(n) , R0 _{(n + 2)} ,...). L0 ″ is a generic name for the left-eye captured images (..., L0 _(n−3) , L0 _(n−1) , L0 _{(n + 1)} , L0 _{(n + 3)} ,. Further, the imaging processing circuit 30 captures the right-eye captured image R0 and the left-eye captured image L0, reads an image in the right-eye readout region from the right-eye captured image R0 as the right-eye image R1, and The operation of reading the image in the left-eye readout region from the eye-captured image L0 as the left-eye image L1 is repeated. In this way, the right-eye image R1 and the left-eye image L1 are alternately read. For example, as illustrated in FIG. 4, the imaging processing circuit 30 uses the upper left of the right-eye captured image R0 (or the left-eye captured image L0) as the origin (0, 0), and sets the readout position coordinates (x, y). An image in the rectangular right eye readout region R101 (or left eye readout region L101), which is the upper left corner, is read out as a right eye image R1 (or left eye image L1). Here, the arrangement of the right-eye readout area R101 and the arrangement of the left-eye readout area L101 can be changed.

The imaging processing circuit 30 may include an imaging element 31, an analog processing unit 32, and a digital processing unit 33. The imaging device 31 captures an image formed on the imaging surface by the optical device 20 by converting light applied to the imaging surface into an electrical signal. As a result, the imaging element 31 has captured the right-eye captured image R0 and the left-eye captured image L0. The imaging element 31 captures the right-eye captured image R0 and the left-eye captured image L0, reads the image in the right-eye readout region R101 from the right-eye captured image R0 as the right-eye image R1, and The operation of reading the image in the left eye readout region L101 from the eye captured image L0 as the left eye image L1 is repeated. The analog processing unit 32 performs analog signal processing (for example, noise reduction, gain control, A / D) on the right-eye image R1 (analog signal) and the left-eye image L1 (analog signal) obtained by the image sensor 31. Conversion). The digital processing unit 33 performs digital signal processing (for example, white balance processing, Y / C conversion) on the right-eye image R1 (digital signal) and the left-eye image L1 (digital signal) processed by the analog processing unit 32. Process).

[3D image processing circuit]
The stereoscopic video processing circuit 10 is connected to the imaging processing circuit 30 and processes the right-eye image R1 and the left-eye image L1 obtained by the imaging processing circuit 30. Further, here, the stereoscopic video processing circuit 10 controls the arrangement of the right eye readout region R101 and the arrangement of the left eye readout region L101 in the imaging processing circuit 30. The stereoscopic video processing circuit 10 includes a memory 11 that temporarily stores an image for the right eye and an image for the left eye, and an image format of the image for the right eye and the image for the left eye that is stored in a predetermined image format (the storage 40). An image processing unit 12 for converting the image processing unit 10 to the image format), a CPU 13 for overall control of the entire stereoscopic video processing circuit 10, and an image recording unit 14 for storing the right-eye image and the left-eye image in the storage 40. It may be included. The memory 11 includes an image memory 101 and an encode buffer 102. The image processing unit 12 includes a parallax detection unit 201, an image interpolation unit 202, and an image shift correction unit 203. The CPU 13 includes a read area control unit 301 and an image shift control unit 302. The image recording unit 14 includes an encoder 401 and a storage control unit 402.

<Image memory>
The image memory 101 stores a right-eye image and a left-eye image obtained by the imaging processing circuit 30, and a right-eye image and a left-eye image obtained by the image processing unit 12.

<Image interpolation unit>
The image interpolation unit 202 reads the right-eye image R1 stored in the image memory 101, and based on the read right-eye image R1, the time when the right-eye image R1 is not captured (the right-eye image A new right-eye image R1 corresponding to a time different from the time corresponding to R1 is interpolated. Also, the image interpolation unit 202 reads the left eye image L1 stored in the image memory 101, and based on the read left eye image L1, the time when the left eye image L1 is not captured (the left eye A new left-eye image L1 corresponding to a time different from the time corresponding to the image L1 is interpolated. In addition, the image interpolation unit 202 stores a new right-eye image R1 and left-eye image L1 in the image memory 101.

Here, the image interpolation processing by the image interpolation unit 202 will be described with reference to FIGS. 5A and 5B. In the following description, for convenience of explanation, the right-eye image R1 corresponding to the time T _(k) is referred to as “right-eye image R1 _(k) ”, and the left-eye image corresponding to the time T _(k) is used. The image L1 is expressed as “left-eye image L1 _(k) ”. As shown in FIG. 5A, when the imaging processing circuit 30 stores the right-eye image R1 _(n) corresponding to the time T _(n) in the image memory 101, the image interpolation unit 202 performs the time T _(n-3). And the left eye image L _(n-3) and the left eye image L _(n-1) respectively corresponding to the time T _(n-1) and the left eye images L1 _(n-3) and L1 _(n Based on ( _-1) , image interpolation processing (for example, motion estimation and motion compensation) is executed to interpolate a new left-eye image L1 _(n-2) corresponding to time T _(n-2) . Further, the image interpolation unit 202 stores the new left eye image L1 _(n−2) in the image memory 101. Next, as illustrated in FIG. 5B, when the imaging processing circuit 30 stores the left-eye image L1 _{(n + 1)} corresponding to the time T _{(n + 1)} in the image memory 101, the image interpolation unit 202 performs the time T _{(n -2)} and right-eye image R _(n-2) and right-eye image R _(n) corresponding to time T _(n) , respectively, and read right-eye images R1 _(n-2) and R1 _(n) Based on the above, the image interpolation process is executed to interpolate a new right-eye image R1 _(n-1) corresponding to the time T _(n-1) . In addition, the image interpolation unit 202 stores the new right-eye image R1 _(n−1) in the image memory 101. By repeating such an operation, it is possible to interpolate images that are not actually captured (right-eye image R1 and left-eye image L1).

<< Parallax detection unit >>
The parallax detection unit 201 detects an image shift amount D201 in the vertical direction between the right-eye image and the left-eye image stored in the image memory 101.

Here, with reference to FIG. 6, the detection process of the image shift amount by the parallax detection unit 201 will be described. Here, the right-eye image R1 and the left-eye image L1 are images of n rows and m columns, and the right-eye image R1 is “2L (horizontal line 2) below the left-eye image L1 in the vertical direction. This)) ” Here, the image shift amount D201 is a value indicating the image shift amount in the vertical direction of the right-eye image R1 with respect to the left-eye image L1, and the right-eye image R1 is in the vertical direction with respect to the left-eye image L1. It is assumed that the value when the image is shifted downward is “positive” and the value when the image R1 for the right eye is shifted upward in the vertical direction with respect to the image L1 for the left eye is “negative”.

First, the parallax detection unit 201 calculates, for each horizontal line of the right-eye image R1, the sum of luminance values of m pixels included in the horizontal line as a right-eye feature amount. Thereby, n right eye feature quantities RS ₍₁₎ , RS ₍₂₎ ,..., RS _(n) respectively corresponding to n horizontal lines of the right eye image R1 are calculated. Similarly, the parallax detection unit 201 calculates the sum of the luminance values of n pixels included in the horizontal line for each horizontal line of the left-eye image L1 as the left-eye feature amount, thereby N left eye feature values LS ₍₁₎ , LS ₍₂₎ ,..., LS _(n) respectively corresponding to n horizontal lines of the image L1 are calculated.

Next, the parallax detection unit 201 changes the vertical image shift amount Δy between the right-eye image R1 and the left-eye image L1 and, according to the image shift amount Δy, the right-eye feature amount RS _{( 1)} , RS ₍₂₎ ,..., RS _(n) and the left eye feature quantity LS ₍₁₎ , LS ₍₂₎ ,..., LS _(n) are changed, and the right eye feature quantity RS is changed. _{_{(1), RS (2)}} , ..., RS (n) and the left-eye feature amount _{_{LS (1), LS (2}} ), ..., and calculates a difference square sum of the _{LS (n).} For example, when the image shift amount Δy is “0”, the sum of squared differences is (RS ₍₁₎ −LS ₍₁₎ ) ² + (RS ₍₂₎ −LS ₍₂₎ ) ² +... + (RS _{( n)} -LS _(n) ) ² When the image shift amount Δy is “+ 2L”, the sum of squared differences is (RS ₍₃₎ −LS ₍₁₎ ) ² + (RS ₍₄₎ −LS ₍₂₎ ) ² +... + (RS _{( n)} −LS _(n−2) ) ²

Next, the disparity detection unit 201, the right-eye feature amount _{_{RS (1), RS (2}} ), ..., RS (n) and the left-eye feature amount _{_{LS (1), LS (2}} ), ..., LS ( The image shift amount Δy when the difference square sum with _n) is minimized is detected as the image shift amount D201. In the case of FIG. 6, the sum of squared differences is minimized when the image shift amount Δy is “+ 2L”, and thus the parallax detection unit 201 determines the image shift amount D201 to be “+ 2L”.

In this way, the parallax detection unit 201 detects the image shift amount D201 in the vertical direction between the right-eye image R1 and the left-eye image L1. Note that the method of detecting the image shift amount by the parallax detection unit 201 is not limited to the above-described method (a method of detecting based on the sum of squared differences), and another method such as block matching may be used.

<Reading area control unit>
The readout area control unit 301 arranges the arrangement of the readout area R101 for the right eye in the imaging processing circuit 30 (specifically, the imaging element 31) and reduces the vertical image shift amount D201 detected by the parallax detection unit 201. At least one of the arrangements of the left-eye readout area L101 is changed.

Here, with reference to FIG. 7, a read area arrangement changing process by the read area control unit 301 will be described. Here, it is assumed that the right-eye image R1 is shifted by the image shift amount Δy downward in the vertical direction with respect to the left-eye image L1. In the case of FIG. 7, the readout region control unit 301 shifts the image shift amount D201 detected by the parallax detection unit 201 (the right-eye image R1 is shifted by the image shift amount Δy downward in the vertical direction with respect to the left-eye image L1. The readout position coordinates of the right eye readout area R101 (the coordinates of the upper left corner of the right eye readout area R101) are changed from the coordinates (x, y) to the coordinates (x, y + Δy). To do. As a result, the arrangement of the right-eye readout region R101 is shifted by the image shift amount Δy downward in the vertical direction.

<Image shift correction unit / image shift control unit>
The image shift control unit 302 controls processing by the image shift correction unit 203 based on the vertical image shift amount D201 detected by the parallax detection unit 201. In response to control by the image shift control unit 302, the image shift correction unit 203 reads the right-eye image R1 and the left-eye image L1 stored in the image memory 101, and the central portion of the read right-eye image R1. The right-eye image R1 based on the vertical image shift amount D201 detected by the parallax detection unit 201 so that the vertical image shift amount between the read-out image L1 and the central portion of the read left-eye image L1 becomes small. At least one of the peripheral portion of the right eye image R1 read from the image memory 101 and the peripheral portion of the left eye image L1 (left eye image L1 read from the image memory 101) is processed. Further, the image shift correction unit 203 stores the processed right-eye image R1 (or the left-eye image L1) in the image memory 101.

Here, with reference to FIG. 8, the image shift correction processing by the image shift correction unit 203 will be described. Here, it is assumed that the right-eye image R1 is shifted by the image shift amount Δy downward in the vertical direction with respect to the left-eye image L1. Also, the right-eye image R1 is an image of n rows and m columns. In this case, the image shift correction unit 203 discards the upper end portion R111 (the image region from the coordinates (x, y) to the coordinates (x, y + Δy)) of the right eye image R1, and the center portion of the right eye image R1. R112 (image area from coordinates (x, y + Δy) to coordinates (x, y + n-2Δy)) is multiplied by the same size, and the lower end portion R113 (coordinates (x, y + n-2Δy) to the coordinates (x, y + n-2Δy)) The image area) up to y + n) is enlarged twice. By processing the image in this way, the central portion R112 of the right-eye image R1 is shifted upward in the vertical direction, so that the central portion R112 of the right-eye image R1 and the central portion of the left-eye image L1 The amount of image shift in the vertical direction can be reduced.

When the right-eye image R1 is shifted upward in the vertical direction by the image shift amount Δy with respect to the left-eye image L1, the image shift correction unit 203 uses the lower end portion R113 (in this case) of the right-eye image R1. , The image region from the coordinates (x, y + n−Δy) to the coordinates (x, y + n)) is discarded, and the central portion R112 (in this case, the coordinates (x, y + 2Δy) to the coordinates (x, y + n) of the right-eye image R1. -Image area up to -Δy) is magnified and the upper end portion R111 of the right-eye image R1 (in this case, the image area from coordinates (x, y) to coordinates (x, y + 2Δy)) is doubled. May be. By processing the image in this way, the central portion R112 of the right-eye image R1 is shifted downward in the vertical direction. Therefore, the central portion R112 of the right-eye image R1 and the central portion of the left-eye image L1 The amount of image shift in the vertical direction can be reduced.

Further, the case where the lower end portion R113 (or the upper end portion R111) of the right eye image R1 is uniformly doubled has been described as an example, but the center of the right eye image R1 (or the left eye image L1) has been described. Other methods, such as a method of reducing the enlargement factor in the vicinity and processing the image so that the enlargement factor increases as it approaches the periphery of the right-eye image R1 (or the left-eye image L1), may be employed.

<Encoder / Storage Control Unit>
The encoder 401 compresses the right eye image and the left eye image stored in the image memory 101 in accordance with a predetermined format, and stores the compressed right eye image and left eye image in the encode buffer 102. The encode buffer 102 stores not only the compressed right eye image and left eye image but also intermediate data generated by image compression by the encoder 401. The storage control unit 402 stores the compressed right eye image and left eye image stored in the encode buffer 102 in the storage 40.

[Operation timing]
Next, the operation timing of the stereoscopic video processing circuit 10 will be described with reference to FIG. Here, images (right-eye image R1 and left-eye image L1) read from the captured image (right-eye captured image R0 and left-eye captured image L0) are referred to as “read-out images”, and image interpolation is performed. Images interpolated by the unit 202 (right-eye image R1 and left-eye image L1) are referred to as “interpolated images”. The vertical image shift amount D201 between the right-eye image R1 _(k) and the left-eye image L1 _(k) corresponding to the time T _(k) is expressed as “image shift amount D201 _(k) ”. To do.

At time T _(n-2) , the imaging processing circuit 30 captures the right-eye captured image R0 _(n-2) and reads the left-eye image read from the left-eye captured image L0 _(n-3). L1 _(n-3) is stored in the image memory 101. Next, at time T _(n−1) , the imaging processing circuit 30 captures the left-eye captured image L0 _(n−1) and reads the right-eye captured from the right-eye captured image R0 _(n−2). The eye image R1 _(n-2) is stored in the image memory 101. Next, at time T _(n) , the imaging processing circuit 30 captures the right-eye captured image R0 _(n) and reads the left-eye image L1 read from the left-eye captured image L0 _(n−1). _(N−1) is stored in the image memory 101.

Next, at time T _{(n + 1)} , the image interpolation unit 202 performs the time T based on the left-eye image L1 _(n-3) and the left-eye image L1 _(n-1) stored in the image memory 101. _The left-eye image L1 _(n-2) corresponding to _(n-2) is interpolated and stored in the image memory 101.

Next, at time T _{(n + 2)} , the parallax detection unit 201 detects the right-eye image R1 _(n-2) (read-out image) and the left-eye image L1 _(n-2 ₎ corresponding to the time T _(n-2). ₎ A vertical image shift amount D201 _(n−2) between (interpolated image) is detected.

Next, at time T _{(n + 3)} , the readout region control unit 301 arranges the right eye readout region R101 and the left eye based on the image shift amount D201 _(n-2) detected by the parallax detection unit 201. At least one of the arrangements of the read areas L101 is changed.

Further, here, the image shift amount _{D201 (n-2)} corresponding to the right-eye captured image _{R0 (n-2)} is left-eye captured image based on the image shift amount _{D201 (n-2)} from being imaged It takes 5 cycle times (5T) until the arrangement of the read area of L0 _{(n + 3)} is corrected. That is, in the right-eye image R1 and the left-eye image L1 obtained during the period from the time T _(n-2) to the time T _{(n + 3)} , the readout region based on the image shift amount D201 _(n-2) This means that the arrangement correction is not reflected. Therefore, after time T _{(n + 3)} , the image shift correction unit 203 performs the right eye image R1 and the left eye image L1 (during the period from time T _(n-2) to time T _{(n + 3).} Right-eye images R1 _(n-3) , R1 _(n-2) , R1 _(n-1) , R1 _(n) , and left-eye images L1 _(n-4) , L1 _(n-3) , L1 _(N-2) , L1 _(n-1) , L1 _{(n + 1)} ) are subjected to image shift correction processing.

As described above, at least one of the arrangement of the right-eye readout area R101 and the arrangement of the left-eye readout area L101 is changed so that the vertical image shift amount D201 detected by the parallax detection unit 201 is reduced. Thus, it is possible to correct the image shift in the vertical direction between the right-eye image R1 and the left-eye image L1. As a result, it is possible to shoot a stereoscopic image that the viewer is watching and is not tired.

Further, the image shift correction processing by the image shift correction unit 203 corrects the image shift in the vertical direction also for the right-eye image R1 and the left-eye image L1 that do not reflect the read area arrangement correction based on the image shift amount D201. it can.

Also, the image interpolation processing by the image interpolation unit 202 can interpolate images that are not actually captured (the right-eye image R1 and the left-eye image L1). Thereby, temporal resolution can be improved and a 3D image with smooth motion can be obtained. In addition, the amount of signal processing in the imaging processing circuit 30 can be reduced by interpolating images that are not actually captured. Thereby, power consumption can be reduced. For example, assuming that 60 right-eye images R1 and 60 left-eye images L1 are required per second in order to form a stereoscopic video, if the above-described image interpolation processing is not performed, the imaging process The circuit 30 must process 120 images per second. On the other hand, when the above-described image interpolation processing is executed, the number of images processed by the imaging processing circuit 30 per second can be reduced from 120 to 60.

Note that the parallax detection unit 201 preferably uses the right-eye image R1 and the left-eye image L1 corresponding to the same time as an object of image shift amount detection processing. By performing the processing in this way, it is possible to appropriately correct the image shift in the vertical direction even when the subject or the camera moves. Note that the parallax detection unit 201 may use the image R1 for the right eye and the image L1 for the left eye corresponding to different times as targets for the image shift amount detection processing.

[Modification of image interpolation processing]
As shown in FIGS. 10A and 10B, the image interpolation unit 202 also uses the right-eye image R1 corresponding to the future time from the right-eye image R1 (or the left-eye image L1) stored in the image memory 101. (Or the left-eye image L1) may be interpolated. For example, as shown in FIG. 10A, when the imaging processing circuit 30 stores the right-eye image R1 _(n) corresponding to the time T _(n) in the image memory 101, the image interpolation unit 202 performs the time T _{(n− 3)} and the left eye image L _(n-3) and the left eye image L _(n-1) corresponding to the time T _(n-1) , respectively, are read out, and the left eye images L1 _(n-3) , L1 are read out. _An image interpolation process (for example, motion estimation or motion compensation) may be executed based on _(n−1) to interpolate a new left eye image L1 _(n) corresponding to time T _(n) . Further, as shown in FIG. 10B, when the imaging processing circuit 30 stores the left-eye image L1 _{(n + 1)} corresponding to the time T _{(n + 1)} in the image memory 101, the image interpolation unit 202 performs the time T _{(n− 2)} and the right-eye image R _(n-2) and the right-eye image R _(n) corresponding to the time T _(n) , respectively, are read out to the right-eye images R1 _(n-2) and R1 _(n) . Based on this, an image interpolation process may be executed to interpolate a new right eye image R1 _{(n + 1)} corresponding to time T _{(n + 1)} . By processing in this way, as shown in FIG. 11, the left eye is picked up based on the image shift amount D201 _(n) after the right-eye captured image R0 _(n) corresponding to the image shift amount D201 _(n) is captured. The time required for correcting the arrangement of the readout region of the captured image L0 _{(n + 3)} is 3 cycle times (3T). That is, as compared with the case of FIG. 9, it is possible to shorten the time required from when the captured image corresponding to the image shift amount D201 is captured until the arrangement of the readout region of the captured image is corrected based on the image shift amount D201.

[Modification of readout area]
As shown in FIG. 12, the imaging processing circuit 30 reads an image in the right-eye readout area R101a (an area wider than the right-eye readout area R101) from the right-eye captured image R0 as the right-eye image R1. Alternatively, an image in the left-eye readout area L101a (an area wider than the left-eye readout area L101) may be read out as the left-eye image L1 from the left-eye captured image L0. In this case, the parallax detection unit 201 and the image interpolation unit 202 process the right-eye image R1 (or the left-eye image L1) wider than the right-eye readout region R101 (or the left-eye readout region L101). It will be. The image shift correction unit 203 also performs the right-eye image R1 and the left-eye image so that the amount of vertical image shift between the center portion of the right-eye image R1 and the center portion of the left-eye image L1 is small. The right-eye readout area R101 and the left-eye readout area L101 are respectively arranged in the image L1, and the image in the right-eye readout area R101 and the image in the left-eye readout area L101 are replaced with a new right-eye image R1 and left You may read as the image L1 for eyes. Alternatively, the encoder 401 responds to the control by the image shift control unit 302 and the vertical image between the central portion of the right eye image R1 and the central portion of the left eye image L1 stored in the image memory 101. The right-eye readout region R101 and the left-eye readout region L101 are arranged in the right-eye image R1 and the left-eye image L1, respectively, so that the shift amount is small, and the image in the right-eye readout region R101 and the left-eye The image in the read-out area L101 may be read out as a new right-eye image R1 and a left-eye image L1, and the new right-eye image R1 and the left-eye image L1 may be subjected to compression processing. As described above, the imaging processing circuit 30 may execute the reading process based on the right-eye reading area R101a and the left-eye reading area L101a including the vertical image shift correction amount as an offset.

[Mounting form]
Note that the image processing unit 12, the CPU 13, and the image recording unit 14 may be formed on the same semiconductor chip to form a SoC (System on Chip) circuit, or each on a different semiconductor chip. It may be formed. The SoC circuit may include not only the image processing unit 12, the CPU 13, and the image recording unit 14, but also the digital processing unit 33. That is, in the SoC circuit, the image processing unit 12, the CPU 13, the image recording unit 14, and the digital processing unit 33 may be formed on the same semiconductor chip. The digital processing unit 33 may be formed on a semiconductor chip different from the SoC circuit.

(Other embodiments)
In the above description, the imaging processing circuit 30 (imaging element 31) may not execute the readout process. That is, the imaging processing circuit 30 captures the right-eye captured image R0 and the left-eye captured image L0 for forming a stereoscopic image by capturing the image formed on the imaging surface by the optical device 20, The right-eye captured image R0 and the left-eye captured image L0 may be output as the right-eye image R1 and the left-eye image L1. In this case, in the stereoscopic video processing circuit 10, the CPU 13 does not have to include the read area control unit 301. Even in such a configuration, the image shift in the vertical direction between the right-eye image R1 and the left-eye image L1 can be corrected by the image shift correction process by the image shift correction unit 203.

As described above, the above-described stereoscopic video processing circuit and stereoscopic video imaging apparatus are useful in fields such as a video camera and a digital still camera with a moving image function.

DESCRIPTION OF SYMBOLS 10 3D image processing circuit 20 Optical device 30 Imaging processing circuit 40 Storage 11 Memory 12 Image processing part 13 CPU
14 Image recording unit 21 Shutter unit 22 Shooting lens 31 Image sensor 32 Analog processing unit 33 Digital processing unit 101 Image memory 102 Encoding buffer 201 Parallax detection unit 202 Image interpolation unit 203 Image shift correction unit 301 Read area control unit 302 Image shift control unit 401 Encoder 402 Storage control unit

Claims

A right-eye captured image and a left-eye captured image for forming a stereoscopic image are captured, an image in a right-eye readout region is read out from the right-eye captured image as a right-eye image, and the left-eye captured image A stereoscopic video processing circuit that processes the right-eye image and the left-eye image obtained by an imaging processing circuit that repeats an operation of reading an image in a left-eye readout region from the image as a left-eye image;
A parallax detection unit for detecting a vertical image shift amount between the right-eye image and the left-eye image;
A readout area control unit that changes at least one of the arrangement of the readout area for the right eye and the arrangement of the readout area for the left eye so that the vertical image shift amount detected by the parallax detection section is reduced. A stereoscopic image processing circuit characterized by that.
In claim 1,
According to the vertical image shift amount detected by the parallax detection unit so that the vertical image shift amount between the central portion of the right-eye image and the central portion of the left-eye image is reduced. A stereoscopic video processing circuit, further comprising an image shift correction unit that processes at least one of a peripheral portion of the right-eye image and a peripheral portion of the left-eye image.
In claim 1 or 2,
An image interpolation unit;
The image for the right eye and the image for the left eye are captured alternately,
The image interpolation unit interpolates a new right-eye image corresponding to a time when the right-eye image is not captured based on the right-eye image, and based on the left-eye image A stereoscopic video processing circuit characterized by interpolating a new left-eye image corresponding to a time when no image is captured.
In claim 3,
The parallax detection unit is a right-eye image corresponding to the same time among the right-eye image and the left-eye image from the imaging processing circuit and the right-eye image and the left-eye image interpolated by the image interpolation unit. A stereoscopic video processing circuit that detects an image shift amount in a vertical direction between a left eye image and a left eye image.
The stereoscopic video processing circuit according to any one of claims 1 to 4,
The imaging processing circuit;
The right-eye video and the left-eye video of the subject corresponding to the right-eye viewpoint and the left-eye viewpoint are alternately passed, and the right-eye video and the left-eye video are alternately passed on the imaging surface of the imaging processing circuit. An optical device for imaging,
The stereoscopic imaging apparatus, wherein the imaging processing circuit acquires the right-eye captured image and the left-eye captured image by capturing an image formed on the imaging surface by the optical device.
A stereoscopic video processing circuit for processing the right-eye image and the left-eye image obtained by an imaging processing circuit that repeats an operation of capturing a right-eye image and a left-eye image for constituting a stereoscopic video, ,
A parallax detection unit for detecting a vertical image shift amount between the right-eye image and the left-eye image;
According to the vertical image shift amount detected by the parallax detection unit so that the vertical image shift amount between the central portion of the right-eye image and the central portion of the left-eye image is reduced. A stereoscopic video processing circuit comprising: an image shift correction unit that processes at least one of a peripheral portion of the right-eye image and a peripheral portion of the left-eye image.
In claim 6,
An image interpolation unit;
The image for the right eye and the image for the left eye are captured alternately,
The image interpolation unit interpolates a new right-eye image corresponding to a time when the right-eye image is not captured based on the right-eye image, and based on the left-eye image A stereoscopic video processing circuit characterized by interpolating a new left-eye image corresponding to a time when no image is captured.
In claim 7,
The parallax detection unit is an image for the right eye corresponding to the same time among the image for the right eye and the image for the left eye from the imaging processing circuit, and the image for the right eye and the image for the left eye interpolated by the image interpolation unit. A stereoscopic video processing circuit that detects an image shift amount in a vertical direction between a left eye image and a left eye image.
A stereoscopic video processing circuit according to any one of claims 6 to 8,
The imaging processing circuit;
The right-eye video and the left-eye video of the subject corresponding to the right-eye viewpoint and the left-eye viewpoint are alternately passed, and the right-eye video and the left-eye video are alternately passed on the imaging surface of the imaging processing circuit. An optical device for imaging,
The stereoscopic imaging apparatus, wherein the imaging processing circuit acquires the image for the right eye and the image for the left eye by capturing an image formed on the imaging surface by the optical device.
In claim 5 or 9,
The optical device is
A shutter unit that alternately passes the video for the right eye and the video for the left eye;
A photographic lens that forms an image of the image passing through the shutter unit on the imaging surface of the imaging processing circuit;
The three-dimensional image photographing device, wherein the shutter unit is detachably connected to the photographing lens.
In claim 5 or 9,
The optical device is
A shutter unit that alternately passes the video for the right eye and the video for the left eye;
A photographic lens that forms an image of the image passing through the shutter unit on the imaging surface of the imaging processing circuit;
The shutter unit is integrated with the photographing lens.