JP2016158266A - Image pickup apparatus and video record reproducing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To capture images for generating a high-quality stereoscopic image with uniform image quality in left/right images.SOLUTION: An image pickup apparatus comprises a mirror, an image pickup device, and another image pickup device. The mirror separates incident light into a first component and a second component. The image pickup device forms an image of the first component and obtains a first image signal to use to generate a parallax detection image. The other image pickup device forms an image of the second component and obtains a second image signal to use to generate a basic image with higher image quality than the parallax detection image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an imaging apparatus, and more particularly to an imaging apparatus that captures a subject as a stereoscopic image, and a video recording / reproducing system.

  Conventionally, a system has been proposed in which a common subject is simultaneously imaged by two left and right video cameras and a stereoscopic image is displayed by simultaneously outputting the left and right images. However, such a system using two video cameras is large in size as an apparatus and lacks in mobility, and the optical axes of the left and right video cameras are likely to be displaced, and an image having an appropriate parallax. It is difficult to get. For example, the distance between the two video cameras may become large, the left and right optical axes may be shifted during zooming due to individual differences in the lenses of each video camera, and the left and right screen sizes may be different. Further, during the focusing operation, the left and right video cameras may be shifted up and down, for example, during the convergence operation in which the left and right video cameras are directed toward the subject.

  When such an optical axis shift occurs in the left and right video cameras, the visual system of the user who views the stereoscopic image is forced to perform information processing different from that of daily life, which causes visual fatigue. Further, when the user views the left and right images as they are without using stereoscopic glasses, the subject looks double, resulting in an unnatural image.

  For this reason, there has been proposed an imaging device that captures an image by separating light from a subject into two light beams by a mirror in a region that becomes a pupil of one lens (see, for example, Patent Document 1). In this imaging apparatus, left and right video data for stereoscopic viewing can be obtained by imaging each of the two separated light beams.

JP 2010-81580 A

  In the above-described prior art, left and right video data for stereoscopic viewing can be obtained by separating light from a subject into right and left by a mirror. However, in this case, the thickness of the separated light changes depending on the incident angle of the subject with respect to the mirror, etc., and the brightness of the left and right images may become unbalanced, possibly degrading the image quality.

  The present technology has been created in view of such a situation, and an object thereof is to perform imaging for generating a high-quality stereoscopic image in which the image quality of the left and right images is uniform.

  The present technology has been made to solve the above-described problems. The first aspect of the present technology includes a photographing lens that collects light from a subject, and parallel light that transmits the collected light. A relay lens that transmits and reflects the parallel light, a mirror that splits light guided by being transmitted or reflected by the semi-transmissive film into left and right, and the split light. First and second imaging lenses that respectively image the first and second imaging lenses that convert the images formed by the first and second imaging lenses into parallax detection images based on electronic signals, respectively. An element, a third imaging lens that forms an image of light reflected or transmitted by the semi-transmissive film but not guided to the mirror, and light imaged by the third imaging lens. Change to basic image by electronic signal An imaging device comprising a third imaging device that. This brings about the effect | action that a basic image and a parallax detection image are produced | generated simultaneously.

  Further, in the first aspect, the parallax image generation unit that generates a parallax image at the same viewpoint as each of the parallax detection images by comparing the basic image and the parallax detection images based on the basic image. May be further provided. This brings about the effect | action of producing | generating a parallax image.

  In the first aspect, the parallax image generation unit compares the basic image and the parallax detection image with each other to search for corresponding pixels, and based on the search result. You may provide the pixel moving part which produces | generates the said parallax image by moving the pixel of the said basic image to the position of the corresponding pixel in the said parallax detection image. This brings about the effect | action that a parallax image is produced | generated by moving the pixel of a basic image to the position of the corresponding pixel in a parallax detection image based on a search result.

  Further, in this first aspect, the video data including the basic image or the parallax detection image as a time-series frame is searched for pixels corresponding to each other by comparing the two frames at different times, and the search result A time direction interpolation unit that interpolates an image at an arbitrary time by obtaining the midpoint coordinates and the pixel value of the corresponding pixel based on the above may be further provided. This brings about the effect of improving the frame rate.

  Further, in the first aspect, the pixel corresponding to each other is searched by comparing any two of the basic image or the parallax detection image, and the midpoint of the corresponding pixel is determined based on the search result. You may make it further comprise the spatial direction interpolation part which interpolates the image of arbitrary viewpoints by calculating | requiring a coordinate and its pixel value. This brings about the effect | action that the parallax image in the viewpoint of arbitrary positions is produced | generated.

  In the first aspect, each of the first to third imaging elements may generate the basic image or the parallax detection image at a rate of 60 frames or more per second. Each of the first and second imaging elements may generate the parallax detection image at a rate of 230 to 250 frames per second. In addition, each of the first and second imaging elements may generate the parallax detection image at a rate of 290 to 310 frames per second. Further, each of the first and second imaging elements may generate the parallax detection image at a rate of 590 to 610 frames per second.

  In addition, the second aspect of the present technology provides a photographing lens that collects light from a subject, a relay lens that transmits the collected light to be parallel light, and a half that transmits and reflects the parallel light. A transmission film, a mirror that splits light guided by being transmitted or reflected by the semi-transmissive film into two left and right, and first and second imaging lenses that image the split light, respectively The first and second imaging elements for converting the light imaged by the first and second imaging lenses into parallax detection images by electronic signals, respectively, and the light reflected or transmitted by the semi-transmissive film A third imaging lens that forms an image of light that has not been guided to the mirror, a third imaging element that converts the light imaged by the third imaging lens into a basic image based on an electronic signal, The basic image and the parallax test A parallax image generation unit that generates a parallax image of the same viewpoint as each of the parallax detection images by comparing with each of the images based on the basic image, and generates the parallax images as left and right video data frames, respectively. The video recording / reproducing system includes: a video generating unit to be recorded in the storage unit; and a video reproducing unit for simultaneously reproducing and displaying the left and right video data recorded in the storage unit. This brings about the effect of capturing and reproducing a high-quality parallax image.

  According to the present technology, it is possible to obtain an excellent effect that it is possible to perform imaging for generating a high-quality stereoscopic image in which the image quality of the left and right images is uniform.

It is a figure showing an example of structure of an image pick-up part in an embodiment of this art. It is a figure showing an example of composition of image sensors 171 thru / or 173 in an embodiment of this art. It is an image figure of the pupil 115 in the imaging part by embodiment of this technique. It is a figure which shows the relationship between the distance D between gravity centers, and a base line length (baseline). It is a figure which shows the relationship between the expansion by zoom and parallax. It is a figure showing an example of 1 composition of an imaging device in a 2nd embodiment of this art. It is a figure showing an example of composition of picture generation part 200 in a 2nd embodiment of this art. It is a figure showing an example of 1 composition of parallax picture generation parts 241 and 242 in a 2nd embodiment of this art. 12 is a flowchart illustrating an operation example of the imaging apparatus according to the second embodiment of the present technology. It is a figure showing an example of composition of picture generation part 200 in a 3rd embodiment of this art. It is a figure showing an example of composition of picture generation part 200 in a 4th embodiment of this art. It is a figure showing an example of composition of time direction interpolation parts 251 and 252 in a 4th embodiment of this art. 22 is a flowchart illustrating an operation example of time direction interpolation units 251 and 252 according to the fourth embodiment of the present technology. It is a figure showing an example of composition of picture generation part 200 in a 5th embodiment of this art. It is a figure showing an example of composition of spatial direction interpolation part 260 in a 5th embodiment of this art. It is a flowchart which shows the operation example of the spatial direction interpolation part 260 in 5th Embodiment of this technique. It is a figure which shows the example of 1 structure of the video recording / reproducing system in 6th Embodiment of this technique.

Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be made in the following order.
1. First embodiment (an example of an imaging unit that generates a basic image and a parallax detection image)
2. Second embodiment (an example of an imaging device that generates a parallax image based on a basic image and a parallax detection image)
3. Third embodiment (an example of an imaging device that generates a parallax image based on a raw image of a basic image and a parallax detection image)
4). Fourth embodiment (an example of an imaging apparatus performing time direction interpolation)
5. Fifth embodiment (an example of an imaging apparatus that performs spatial direction interpolation)
6). Sixth embodiment (example of video recording / playback system)

<1. First Embodiment>
[Structure of imaging unit]
FIG. 1 is a diagram illustrating a structure example of an imaging unit according to an embodiment of the present technology. FIG. 4A is a side cross-sectional view as viewed from the left toward the subject, and FIG. 4B is a top cross-sectional view. In FIG. 5B, the upper part is the right direction toward the subject, and the lower part is the left direction toward the subject. In FIG. 5B, the upper mirror 142, the condenser lens 153, and the image sensor 173 are not shown.

  The imaging unit receives incident light 101 from a subject and forms images on left and right imaging elements 171 and 172 and a central imaging element 173 to generate left and right and central video data. An interchangeable lens 110 is attached to the imaging unit main body via a lens mount 120. The interchangeable lens 110 is a lens group that condenses incident light 101 from a subject. The interchangeable lens 110 is a lens group such as a focus lens for focusing and a zoom lens for enlarging the subject, as well as the interchangeable lens 110. A diaphragm 113 is provided. The pupil 115 is an image of an aperture stop when the lens is viewed from the subject side or the imaging side. The interchangeable lens 110 is an example of a photographing lens described in the claims.

  The lens mount 120 attaches the interchangeable lens 110 to the imaging unit main body. Inside the lens mount 120, the condensed light is once imaged and becomes an inverted image in which the left and right are reversed.

  A relay lens unit 130 is arranged at the next stage of the lens mount 120. The relay lens unit 130 includes a lens that transmits the light collected up to the lens mount 120 to the parallel light region 140. The light condensed up to the lens mount 120 forms an aerial image 131. Due to the lens in the relay lens unit 130, the diffused light from the point light source at the objective focal position (that is, the position of the subject) of the interchangeable lens 110 becomes parallel light in the region 140. In this parallel light region 140, a diaphragm different from the diaphragm 113 may be further provided. The relay lens unit 130 is an example of a relay lens described in the claims.

  A transmissive mirror 141 and mirrors 143 to 146 are arranged at the next stage of the relay lens unit 130. The transmission type mirror 141 and the mirrors 143 to 146 are arranged in the position of the parallel light region 140 and are spectroscopic mirrors that split the collected light. First, the entire light quantity of the same subject is transmitted and reflected by a transmission type mirror 141 having an arbitrarily set reflectance, and is split into two. In this case, the images included in the transmitted and reflected light are the same.

  The transmissive mirror 141 is a mirror that reflects and transmits received light. The transmission mirror 141 can be realized by, for example, an AR coating (Anti-Reflection Coat) in which a thin film is deposited on the glass surface. In this AR coating, the transmittance can be controlled by controlling the reflectance of the thin film. For example, when a half mirror with 50% transmission and 50% reflection is used, 50% of the incident light is reflected on the mirror 142 side and 50% of the incident light is transmitted on the mirrors 143 to 146 side.

  The light reflected by the transmissive mirror 141 is reflected by the mirror 142 installed in the upward direction and enters the condenser lens 153. Of the light transmitted through the transmissive mirror 141, the component of the light viewed from the left side toward the subject is reflected by the mirrors 143 and 145 for left-right reversal. Similarly, the light component viewed from the right side toward the subject among the light transmitted through the transmission type mirror 141 is reflected by the mirrors 144 and 146. Thus, the light transmitted through the transmission type mirror 141 is split left and right by the mirrors 142 to 146. Since the positions where the mirrors 143 and 144 are disposed are included in the parallel light region 140, they are split into light components viewed from the left side and the right side, respectively. As in the above example, when 50% of the incident light is transmitted to the mirrors 143 to 146, the transmitted light is split by the mirrors 143 and 144, and the amount of light becomes 25% of the incident light. .

  The light split by the transmission mirror 141 and the mirrors 143 and 145, that is, the light of the component when the subject is viewed from the left side is incident on the focusing lens 151 for image formation. Further, the light split by the transmission mirror 141 and the mirrors 144 and 146, that is, the light of the component when the subject is viewed from the right side, is incident on the focusing lens 152 for image formation. The light reflected by the transmissive mirror 141 and the mirror 142, that is, the light of the component when the subject is viewed from the center, enters the focusing lens 153 for image formation.

  The condensing lenses 151 and 152 image the incident light on the light receiving surfaces of the image sensors 171 and 172, respectively. The condenser lens 153 forms an image of the incident light on the light receiving surface of the image sensor 173. Each of the light incident on the image sensors 171 to 173 becomes an erect image. Here, as an example, the light transmitted through the transmissive mirror 141 is split left and right and the reflected light is formed into a single image, but these may be interchanged. That is, the light reflected by the transmissive mirror 141 may be divided into left and right, and the transmitted light may be directly imaged. The condensing lenses 151 and 152 are examples of the imaging lens described in the claims.

  The image sensors 171 to 173 are photoelectric conversion elements that convert light incident from the condenser lenses 151 to 153 into electronic signals, respectively. The image sensors 171 to 173 are realized by, for example, a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS) image sensor, or the like.

  The image sensors 171 and 172 are used to generate a parallax detection image for detecting stereoscopic parallax information. The image sensor 173 is used to generate a basic image that is a basis of a parallax image used for stereoscopic viewing. The basic image is required to have higher image quality than the parallax detection image. This is because the parallax detection image is for detecting parallax information, while the image quality of the basic image is reflected in the image quality of the parallax image.

  FIG. 2 is a diagram illustrating a configuration example of the imaging elements 171 to 173 according to the embodiment of the present technology. FIG. 4A shows a configuration example of the image sensor 173, and FIG. 4B shows a configuration example of the image sensors 171 and 172. In this example, the image sensors 171 and 172 for generating the parallax detection image have the number of pixels of 1920 pixels in the horizontal direction and 1080 pixels in the vertical direction, respectively. On the other hand, the image sensor 173 for generating a basic image has the number of pixels of 3840 pixels in the horizontal direction and 2160 pixels in the vertical direction. Both of these have a ratio of horizontal 16: vertical 9. However, the present technology can be applied even when the ratio between the two is not equal.

  The sizes of the image sensors 171 and 172 are, for example, ½ inch diagonal. On the other hand, the size of the image sensor 173 is, for example, 2/3 inch diagonal.

  By setting the resolutions of the imaging elements 171 to 173 in this way, it is possible to generate a basic image with higher image quality than the parallax detection image.

  In addition, the image generation rate in the image sensors 171 and 172 and the image generation rate in the image sensor 173 may be the same or different. When the number of pixels of the image sensor 173 is higher than that of the image sensors 171 and 172, it is conceivable to set the image generation rate of the image sensor 173 low. In the case of the number of pixels described above, for example, the image generation rate of the image sensors 171 and 172 may be 240 frames / second, and the image generation rate of the image sensor 173 may be 120 frames / second. Thereby, the spatial resolution of the image sensor 173 and the time resolution of the image sensors 171 and 172 can be used properly.

  The standard frame rate is 24 frames per second (24 Hz) for movies and 60 fields per second (60 Hz) for television. In the embodiment of the present technology, in consideration of motion blur and jerkiness, a captured image is generated from an electronic signal at a rate of 60 frames per second (60 Hz) or more, preferably 230 to 250 frames per second (240 Hz ± 10 Hz). To do. This eliminates the lack of resolution in the time direction. Furthermore, when considering the broadcasting system, the common multiple of 50 frames per second (50 Hz) used in Europe and 60 frames per second (60 Hz) used in Japan / America is 290 to 310 frames per second (300 Hz ± 10 Hz). Rate is preferred. Furthermore, as a common multiple obtained by adding 24 frames per second (24 Hz) of a movie, the rate of 590 to 610 frames per second (600 Hz ± 10 Hz) is also easy to perform image processing such as image composition and rate conversion processing. It is important. Therefore, it is desirable to take these into consideration when setting the image generation rates of the image sensors 171 to 173.

[Split pupil]
FIG. 3 is an image diagram of the pupil 115 in the imaging unit according to the embodiment of the present technology. A pupil is an image of an aperture stop when the lens is viewed from the subject side or the imaging side. In the imaging unit according to the embodiment of the present technology, when the radius of a circle corresponding to the pupil 115 is r, the following equation is established.
2r = f / F (Formula 1)
However, f is a focal length and F is an F value. Therefore, it can be seen that when the focal length is fixed, the diameter 2r of the pupil 115 and the F value are in an inversely proportional relationship.

In the embodiment of the present invention, in order to disperse the collected light to the left and right at the position of the parallel light region, the left semicircle and the right semicircle obtained by dividing the circle of the pupil 115 into the left and right are considered. As described above, the stereoscopic effect is obtained based on the parallax between the eyes (relative parallax). At this time, it is considered that the optical axis that determines the parallax passes through the center of gravity of each of the left and right semicircles. The center of gravity of the semicircle with radius r can be obtained geometrically and is located at a distance of 4r / 3π from the center of the circle. Therefore, it can be seen that the distance between the centroid 501 of the left semicircle and the centroid 502 of the right semicircle (distance D between centroids) is as follows.
D = 8r / 3π (Formula 2)
That is, it can be seen that the distance D between the centers of gravity is reduced in proportion to the stop 113. In other words, the resulting three-dimensional effect can be adjusted by changing the aperture of the diaphragm 113. The results of experiments conducted to confirm these assumptions are shown below.

[Relationship between center of gravity distance and baseline length (baseline)]
FIG. 4 is a diagram illustrating a relationship between the distance D between the centroids and the baseline length (baseline). Here, the theoretical value of the distance between the centers of gravity and the experimental value of the baseline length are shown for the two types of lenses #A and #B as the interchangeable lens 110.

  The lens #A is a zoom lens having an open F value of 1.8 and a focal length of 10 to 100 [mm (millimeter)]. The zoom ratio of this lens #A is 10 times, and the focal length at the wide end is 10 [mm]. The lens #B is a zoom lens having an open F value of 2.8 and a focal length of 13.5 to 570 [mm]. The zoom ratio of this lens #B is 42 times, and the focal length at the wide end is 13.5 [mm]. In both cases, the shooting distance is assumed to be 6.5 [m (meter)].

  From the above formulas 1 and 2, the distances D between the centers of gravity of the lenses #A and #B are calculated to be 23.1 [mm] and 14.9 [mm], respectively. On the other hand, the baseline lengths obtained by experiments in the actual apparatus were 20.0 [mm] and 12.0 [mm] for lenses #A and #B, respectively. From this experimental result, although a decrease from the theoretical value presumed to be caused by the diffraction effect is observed, it can be seen that the distance D between the centers of gravity of the semicircles of the pupil, which is the image of the stop 113, is almost equal to the baseline length. . Further, from the above-described formula 2, it is shown that the center-of-gravity distance D can be changed by the aperture of the diaphragm 113, and therefore the base line length can also be controlled by the aperture of the diaphragm 113.

  According to the configuration example in the embodiment of the present technology, the minimum value of the distance D between the centroids can be assumed to be approximately 7 [mm]. It is considered that a stereoscopic effect can be felt if the base line length is a value of this level. In particular, when the shooting distance is long, it is considered that the stereoscopic effect cannot be obtained unless the baseline length is long. When the baseline length is increased, the stereoscopic effect becomes clearer when the length is about 32 [mm]. On the other hand, the background blur increases. When the base line length exceeds 65 [mm], it is considered that a miniature garden effect occurs and the image is unnatural. Therefore, it is considered that the base line length is about 7 to 65 [mm] as a range that naturally appears as a stereoscopic image.

[Relationship between zoom magnification and parallax]
FIG. 5 is a diagram illustrating a relationship between zooming and parallax. In FIG. 5A, the left eye position is L, the right eye position is R, and points on the subject are A and B. When the angle LAR (absolute parallax) that anticipates point _A is the convergence angle θ _A of point _A and the angle LBR (absolute parallax) that anticipates point _B is the convergence angle θ _{B of} point B, the parallax between points A and B (Relative parallax) d is given by the following equation. Hereinafter, the relative parallax is simply referred to as “parallax”.
d = θ _B −θ _A
Here, when the angle ALB is h and the angle ARB is g, the convergence angle θ _A is substantially equal to h, and the convergence angle θ _B is approximately equal to g. Therefore, the following equation holds.
d = g−h

Moreover, the interocular distance D, the distance D _A from both eyes to the point _A, the distance from the eyes to the point B and D _B, and the distance between points A and B as seen from both eyes δ Then
d≈Dδ / (D _A ² −δD _A )
Here, from D _A , DB >> D,
d≈Dδ / D _A ²
Holds.

FIG. 5B is a diagram illustrating a positional relationship when n-fold enlargement is performed as compared with FIG. Here, a dash is added to the end of each symbol for the angle, position, and distance changed after zooming. Because it is n times magnification
g ′ = ng
h ′ = nh
It becomes. At this time, the parallax d ′ is expressed as the following equation.
d '= D _B ' -D _A '
= G'-h '
= N (g-h)
= Nd
That is, n-fold parallax occurs by n-fold enlargement. This means that the stereoscopic effect increases when zooming to the tele end side. In other words, in zoom shooting, an appropriate parallax can be obtained even with a short baseline length.

  As described above, according to the first embodiment of the present technology, the parallax of an image presented to both eyes is obtained by splitting the light collected by one interchangeable lens 110 left and right by the mirrors 143 to 146. Can be appropriately reduced. The parallax obtained in the embodiment of the present technology can be controlled by the aperture of the diaphragm 113 and the zoom ratio (magnification ratio) in zoom photography. In general, the sensitivity of the eye to parallax is high, and the normal visual acuity is in the order of minutes in terms of viewing angle, whereas parallax has a resolution that is one order higher (CWTyler, "Spatial Organization of Binocular Disparity Sensitivity", Vision Research, Vol. 15, pp.583-590, Pergamon Press (1975). Therefore, appropriately reducing the parallax is important for reducing the visual fatigue by allowing the user to naturally perceive a three-dimensional effect even under a parallax condition smaller than the above example.

  Further, according to the first embodiment, a high-quality basic image can be captured in addition to the parallax detection image. By using these basic images and parallax detection images, high-quality parallax images can be generated as follows.

<2. Second Embodiment>
[Configuration of imaging device]
FIG. 6 is a diagram illustrating a configuration example of the imaging device according to the second embodiment of the present technology. The video recording / playback system includes an imaging unit 100, a video generation unit 200, and a video storage unit 300.

  The imaging unit 100 is the imaging unit described as the first embodiment. The imaging unit 100 receives incident light from a subject, generates parallax detection images by the imaging elements 171 and 172, and generates a basic image by the imaging element 173.

  The video generation unit 200 generates a high-quality parallax image based on the basic image and the parallax detection image output from the imaging unit 100 and records the video data of the parallax image in the video storage unit 300. The configuration of the video generation unit 200 will be described later.

  The video storage unit 300 stores video data of a parallax image output from the video generation unit 200.

[Configuration of video generator]
FIG. 7 is a diagram illustrating a configuration example of the video generation unit 200 according to the second embodiment of the present technology. The video generation unit 200 includes signal processing units 211 to 213, image memories 221 and 222, parallax image generation units 241 and 242, and encoding units 231 and 232.

  The signal processing units 211 to 213 receive the video data of the basic image and the parallax detection image output from the imaging unit 100, respectively, and perform predetermined signal processing. The signal processing units 211 to 213 perform A / D (Analog to Digital) conversion on the captured image data and perform various signal processing such as demosaic processing and white balance adjustment processing.

  The parallax image generation units 241 and 242 generate parallax images based on the basic image and the parallax detection image. Details of the parallax image generation units 241 and 242 will be described later.

  The image memories 221 and 222 are memories that temporarily hold the video data processed by the signal processing units 211 to 213 and the video data of the parallax images generated by the parallax image generation units 241 and 242.

  The encoding units 231 and 232 encode video data of parallax images held in the image memories 221 and 222 and output the encoded video data to the video storage unit 300.

[Configuration of parallax image generation unit]
FIG. 8 is a diagram illustrating a configuration example of the parallax image generation units 241 and 242 according to the second embodiment of the present technology. Each of the parallax image generation units 241 and 242 includes a corresponding pixel search unit 245 and a pixel moving unit 246.

  The corresponding pixel search unit 245 searches for a corresponding pixel in the basic image for each pixel of the parallax detection image. The corresponding pixel search unit 245 provides a search area in each of the left and right images of the parallax detection image, and searches using a partial area of the basic image as a template. In this search, a correlation coefficient method, SSDA (Sequential Similarity Detection Algorithm) method, least square matching, or the like is used. That is, evaluation by correlation calculation, absolute difference sum calculation, or the like is performed for each fixed unit of pixels, and a pixel having the maximum correlation or a pixel having the minimum difference absolute sum is searched as a corresponding pixel.

  Here, for example, when searching for a corresponding pixel using a sum of squares of differences or a sum of absolute differences, evaluation is performed as follows. First, a search area of horizontal p pixels × vertical q pixels in a parallax detection image of horizontal X pixels × vertical Y pixels is assumed. Also, a search area of horizontal m pixels × vertical n pixels in the basic image is assumed. However, m is an integer smaller than p and n is an integer smaller than q. In the search area of the parallax detection image, the difference square sum or the difference absolute value sum is obtained while sequentially moving the search area of the basic image.

また、差分絶対値和（ＳＡＤ：Sum of Absolute Difference）は次式により求められる。

この差分二乗和または差分絶対値和が最小となる位置のテンプレートの中心に位置する画素が対応画素になる。差分絶対値和が最小となるのは、基本画像と視差検出画像とが最も類似している領域であるから、対応画素には同じ被写体が表示されているものと推定することができる。 When the pixel value of the parallax detection image is I (i, j) and the pixel value of the basic image is T (i, j), a sum of squared difference (SSD) is obtained by the following equation.

Also, the sum of absolute differences (SAD) is obtained by the following equation.

The pixel located at the center of the template at the position where the difference square sum or the difference absolute value sum is minimum is the corresponding pixel. Since the difference absolute value sum is the smallest in the region where the basic image and the parallax detection image are most similar, it can be estimated that the same subject is displayed in the corresponding pixel.

  On the other hand, for example, when searching for corresponding pixels using normalized cross-correlation, evaluation is performed as follows. The relationship between the search area for the parallax detection image and the search area for the basic image is the same as described above. In the search area of the parallax detection image, the normalized cross correlation is obtained while sequentially moving the search area of the basic image.

この正規化相互相関が最大となる位置のテンプレートの中心に位置する画素が対応画素になる。この正規化相互相関を利用することにより、視差検出画像と基本画像とに輝度差がある場合においても、高い精度で対応画素を検出することができる。 Normalized correlation coefficient (NCC) is obtained by the following equation.

The pixel located at the center of the template where the normalized cross-correlation is maximized is the corresponding pixel. By utilizing this normalized cross-correlation, even when there is a luminance difference between the parallax detection image and the basic image, the corresponding pixel can be detected with high accuracy.

  The pixel moving unit 246 generates a parallax image by moving the pixel of the basic image to the position of the corresponding pixel of the parallax detection image based on the information of the corresponding pixel searched by the corresponding pixel search unit 245. . When the number of pixels of the basic image and the parallax detection image does not match, the pixel value of the parallax image may be obtained by interpolation from a plurality of pixels of the basic image.

  In this embodiment, only the parallax information is used for the parallax detection image, and the image quality such as brightness and resolution reflects that of the basic image. Thereby, parallax images with uniform image quality on the left and right can be obtained.

[Operation of imaging device]
FIG. 9 is a flowchart illustrating an operation example of the imaging apparatus according to the embodiment of the present technology. First, incident light from the subject is guided to the parallel light region 140 via the interchangeable lens 110 and the relay lens unit 130, and is separated by the transmission type mirror 141 and the mirrors 143 to 146 (step S901).

  The light guided to the condenser lens 153 by the transmissive mirror 141 and the mirror 142 is photoelectrically converted by the image sensor 173 to generate a basic image (step S902). Further, the light guided to the condenser lenses 151 and 152 by the mirrors 143 to 146 is photoelectrically converted by the image sensors 171 and 172, and a parallax detection image is generated (step S902). The basic image and the parallax detection image are generated independently, and the generation rate of each image may be different.

  In addition, the corresponding pixel search unit 245 of the parallax image generation units 241 and 242 searches for a corresponding pixel between the basic image and the parallax detection image (step S903). This search for corresponding pixels is performed for each of the left and right parallax detection images.

  Then, the pixel moving unit 246 of the parallax image generating units 241 and 242 generates a parallax image by moving the pixel of the basic image to the position of the corresponding pixel of the parallax detection image based on the information of the corresponding pixel ( Step S904).

  As described above, according to the second embodiment, by generating a parallax image using pixels of one high-quality basic image, the image quality such as brightness and resolution is reflected in the parallax image. It is possible to obtain a parallax image with uniform image quality between the left and right.

<3. Third Embodiment>
In the second embodiment, the parallax image is generated based on the basic image and the parallax detection image held in the image memories 221 and 222. In the third embodiment, the parallax image is held in the image memories 221 and 222. A parallax image is generated before being processed. That is, in the third embodiment, a parallax image is generated without using the signal processing unit 213 in a state where the basic image and the parallax detection image are RAW images. The overall configuration of the imaging device is the same as that described with reference to FIG.

[Configuration of video generator]
FIG. 10 is a diagram illustrating a configuration example of the video generation unit 200 according to the third embodiment of the present technology. The video generation unit 200 in the third embodiment is configured by eliminating the signal processing unit 213 in FIG. 7 and connecting the parallax image generation units 241 and 242 to the signal processing units 211 and 212, respectively.

  The signal processing units 211 and 212 receive the basic image and the parallax detection image video data output from the imaging unit 100, respectively, perform A / D conversion, and supply them to the parallax image generation units 241 and 242, respectively.

  The parallax image generation units 241 and 242 generate parallax images based on the basic image and the video data of the parallax detection images supplied from the signal processing units 211 and 212, respectively, and supply the parallax images to the signal processing units 211 and 212, respectively. Note that the configuration of the parallax image generation units 241 and 242 is the same as that described with reference to FIG.

  Then, the signal processing units 211 and 212 perform various signal processing such as demosaic processing and white balance adjustment processing on the parallax images supplied from the parallax image generation units 241 and 242, and hold them in the image memories 221 and 222, respectively. Let

  The encoding units 231 and 232 encode the video data of the parallax images held in the image memories 221 and 222 and output the encoded video data to the video storage unit 300.

  Thus, according to the third embodiment, a higher-quality parallax image can be obtained by generating a parallax image in a state where the basic image and the parallax detection image are RAW images.

<4. Fourth Embodiment>
In the fourth embodiment, an image at an arbitrary time is generated by interpolating parallax images in the time direction. The overall configuration of the imaging device is the same as that described with reference to FIG.

[Configuration of video generator]
FIG. 11 is a diagram illustrating a configuration example of the video generation unit 200 according to the fourth embodiment of the present technology. The video generation unit 200 according to the fourth embodiment is different from the second embodiment in that the time direction interpolation units 251 and 252 are connected to the image memory 221 or 222, respectively. The time direction interpolation units 251 and 252 perform time direction interpolation on the video data of the parallax images held in the image memories 221 and 222, respectively.

[Configuration of time direction interpolation unit]
FIG. 12 is a diagram illustrating a configuration example of the time direction interpolation units 251 and 252 in the fourth embodiment of the present technology. Each of the time direction interpolation units 251 and 252 includes a frame buffer 254, a corresponding pixel search unit 255, and an interpolation processing unit 256.

  The frame buffer 254 is a buffer that holds video data of a parallax image for each frame. The video data of the parallax image is composed of time-series frames, and by temporarily holding the frames in the frame buffer 254, the corresponding pixel search unit 255 can compare different frames. For example, by holding the frame immediately before the video data of the parallax image in the frame buffer 254, both the current frame supplied from the image memory 221 or 222 and the frame immediately before it are stored in the corresponding pixel search unit 255. Can be supplied. Note that since there are left and right images in the parallax image, the frame buffer 254 needs to hold at least two frames.

  The corresponding pixel search unit 255 searches for corresponding pixels in both frames by comparing the current frame of the video data of the parallax image and the frame held in the frame buffer 254. The corresponding pixel search unit 255 provides a search region in each of the left and right images of the parallax image held in the frame buffer 254, and searches using a partial region of the current parallax image as a template. In this search, a method similar to that of the above-described corresponding pixel search unit 245 can be adopted, and thus detailed description thereof is omitted.

  The interpolation processing unit 256 generates a pixel value at an arbitrary time by interpolation based on information on the corresponding pixel searched by the corresponding pixel search unit 255. Here, the parallax image held in the frame buffer 254 is set as past data, and the current parallax image supplied from the image memory 221 or 222 is set as current data. First, the pixel value s (u, v) is calculated from the average value of the two corresponding pixels at the midpoint coordinates (u, v) of the movement vector connecting the corresponding pixels from the past data to the current data.

The uv space in this case does not necessarily match the XY space of the parallax image. Therefore, the pixel value F (X, Y) in the XY space is generated by interpolation processing from the pixel values s (u, v) of the entire area. As the interpolation processing in this case, for example, the following linear interpolation method can be employed. Here, F ₀ is a pixel value F (X, Y) to be obtained, s _i is a known pixel value s (u, v), and t is the number of adjacent points used in the calculation. Further, w _i is a weight, and the reciprocal of the distance to the coordinates (X, Y) can be used.

  Here, the pixel value at the midpoint of the movement vector is obtained by calculating the average value of the pixel values of the corresponding pixels of the past data and the current data. Pixel values at the time may be generated.

[Operation of time direction interpolation unit]
FIG. 13 is a flowchart illustrating an operation example of the time direction interpolation units 251 and 252 according to the fourth embodiment of the present technology. The time direction interpolation units 251 and 252 repeat the following process for each frame rate of the parallax image (L910).

  First, a new parallax image frame f_new (X, Y) is acquired from the image memory 221 or 222 (step S911). In addition, the frame held in the frame buffer 254 is set to f_old (X, Y) (step S912). Then, the corresponding pixel search unit 255 searches for a corresponding pixel in both frames by comparing the frame f_new (X, Y) and the frame f_old (X, Y) of the parallax image (step S913).

  The interpolation processing unit 256 generates the midpoint coordinates (u, v) of the movement vector connecting the searched corresponding pixels (step S914), and calculates the pixel value s (u, v) from the average value of the pixel values of the two corresponding pixels. v) is calculated (step S915). Then, the interpolation processing unit 256 obtains a pixel value F (X, Y) by a linear interpolation method or the like (step S916). The pixel value F (X, Y) is output as an intermediate time image (step S917).

  Then, before the processing of the next frame of the parallax image, the frame f_new (X, Y) of the parallax image is held in the frame buffer 254 (step S918).

  Thus, according to the fourth embodiment, the frame rate of a parallax image can be improved. Here, although an embodiment in which a parallax image is interpolated in the time direction has been described, a parallax detection image or a basic image may be interpolated.

<5. Fifth embodiment>
In the fifth embodiment, a parallax image is interpolated in the spatial direction to generate a viewpoint image at an arbitrary position. The overall configuration of the imaging device is the same as that described with reference to FIG.

[Configuration of video generator]
FIG. 14 is a diagram illustrating a configuration example of the video generation unit 200 according to the fifth embodiment of the present technology. The video generation unit 200 according to the fifth embodiment is different from the second embodiment in that a spatial direction interpolation unit 260 is connected to the image memories 221 and 222. The spatial direction interpolation unit 260 performs spatial direction interpolation on the video data of the parallax images held in the image memories 221 and 222.

[Configuration of spatial direction interpolation unit]
FIG. 15 is a diagram illustrating a configuration example of the spatial direction interpolation unit 260 according to the fifth embodiment of the present technology. The spatial direction interpolation unit 260 includes a corresponding pixel search unit 265 and an interpolation processing unit 266.

  The corresponding pixel search unit 265 searches for a corresponding pixel in both frames by comparing the left image frame and the right image frame of the parallax image. The corresponding pixel search unit 265 provides a search area in one of the left and right images, and searches using a partial area of the other image as a template. In this search, a method similar to that of the above-described corresponding pixel search unit 245 can be adopted, and thus detailed description thereof is omitted.

  The interpolation processing unit 266 generates the pixel value at the viewpoint at an arbitrary position based on the information on the corresponding pixel searched by the corresponding pixel search unit 265 by interpolation. Here, one image is set as first viewpoint data, and the other image is set as second viewpoint data. First, the pixel value s (u, v) is calculated from the average value of the two corresponding pixels at the midpoint coordinates (u, v) of the vector connecting the corresponding pixels from the first viewpoint data to the second viewpoint data. Is done.

  The uv space in this case does not always match the XY space of the parallax image as in the case of the fourth embodiment. Therefore, the pixel value F (X, Y) in the XY space is generated by interpolation processing from the pixel values s (u, v) of the entire area. The contents of the interpolation processing are the same as in the case of the fourth embodiment, and detailed description thereof is omitted.

  Here, the pixel value at the midpoint of the vector is obtained by calculating the average value of the pixel values of the corresponding pixels of the first viewpoint data and the second viewpoint data. However, a point with an arbitrary ratio is calculated with a linear ratio. Thus, a pixel value at a viewpoint at an arbitrary position may be generated.

[Operation of spatial direction interpolation unit]
FIG. 16 is a flowchart illustrating an operation example of the spatial direction interpolation unit 260 in the fifth embodiment of the present technology. The spatial direction interpolation unit 260 repeats the following processing for each frame rate of the parallax image (L920).

  First, the frame f_left (X, Y) of the left image of a new parallax image is acquired from the image memory 221 (step S921). In addition, the frame f_right (X, Y) of the right image of the new parallax image is acquired from the image memory 222 (step S922). Then, the corresponding pixel search unit 265 searches for a corresponding pixel in both frames by comparing the frame f_left (X, Y) and the frame f_right (X, Y) of the parallax image (step S923).

  The interpolation processing unit 266 generates the midpoint coordinates (u, v) of the vectors connecting the searched corresponding pixels (step S924), and calculates the pixel value s (u, v) from the average value of the pixel values of the two corresponding pixels. ) Is calculated (step S925). Then, the interpolation processing unit 266 obtains the pixel value F (X, Y) by a linear interpolation method or the like (step S926). This pixel value F (X, Y) is output as an intermediate time image (step S927).

  Thus, according to the fifth embodiment, a parallax image at a viewpoint at an arbitrary position can be generated. Although it is assumed here that the corresponding pixels are searched based on the left image and the right image of the parallax image, the corresponding pixels in both images are searched by comparing the basic image with one of the left and right images of the parallax image. You may make it do.

<6. Sixth Embodiment>
[Configuration of video recording / playback system]
FIG. 17 is a diagram illustrating a configuration example of a video recording / reproducing system according to the sixth embodiment of the present technology. This video recording / playback system includes an imaging unit 100, a video generation unit 200, a video storage unit 300, a video playback unit 400, and a display unit 500. The imaging unit 100, the video generation unit 200, and the video storage unit 300 are the same as those described with reference to FIG.

  The video playback unit 400 reads and plays back video data stored in the video storage unit 300. The video reproduction unit 400 includes decoding units 411 and 412 and display control units 421 and 422 corresponding to the video data of the left and right parallax images. The decoding units 411 and 412 decode the video data of the parallax image read from the video storage unit 300. The display control units 421 and 422 are for controlling the video data of the parallax images decoded by the decoding units 411 and 412 to be displayed on the display unit 500.

  The display unit 500 displays the video data of the parallax image output from the video playback unit 400. As this display unit 500, for example, a circularly polarized light or linearly polarized light filter is attached to two projectors, and parallax images for left and right eyes are respectively presented, and viewed with circularly polarized light or linearly polarized glasses corresponding to the display. Embodiments are possible. Similarly, in a flat panel display with a filter, a parallax image for the left and right eyes may be simultaneously presented, and a lenticular lens, a stereoscopic display device without glasses such as a parallax barrier method, or the like may be used. As described above, in the embodiment of the present technology, visual fatigue is reduced by presenting images for the left and right eyes alternately instead of presenting them alternately.

  Further, in the embodiment of the present technology, motion blur and jerkiness are increased by increasing the frame rate from generation of video data in the imaging unit 100 to display of video data in the display unit 500. To solve the problem. In addition to a decrease in MTF (Modulation Transfer Function) at the time of shooting, blurring due to movement occurs more often due to slipping on the retina of the image, especially when tracking a moving subject in a hold-type display (following vision). . Here, the hold-type display means that an image is continuously displayed on a film, a liquid crystal projector, or the like during a frame period. In addition, jerkiness means that the smoothness of the image is lost and the movement becomes jerky. This jerkiness often occurs when an image shot using a high-speed shutter is viewed with a fixed line of sight (fixed vision). Such degradation in moving image quality involves the frame rate of shooting and display, the aperture ratio (opening time / frame time) of camera shooting, visual characteristics, and the like.

  As described above, according to the sixth embodiment of the present technology, a high-quality parallax image can be captured and reproduced.

  The above-described embodiment shows an example for embodying the present technology, and the matters in the embodiment and the invention-specific matters in the claims have a corresponding relationship. Similarly, the invention specific matter in the claims and the matter in the embodiment of the present technology having the same name as this have a corresponding relationship. However, the present technology is not limited to the embodiment, and can be embodied by making various modifications to the embodiment without departing from the gist thereof.

In addition, this technique can also take the following structures.
(1) a photographic lens that collects light from the subject;
A relay lens that transmits the collected light to be parallel light;
A semi-transmissive film that transmits and reflects the parallel light;
A mirror that splits light guided by being transmitted or reflected by the semi-transmissive film into left and right two parts;
First and second imaging lenses that respectively image the dispersed light;
First and second imaging elements for converting light imaged by the first and second imaging lenses into parallax detection images by electronic signals, respectively;
A third imaging lens that forms an image of light reflected or transmitted by the semi-transmissive film but not guided to the mirror;
An image pickup apparatus comprising: a third image pickup device that converts light imaged by the third image forming lens into a basic image using an electronic signal.
(2) The apparatus further includes a parallax image generation unit that generates a parallax image at the same viewpoint as each of the parallax detection images by comparing the basic image and the parallax detection images based on the basic image. The imaging device according to 1).
(3) The parallax image generation unit
A corresponding pixel search unit for searching for pixels corresponding to each other by comparing each of the basic image and the parallax detection image;
The imaging apparatus according to (2), further including: a pixel moving unit that generates the parallax image by moving a pixel of the basic image to a position of a corresponding pixel in the parallax detection image based on the search result.
(4) The video data including the basic image or the parallax detection image as a time-series frame is searched for corresponding pixels by comparing two frames at different times, and the corresponding is performed based on the search result. The imaging apparatus according to (2) or (3), further including a time direction interpolation unit that interpolates an image at an arbitrary time by obtaining a middle point coordinate of the pixel and a pixel value thereof.
(5) Comparing any two of the basic image and the parallax detection image to search for corresponding pixels, and based on the search result, determine the midpoint coordinates of the corresponding pixels and their pixel values. The imaging apparatus according to (2) or (3), further including a spatial direction interpolation unit that interpolates an image of an arbitrary viewpoint by obtaining.
(6) The imaging device according to any one of (1) to (5), wherein each of the first to third imaging elements generates the basic image or the parallax detection image at a rate of 60 frames or more per second. .
(7) The imaging device according to any one of (1) to (6), wherein each of the first and second imaging elements generates the parallax detection image at a rate of 230 to 250 frames per second.
(8) The imaging device according to any one of (1) to (6), wherein each of the first and second imaging elements generates the parallax detection image at a rate of 290 to 310 frames per second.
(9) The imaging device according to any one of (1) to (6), wherein each of the first and second imaging elements generates the parallax detection image at a rate of 590 to 610 frames per second.
(10) a photographic lens that collects light from the subject;
A relay lens that transmits the collected light to be parallel light;
A semi-transmissive film that transmits and reflects the parallel light;
A mirror that splits light guided by being transmitted or reflected by the semi-transmissive film into left and right two parts;
First and second imaging lenses that respectively image the dispersed light;
First and second imaging elements for converting light imaged by the first and second imaging lenses into parallax detection images by electronic signals, respectively;
A third imaging lens that forms an image of light reflected or transmitted by the semi-transmissive film but not guided to the mirror;
A third imaging element that converts light imaged by the third imaging lens into a basic image by an electronic signal;
A parallax image generating unit that generates a parallax image of the same viewpoint as each of the parallax detection images by comparing the basic image and each of the parallax detection images;
A video generation unit for generating the parallax images as left and right video data frames and recording the frames in a storage unit;
A video recording / reproducing system comprising: a video reproducing unit that simultaneously reproduces and displays the left and right video data recorded in the storage unit.

DESCRIPTION OF SYMBOLS 100 Image pick-up part 110 Interchangeable lens 115 Pupil 120 Lens mount 130 Relay lens part 140 Parallel light area 141 Transmission type mirror 142-146 Mirror 151-153 Condensing lens 171-173 Image pick-up element 200 Image | video production | generation part 211-213 Signal processing part 221, 222 Image memory 231, 232 Coding unit 241, 242 Parallax image generation unit 245, 255, 265 Corresponding pixel search unit 246 Pixel moving unit 251 Time direction interpolation unit 254 Frame buffer 256, 266 Interpolation processing unit 260 Spatial direction interpolation unit 300 Video Storage unit 400 Video playback unit 411, 412 Decoding unit 421, 422 Display control unit 500 Display unit

Claims (12)

A mirror that splits incident light into a first component and a second component;
An imaging device that forms an image of the first component and obtains a first image signal used when generating a parallax detection image;
An image pickup apparatus comprising: another image pickup device that forms an image of the second component and obtains a second image signal used when generating a basic image that is a higher quality image than the parallax detection image. The imaging device according to claim 1, wherein the mirror is a transmissive mirror, and splits the incident light into the first component and the second component by transmitting or reflecting the incident light. The imaging device according to claim 1, wherein the imaging element that obtains the first image signal obtains a left image signal and a right image signal having parallax.   The imaging apparatus according to claim 1, further comprising an image generation unit that generates an image based on the first image signal and the second image signal. A mirror that splits incident light into a first component and a second component; an imaging device that forms an image of the first component to obtain a first image signal used when generating a parallax detection image; The second component is imaged and output from an imaging device including another imaging element that obtains a second image signal used when generating a basic image that is a higher quality image than the parallax detection image. An image processing apparatus comprising an image generation unit that generates an image based on a first image signal and the second image signal. The image generation unit generates the parallax detection image based on the first image signal, generates the basic image based on the second image signal, and based on the parallax detection image and the basic image The image processing apparatus according to claim 5, wherein the image is generated. The image generation unit
Comparing each of the first image signal and the second image signal to search for corresponding pixel signals;
The image processing apparatus according to claim 5, wherein the image is generated by replacing a pixel signal of the second image signal with a corresponding pixel signal in the first image signal based on a result of the search. The first image signal is a left image signal and a right image signal having parallax obtained from the first component,
The image processing apparatus according to claim 5, wherein the image generation unit generates the image based on the left image signal, the right image signal, and the second image signal.   The image processing apparatus according to claim 5, wherein the image generation unit generates the image at the same viewpoint as the parallax detection image based on the first image signal and the second image signal. The image processing apparatus according to claim 5, wherein the image generation unit generates the image at an arbitrary time based on the first image signal and the second image signal.   The image processing apparatus according to claim 5, wherein the image generation unit generates the image at an arbitrary viewpoint based on the first image signal and the second image signal. A system comprising an imaging device and an image processing device,
The imaging device
A mirror that splits incident light into a first component and a second component;
An imaging device that forms an image of the first component and obtains a first image signal used when generating a parallax detection image;
Another imaging element that forms the second component and obtains a second image signal that is used when generating a basic image that is an image having a higher number of pixels than the parallax detection image;
The image processing apparatus includes:
A system comprising an image generation unit that generates an image based on the first image signal and the second image signal output from the imaging device.

Images