JP6907616B2 - Stereoscopic image imaging / display combined device and head mount device - Google Patents

Description

The present invention relates to an image pickup / display combined device having both functions of capturing and displaying a stereoscopic image, and a head mount device using the same.

An imaging device that records a light field (a set of position, angle, and intensity information on the imaging surface of light rays incident on the imaging surface) and a display device that reproduces the recorded light field using a microlens array are already known. In recent years, it has been put into practical use due to advances in image pickup elements and display elements and advances in data processing technology. In these light field devices, since the image pickup device and the display device are considered to be independent, it is impossible to record and reproduce the light field with one device.

For example, LYTRO, the only light field camera on the market for the general public, has a light field after passing through an imaging lens equivalent to that used in ordinary cameras, that is, on the imaging surface of light rays incident on the imaging surface. It is an imaging device that can record a set of position, angle, and intensity information in. By performing image processing after shooting from the acquired light field information, refocusing, fine adjustment of the viewpoint, etc. are realized. However, the output obtained by image processing is a two-dimensional image like a conventional camera, and it is not possible to observe a stereoscopic image by a simple method from the obtained light field information. In addition, a huge amount of calculation processing is required to obtain a two-dimensional image from the light field information.

Light field displays include the integral display that NHK is continuing to develop and the Near-eye Light Field Displays that nVIDIA has developed. These displays can output data taken by a light field camera or data created by CG as a stereoscopic image. However, enormous calculation processing is required for conversion from CG data to a stereoscopic image and conversion for correcting the difference in the optical system between the light field camera used for shooting and the display.

That is, in most cases, the sampling pattern of the light field data differs between the camera and the display, so when displaying the light field data obtained by shooting with the light field camera as a stereoscopic image on the display, the captured light field It is necessary to perform conversion processing (resampling) from the data to the light field data to be displayed.

For example, Japanese Patent Application Laid-Open No. 2009-518877 (Patent Document 1) discloses a method for resampling the light field data, which can minimize the artifacts generated in the conversion process of the light field data. However, since the sampling patterns of the camera and the display are different, it is not a fundamental solution.

Japanese Patent Application Laid-Open No. 11-122544 (Patent Document 2) includes an imaging panel in which a large number of imaging pixels are arranged in a matrix, and a large number of imaging pixels arranged in front of the imaging panel at predetermined intervals. Disclosed is an imaging / display device including a telescope array panel in which micro telescopes are arranged in a matrix, and a display device in which display pixels are arranged in a matrix between the imaging pixels. In this image pickup / display device, an image in front of a person's field of view can be captured by the image pickup device, and the captured image can be enlarged and displayed by the display device to be displayed as an image on a large screen. However, since the image pickup / display device described in Patent Document 2 is not for capturing and displaying a light field, it can only perform an enlarged display, cannot display a stereoscopic image, and expresses a sense of distance. Insufficient.

Special Table 2009-518877 Japanese Unexamined Patent Publication No. 11-122544

Therefore, an object of the present invention is to eliminate the computational processing that is a problem in current light field cameras and displays, and to provide a new photographic / video experience by integrating a shooting and display device. ..

In order to solve the above-mentioned problems, the device for capturing and displaying a stereoscopic image according to the first aspect of the present invention is a light receiving and displaying element composed of a plurality of light receiving pixels and a plurality of display pixels arranged on a plane. And a plurality of optical systems that are regularly arranged side by side on the pixel arrangement surface side of the light receiving / displaying element and convert light from a subject into light beams of a parallax image group having different parallaxs from each other. A stereoscopic image can be recorded by capturing an image with the corresponding light receiving pixel, and the stereoscopic image is displayed by converting the light beam of the differential image group from the display pixel into a light beam of the stereoscopic image by each of the optical systems. It is characterized by being able to do.

In order to solve the above-mentioned problems, the device for capturing and displaying a stereoscopic image, which is the second aspect of the present invention, includes a light-receiving / light-emitting element composed of a plurality of light-receiving / light-emitting pixels arranged on a plane. It has a plurality of optical systems that are regularly arranged side by side on the pixel arrangement surface side of the light-receiving / light-emitting element and convert light from a subject into light fluxes of a parallax image group having different parallaxs from each other. A stereoscopic image can be recorded by taking an image with the corresponding light-receiving / light-emitting pixel, and the light flux of the differential image group from the light-receiving / light-emitting pixel is converted into a light flux of a stereoscopic image by each of the optical systems. It is characterized in that it can display a stereoscopic image.

In order to solve the above-mentioned problems, the device for capturing and displaying a stereoscopic image according to the third aspect of the present invention has a specific number of light receiving pixels arranged on a plane and a pixel arrangement of the specific number of light receiving pixels. A plurality of light receiving units consisting of a photographing optical system arranged on the surface side, and
It has a plurality of display units including a specific number of display pixels arranged between the light receiving units and arranged on a plane, and a display optical system arranged on the pixel arrangement surface side of the specific number of display pixels. death,
A stereoscopic image can be recorded by converting the light from the subject into light fluxes of parallax image groups having different parallax and taking an image by the plurality of light receiving units.
It is characterized in that a stereoscopic image is displayed by converting the luminous flux of the parallax image group into the luminous flux of a stereoscopic image by the plurality of display units.

The present invention eliminates the need for complicated calculation processing, which is a problem in current light field cameras and displays, and can provide a new photographic / video experience by integrating a shooting and display device. Here, "new photo / video experience" means, specifically, easy shooting / viewing of 1x light field photos / videos that can express all of binocular parallax, congestion, and eye focus sensation, videophone, and videophone. It is possible to make natural eye contact with a video conferencing system.

It is a schematic cross-sectional view which shows the 1st aspect of the apparatus for taking and displaying the stereoscopic image of this invention. It is a schematic diagram which shows the 1st aspect of the device for both image pickup and display of a stereoscopic image of this invention. It is a schematic diagram which shows (a) a state of taking a picture, and (b) a state of displaying by using the apparatus for both image capture and display of a stereoscopic image of this invention. It is a schematic cross-sectional view which shows the 2nd aspect of the apparatus for taking and displaying the stereoscopic image of this invention. It is a schematic diagram which shows the 2nd aspect of the apparatus for both image capturing and displaying a stereoscopic image of this invention. It is a schematic cross-sectional view which shows the 3rd aspect of the apparatus for taking and displaying the stereoscopic image of this invention. It is a schematic diagram which shows the 3rd aspect of the device for both image pickup and display of a stereoscopic image of this invention. It is a schematic diagram which shows an example of the head mount device of this invention. It is a schematic diagram which shows another example of the head mount device of this invention. It is a schematic diagram which shows still another example of the head mount device of this invention.

[1] Device for capturing and displaying stereoscopic images The device for capturing and displaying stereoscopic images of the present invention has a function as a light receiving element (imaging element) and a function as a display element (liquid crystal display element, light emitting element, etc.). It is a light field camera and display equipped with a light receiving / displaying element (device) having both functions. At the time of shooting, the light field information is acquired by the light receiving element, and at the time of observation, the same optical system as at the time of imaging or almost the same. Using the optical system, a stereoscopic image is displayed by the display element based on the acquired light field information. A light-receiving / display-combined element is an element having a light-receiving pixel and a display pixel on the same plane, and by using such an element and an optical system, a complicated conversion is performed when displaying a stereoscopic image from light field information. No processing is required.

Such light-receiving / display elements include (a) an element in which light-receiving pixels and display pixels are alternately arranged, (b) a light-receiving / light-emitting element in which one pixel has both light-receiving and light-emitting functions. (c) This can be realized by an element in which the light receiving element unit and the display element unit are alternately arranged. Here, the light receiving element unit is a minute light receiving element composed of a specific number of light receiving pixels, and the display element unit is a minute display element composed of a specific number of display pixels. The arrangement of the light receiving pixels and the display pixels of the aspect (a) or the arrangement of the light receiving element units and the display element units of the aspect (c) as a whole while maintaining disorder even in the method of arranging them alternately. It may be a method of evenly arranging. In the present application, the light receiving pixel and the display pixel are one pixel having a light receiving function and a display function, respectively, and the light receiving element and the display element are formed by arranging a plurality of light receiving pixels and display pixels, respectively. It is a device. As the light receiving pixel, for example, a photosensor such as a photodiode or a phototransistor can be used, and as the display pixel, a light emitting pixel such as an LED or an organic EL can be used in addition to the liquid crystal display pixel.

When the elements of aspects (a) and (b) are used, an optical system in which the light receiving pixel and the display pixel (or the light receiving / emitting combined pixel) are common can be used, so that the same conditions as those at the time of imaging can be used. A stereoscopic image can be displayed with. In the case of aspect (b), the configuration is ideal because it is optically the same when receiving light and when emitting light. However, in aspect (b), since the light receiving pixel and the light emitting pixel are common, in principle, light receiving (imaging) and light emitting (display) cannot be performed at the same time. Further, in the cases of the mode (a) and the mode (c), since the light receiving pixel and the display pixel, or the light receiving unit and the display unit are adjacent to each other, when the light receiving (imaging) and the display are performed at the same time, the display light (light emission). The element or the backlight for liquid crystal display) may act as stray light on the light receiving pixels, causing fog in imaging. However, for example, by shifting the timing between the light receiving operation and the display operation and switching between the light receiving operation and the display operation in a very short time, it is possible to substantially perform imaging and display at the same time.

When the element of aspect (c) is used, one optical system (microlens or the like) is arranged corresponding to each of the light receiving element unit and the display element unit, and an independent light receiving unit and a display unit are arranged for each optical system. By configuring it, light reception (imaging) and display can be performed at the same time. In this case as well, complicated conversion processing becomes unnecessary by making the design of the optical system in the vicinity at least the same.

These aspects will be described in detail below.

(1) First Aspect The first aspect of the device for capturing and displaying a stereoscopic image is a light receiving / displaying element composed of a plurality of light receiving pixels and a plurality of display pixels arranged on a plane, and the light receiving / displaying device. It has a plurality of optical systems that are regularly arranged side by side on the pixel arranging surface side of the dual-purpose element and convert light from a subject into light beams of a parallax image group having different parallaxs from each other, and each of the parallax images corresponds to the light receiving light. A stereoscopic image can be recorded by taking an image with pixels, and a stereoscopic image can be displayed by converting the light beam of the differential image group from the display pixel into a light beam of a stereoscopic image by each of the optical systems.

In the stereoscopic image imaging / display combined device 100 (light field camera / display) in the first aspect, as shown in FIG. 1 (a), a plurality of light receiving pixels 11 and a plurality of display pixels 12 are alternately arranged on a plane. It has an arranged light receiving / displaying element 1 and a plurality of optical systems 2 arranged on the pixel arranging surface 1s side (subject side and observing side) of the light receiving / displaying element 1. The light receiving / displaying element 1 constitutes a light receiving / displaying element unit 1a composed of a specific number of light receiving pixels and a specific number of display pixels at positions corresponding to each optical system 2a. As shown in FIG. 1 (b), one combination (unit) consisting of the light receiving / displaying element unit 1a and the corresponding optical system 2a is referred to as a light receiving / displaying unit 3. In each light receiving / displaying element unit 1a, a specific number of light receiving pixels 11 and a specific number of display pixels 12 are alternately arranged on a plane. The number of the light receiving pixels 11 and the display pixels 12 arranged in one display element unit 3 can be appropriately set.

FIG. 2 schematically shows the image pickup / display combined device 100 in the first aspect. The squares of the imaging / display combined device 100 represent one light receiving / displaying unit 3. The figure shows an example in which each light receiving / display unit 3 is arranged in a square shape, but the outer diameter of the light receiving / display unit may be hexagonal and may be filled in a honeycomb shape, or a plurality of other polygons may be filled on the same surface. It may be arranged. Each light receiving / displaying unit 3 includes an optical system 2a and a light receiving / displaying element unit 1a arranged below the optical system 2a.

One light receiving / displaying element unit 1a is composed of a red filter unit (R) 51, two green filter units (G) 52, and a blue filter unit (B) 53 arranged in a Bayer type as one unit. Units are arranged in a plurality of units. In each of the red filter unit (R) 51, the green filter unit (G) 52, and the blue filter unit (B) 53, two light receiving pixels 11 and two display pixels 12 are alternately arranged side by side. Is provided with a red filter, a green filter or a blue filter. In FIG. 2, only the green filter section (G) 52 is shown, but the same applies to the other filter sections. When the display pixel is a liquid crystal display pixel, a backlight can be provided on the opposite side of the pixel arrangement surface. A light-shielding wall 4 may be provided at the boundary between the light-receiving / display units 3 so that the light from the adjacent optical system does not cover the boundary. It is not necessary to provide these red, green, and blue filters for applications that do not require color imaging and display.

FIG. 3A is a schematic diagram showing a method of recording (imaging) a stereoscopic image using the stereoscopic image imaging / display combined device 100. Light from a subject, a plurality of optical systems 2 are converted into light beams of different parallax image group of parallax, receiving and display combined elements arranged corresponding to the optical system 2a ₁ to 2A region ₇ unit 1a ₁ to 1A ₇ Is incident on the image and is imaged by the imaging pixel. Since the images captured by the light receiving / displaying element units 1a _{1 to 1a 7} have different parallax, the data of these parallax image groups has the stereoscopic information of the subject.

FIG. 3B is a schematic diagram showing a method of displaying (observing) a stereoscopic image using the stereoscopic image imaging / display combined device 100. At the time of observation, the observer observes from the optical system 2 side through each optical system 2a _{1 to 2a 7} . At this time, when the data of the parallax image group obtained by the imaging is symmetrically converted and displayed by the light receiving / displaying element units 1a _{1 to 1a 7} , the light beam of the parallax image group is three-dimensional by _{each optical system 2a 1 to 2a 7.} It is converted into the light beam of the image and recognized as a stereoscopic image by the observer.

The stereoscopic image imaging / display combined device 100 can be, for example, a camera / display device that captures, reproduces, and displays a stereoscopic moving image. At this time, the stereoscopic image imaging / display combined device 100 has a function of converting and saving a series of moving images into disparity image group data by a stereoscopic image imaging function, and reproduces the image at the same frame rate as the frame rate at the time of imaging. It has a function of displaying a stereoscopic moving image. The stereoscopic image imaging / display device 100 can also function as, for example, a camera / display device that captures and displays a stereoscopic still image. In this case, the stereoscopic image imaging / display combined device 100 has a function of acquiring and storing parallax image group data for each still image, and also has a function of displaying the obtained parallax image group data.

Examples of the optical system include a lens array in which lenses are arranged, a diffractive optical element array in which diffractive optical elements are arranged, and a pinhole array in which pinholes are arranged. A lens array (microlens array) that is easy to design and has excellent optical performance is preferable.

(2) Second Aspect The device 200 for capturing and displaying a stereoscopic image in the second aspect has, as shown in FIG. 4, a plurality of light receiving / emitting combined pixels 13 arranged on a plane for both receiving and emitting light. It has an element 201 and a plurality of optical systems 2 that are regularly arranged side by side on the pixel arrangement surface 201s side of the light receiving / emitting light emitting element 201 and convert light from a subject into light beams of a parallax image group having different parallaxs from each other. A stereoscopic image can be recorded by capturing each of the differential images with the corresponding light-receiving / light-emitting pixel 201, and the light beam of the differential image group from the light-receiving / light-emitting pixel 201 can be captured by the respective optical systems 2a. It is characterized in that a stereoscopic image can be displayed by converting it into a light beam of a stereoscopic image.

The device 200 for capturing and displaying a stereoscopic image according to the second aspect receives light instead of the light receiving and displaying element 1 in which a plurality of light receiving pixels 11 and a plurality of display pixels 12 are alternately arranged on a plane. Similar to the first aspect 100 (see FIG. 1) of the device for capturing and displaying a stereoscopic image except that the light receiving / emitting element 201 formed by arranging the light receiving / emitting combined pixels 13 having both functions of light emitting is used. It is the composition of. Therefore, the differences from the stereoscopic image imaging / display combined device 100 in the first aspect will be mainly described.

The light-receiving / light-emitting pixel 13 is a pixel that can function as a light-receiving pixel or a light-emitting pixel by electrical control, and one pixel (pixel) functions as both light-receiving and light-emitting. have. For this reason. Since the luminous flux incident at the time of light reception (shooting) and the light flux emitted at the time of light emission (display) pass through the same optical system 2a, it is an ideal configuration for the device for capturing and displaying a stereoscopic image of the present invention. ..

As shown in FIG. 4A, the stereoscopic image imaging / display combined device 200 in the second aspect includes a plurality of optical systems 2 and a specific number of light receiving / emitting combined pixels 13 corresponding thereto. Here, a specific number of light-receiving / light-emitting pixels 13 is referred to as a light-receiving / light-emitting element unit 201a. FIG. 4B shows a light receiving / light emitting unit 203 composed of one optical system 2 and one light receiving / light emitting element unit 201a. The number of the light-receiving / light-emitting pixels 13 arranged in one light-receiving / light-emitting element unit 201a can be appropriately set.

FIG. 5 schematically shows the image pickup / display combined device 200 in the second aspect. The squares of the image pickup / display combined device 200 represent one light receiving / emitting unit 203. The figure shows an example in which each light receiving / light emitting unit 203 is arranged in a square shape. However, the outer diameter of the light receiving / light emitting unit may be hexagonal and may be filled in a honeycomb shape, or a plurality of other polygons may be filled on the same surface. It may be arranged. Each light-receiving / light-emitting unit 203 includes an optical system 2a and a light-receiving / light-emitting element unit 201a arranged below the optical system 2a.

One light-receiving / light-emitting element unit 201a is composed of a red filter unit (R) 51, two green filter units (G) 52, and a blue filter unit (B) 53 arranged in a Bayer shape as one unit. Units are arranged in a plurality of units. Each of the red filter unit (R) 51, the green filter unit (G) 52, and the blue filter unit (B) 53 is arranged with pixels 13 for both light reception and light emission, and a red filter, a green filter, or blue is placed on the pixels 13. A filter is provided. A light-shielding wall 4 may be provided at the boundary between the light-receiving / light-emitting units 203 so that the light from the adjacent optical system does not cover the boundary. It is not necessary to provide these red, green, and blue filters for applications that do not require color imaging and display.

The method of recording (imaging) and displaying (observing) a stereoscopic image using the stereoscopic image imaging / display combined device 200 in the second aspect is the same as the method in the first aspect. The same applies to the optical system that can be used.

(3) Third Aspect As shown in FIG. 6, the device 300 for capturing and displaying a stereoscopic image according to the third aspect has a specific number of light receiving pixels 11 arranged on a plane and the specific number of light receiving pixels. A plurality of light receiving units 301 composed of a photographing optical system 21 arranged on the pixel arrangement surface 301s side of 11, a specific number of display pixels 12 arranged between the light receiving units 301 and arranged on a plane, and the said It has a plurality of display units 302 composed of a display optical system 22 arranged on the pixel arrangement surface 301s side of a specific number of display pixels 12, and the plurality of light receiving units 301 disparate the light from the subject from each other. A stereoscopic image can be recorded by converting the light beam of a different parallax image group into a light beam and taking an image, and the stereoscopic image is displayed by converting the light beam of the disparity image group into a light beam of a stereoscopic image by the plurality of display units 302. It is characterized by.

The stereoscopic image imaging / display combined device 300 according to the third aspect has a configuration in which a plurality of light receiving units 301 and a plurality of display units 302 are evenly arranged on a plane alternately or with disorder. Each light receiving unit 301 has a light receiving element unit 311 having a specific number of light receiving pixels 11 and a photographing optical system 21, and each display unit 302 has a display element unit 312 having a specific number of display pixels 12 and a display optical system. Has 22 and. Here, a specific number of light receiving pixels 11 is referred to as a light receiving element unit 311 and a specific number of light receiving pixels 12 is referred to as a display element unit 312. It is preferable to provide a light-shielding wall 4 between each light receiving unit 301 and the display unit 302 so that the light from the display unit 302 does not cover the light receiving unit 301. It is preferable that the photographing optical system 21 and the display optical system 22 have the same optical performance, and it is preferable that at least the optical design of the nearby optical system is the same. The number of a specific number of light receiving pixels 11 constituting the light receiving element unit 311 can be appropriately designed, and similarly, the display element unit 312 can be appropriately designed with a specific number of display pixels 12.

FIG. 7 schematically shows the image pickup / display combined device 300 in the third aspect. One square of the imaging / display combined device 300 represents one light receiving unit 301 or display unit 302, and in the figure, the light receiving unit 301 and the display unit 302 are arranged alternately. The arrangement method of the light receiving unit 301 and the display unit 302 may be a square arrangement, a honeycomb-shaped filling arrangement, or a plurality of other polygons may be filled and arranged on the same surface as in the first aspect. You may.

One light receiving unit 301 has a red filter unit (R) 51, two green filter units (G) 52, and a blue filter unit (B) 53 arranged in a Bayer shape as one unit, and there are a plurality of these units. It is arranged and configured. Light receiving pixels 11 are arranged in each of the red filter unit (R) 51, the green filter unit (G) 52, and the blue filter unit (B) 53, respectively. One display unit 302 has a red filter unit (R) 51, two green filter units (G) 52, and a blue filter unit (B) 53 arranged in a Bayer shape as one unit, and there are a plurality of these units. It is arranged and configured. Display pixels 12 are arranged in each of the red filter unit (R) 51, the green filter unit (G) 52, and the blue filter unit (B) 53, respectively. It is not necessary to provide these red, green, and blue filters for applications that do not require color imaging and display.

In the device 300 for capturing and displaying a stereoscopic image in the third aspect, the light receiving (imaging) unit and the display unit are independent, but the method and display (observation) of recording (imaging) the stereoscopic image using this device. ) Is the same as the method in the first aspect. In the third aspect, since the light receiving (imaging) unit and the display unit are independent, it is possible to perform imaging and display at the same time. Further, the optical system that can be used can be the same.

[2] Head-mounting device Since the device for capturing and displaying a stereoscopic image of the present invention can shoot and display on the same surface, the pixel-arranged surface faces the subject at the time of shooting, and the pixel-arranged surface faces the observer at the time of display. By making it possible to change the orientation of the device for capturing and displaying a stereoscopic image so as to face the eyes, a head mount device capable of easily performing imaging and display (appreciation) can be obtained.

For example, as shown in FIG. 8, the stereoscopic image imaging / display device 401 can be flipped up and down by 180 ° on the upper edge of the glasses, that is, with the head mount device 400 in the form of flip-up sunglasses. can do. At the time of shooting, as shown in Fig. 8 (a), by flipping up the image pickup / display device 401 to the forehead so that the pixel arrangement surface 401s faces the subject, a stereoscopic image equivalent to the visual scene is obtained. Images can be recorded. At the time of viewing, as shown in FIG. 8B, the imaging / displaying device 401 can be rotated 180 ° and lowered in front of the eyes to observe the pixel arrangement surface 401s toward the eyes. With such a head mount device 400, it is very easy to record and reproduce the scene that one sees as it is with a realistic sense of distance.

As shown in FIG. 9A, the head mount device 500 having another configuration has a first half mirror 501 arranged in front of the field of view, and approximately 45 between the first half mirror 501 and the human eye. It includes a second half mirror 502 arranged at an angle of °, and a device 503 for capturing and displaying a stereoscopic image of the present invention arranged at a position where the reflected light from the second half mirror 502 enters. A backlight 504 may be provided on the back surface of the stereoscopic image imaging / display combined device 503, if necessary.

At the time of shooting, as shown in FIG. 9B, the light from the subject incident through the first half mirror 501 is passed through the second half mirror 502 and guided to the human eye, and at the same time, the second half mirror 502 It is reflected by and led to the device 503 for capturing and displaying a stereoscopic image. Therefore, an image equivalent to the image reflected by the second half mirror 502 and formed on the stereoscopic image imaging / display device 503 is observed on the virtual imaging surface 505.

At the time of display, as shown in FIG. 9C, the stereoscopic image displayed by the stereoscopic image imaging / display device 503 is reflected by the second half mirror 502 and the first half mirror 501, and further, the second half mirror 501 is used. Pass through the half mirror 502 and guide it to the human eye. Therefore, to the observer, the stereoscopic image imaging / display device 503 appears as a virtual display surface 506. The virtual display surface 506 is the distance along the optical path of the first half mirror 501 to the second half mirror 502 to the stereoscopic image imaging / display device 503 with respect to the virtual imaging surface 505. Since it is located far away, the position where the virtual image of the subject can be seen is also far away by the same distance. Normally, the distance between the virtual imaging surface 505 and the virtual display surface 506 is considerably shorter than the distance between the subject and the observer, so this is not a problem, but the position of the virtual display surface 506 is not a problem. There may be a problem that the viewing angle becomes narrower as the distance increases.

As an example of solving the above problem, for example, a head mount device 600 as shown in FIG. 10 can be mentioned. The head mount device 600 has the same configuration as the head mount device 500 shown in FIG. 9 except that the first half mirror 601 is a curved surface. The first half mirror 601 is, for example, a very thin meniscus lens having the same radius of curvature on both sides, and the center of curvature of the reflecting surface located on the observer side is, for example, a virtual imaging surface 605 (second half mirror 602). It is configured to be at the position of the surface on which an image equivalent to the image formed on the stereoscopic image imaging / display device 603 is observed. With such a configuration, the lens effect is not exerted on the transmitted light, and the position of the virtual imaging surface 605 and the position of the virtual display surface 606 match. That is, the shooting and the display are performed at the same position, and the problem that the viewing angle is narrowed is solved.

In this example, the first half mirror 601 is a thin meniscus lens with the same radius of curvature on both sides, but the curved surface is an aspherical surface for aberration correction, and the joint surface is configured as a reflective surface as a joint lens. The half mirror may also be a curved surface. In either case, it is possible to reproduce the same scene as seen with the naked eye by preventing the system consisting of the first half mirror and the second half mirror from having a lens effect on transmitted light. can. Also, for observers who usually wear glasses, the system consisting of the first half mirror and the second half mirror is configured to have the same lens effect as the glasses normally used for transmitted light. By doing so, it is possible to express the same scene as seen with the glasses on.

[3] Other application examples Since the device for capturing and displaying a stereoscopic image of the present invention can capture and display a light field using the same or equivalent optical system, the imaging surface (display surface) and screen size are large. Binocular parallax / motor parallax / convergence can be reproduced, and binocular parallax / convergence / focus sensation can be reproduced for small objects (glasses type, etc.), and a natural sense of distance can be expressed. By integrating shooting and display, shooting and display can be performed on the same surface. Therefore, if the imaging surface (display surface) or screen size is large, the shooting function and display function can be operated at the same time to make a videophone / videophone. It can be suitably used as a conference system. Since the shooting and display positions match, you can make natural eye contact, and even if multiple people are participating, you can make eye contact with any person. Also, since you can shoot and display on the same surface, you can realize an optical "camouflage" by covering the surroundings of things that you do not want to stand out with this device and displaying the light field received on the back surface as it is or by processing it. can.

A huge amount of light rays are required to reproduce the light field, and the amount of information for that is huge. However, only a small part of the light rays displayed by the light field display device enter the observer's eyes and are perceived. Since the device for capturing and displaying a stereoscopic image of the present invention has a photographing function on the same surface as the display surface, the light beam perceived by the observer at the time of display can be accurately recognized by recognizing the observer's eyes or face from the photographed data. Can be well identified. Utilizing this, in the process of display and data processing and data transmission for that purpose, the amount of calculation and data communication can be significantly reduced by handling only those related to the light rays that reach the vicinity of the observer's eyes or face. be able to.

100 ... The first aspect of the device for both imaging and display
1 ・ ・ ・ Light receiving / displaying element
1a ・ ・ ・ Element unit for both light receiving and displaying
1s ・ ・ ・ Pixel arrangement surface
11 ・ ・ ・ Light receiving pixel
12 ... Display pixel
13 ・ ・ ・ Pixel for both light reception and light emission
2 ... Multiple optical systems
2a ・ ・ ・ Optical system
3 ・ ・ ・ Light receiving / display unit
4 ・ ・ ・ Shading wall
51 ・ ・ ・ Red filter section
52 ・ ・ ・ Green filter section
53 ・ ・ ・ Blue filter section
200: The second aspect of the device for both imaging and display
201 ・ ・ ・ Element for both light receiving and emitting light
201a ・ ・ ・ Element unit for both light receiving and emitting light
201s ・ ・ ・ Pixel arrangement surface
203 ・ ・ ・ Light receiving / light emitting unit
300 ... Third aspect of the device for both imaging and display
21 ・ ・ ・ Shooting optical system
22 ・ ・ ・ Display optical system
301 ・ ・ ・ Light receiving unit
301s ・ ・ ・ Pixel arrangement surface
302 ・ ・ ・ Display unit
311 ・ ・ ・ Light receiving element unit
312 ・ ・ ・ Display element unit
400 ・ ・ ・ Head mount device
401 ・ ・ ・ Imaging / display combined device
401s ・ ・ ・ Pixel arrangement surface
500 ・ ・ ・ Head mount device
501 ・ ・ ・ First half mirror
502 ・ ・ ・ Second half mirror
503 ・ ・ ・ Imaging / display combined device
504 ・ ・ ・ Backlight
505 ・ ・ ・ Virtual imaging surface
506 ・ ・ ・ Virtual display surface
600 ・ ・ ・ Head mount device
601 ・ ・ ・ First half mirror
602 ・ ・ ・ Second half mirror
603 ・ ・ ・ Imaging / display combined device
604 ・ ・ ・ Backlight
605 ・ ・ ・ Virtual imaging surface
606 ・ ・ ・ Virtual display surface

Claims (18)

A light-receiving / display element composed of a plurality of light-receiving pixels and a plurality of display pixels arranged on a flat surface,
It has a plurality of optical systems that are regularly arranged side by side on the subject side of the light receiving / displaying element and convert light from the subject into light fluxes of parallax image groups having different parallax from each other.
A stereoscopic image can be recorded by capturing each parallax image with the corresponding light receiving pixel.
A device for capturing and displaying a stereoscopic image capable of displaying a stereoscopic image by converting the luminous flux of the parallax image group from the display pixels into a luminous flux of a stereoscopic image by each of the optical systems.
Using the position information of the observer's face or pupil acquired by the imaging function of the stereoscopic image imaging / display combined device, light rays reaching the vicinity of the pupil or face in data processing and data transmission related to display and display. A device for capturing and displaying stereoscopic images, which is characterized by handling only those related to. A light-receiving / light-emitting element composed of a plurality of light-receiving / light-emitting pixels arranged on a flat surface,
It has a plurality of optical systems that are regularly arranged side by side on the subject side of the light-receiving / light-emitting element and convert light from the subject into light fluxes of parallax image groups having different parallax.
A stereoscopic image can be recorded by capturing each parallax image with the corresponding light receiving / emitting pixel.
A device for capturing and displaying a stereoscopic image capable of displaying a stereoscopic image by converting the luminous flux of the parallax image group from the light receiving / emitting pixel into a luminous flux of a stereoscopic image by each of the optical systems.
Using the position information of the observer's face or pupil acquired by the imaging function of the stereoscopic image imaging / display combined device, light rays reaching the vicinity of the pupil or face in data processing and data transmission related to display and display. A device for capturing and displaying stereoscopic images, which is characterized by handling only those related to. In the device for capturing and displaying a stereoscopic image according to claim 1 or 2.
A device for capturing and displaying a stereoscopic image, wherein the plurality of optical systems are composed of a lens array. In the device for capturing and displaying a stereoscopic image according to claim 1 or 2.
A device for capturing and displaying a stereoscopic image, wherein the plurality of optical systems are composed of diffractive optical elements. In the device for capturing and displaying a stereoscopic image according to claim 1 or 2.
A device for capturing and displaying a stereoscopic image, wherein the plurality of optical systems are composed of a pinhole array. It has a plurality of light receiving units including a specific number of light receiving pixels arranged on a plane and a single photographing optical system arranged on the subject side of the specific number of light receiving pixels.
It has a plurality of display units including a specific number of display pixels arranged on a plane and a single display optical system arranged on the observation side of the specific number of display pixels, and the light receiving unit is the display unit of the display unit. Placed in between
A stereoscopic image can be recorded by converting the light from the subject into light fluxes of parallax image groups having different parallax and taking an image by the plurality of light receiving units.
A device for capturing and displaying a stereoscopic image, which displays a stereoscopic image by converting the luminous flux of the parallax image group into a luminous flux of a stereoscopic image by the plurality of display units. The three-dimensional image capturing / displaying apparatus according to claim 6, wherein the light receiving unit and the display unit are arranged alternately or irregularly and uniformly as a whole. A device for both image capture and display. In the device for capturing and displaying a stereoscopic image according to claim 6 or 7.
A device for capturing and displaying a stereoscopic image, wherein the photographing optical system and the display optical system are composed of a lens array. In the device for capturing and displaying a stereoscopic image according to claim 6 or 7.
A device for capturing and displaying a stereoscopic image, wherein the photographing optical system and the display optical system are composed of diffractive optical elements. In the device for capturing and displaying a stereoscopic image according to claim 6 or 7.
A device for capturing and displaying a stereoscopic image, wherein the photographing optical system and the display optical system are configured by a pinhole array. In the device for capturing and displaying a stereoscopic image according to any one of claims 6 to 10.
A device for capturing and displaying a stereoscopic image, wherein the photographing optical system and the display optical system have the same optical characteristics. In the device for capturing and displaying a stereoscopic image according to any one of claims 6 to 11.
Using the position information of the observer's face or pupil acquired by the imaging function of the stereoscopic image imaging / display combined device, light rays reaching the vicinity of the pupil or face in data processing and data transmission related to display and display. A device for capturing and displaying stereoscopic images, which is characterized by handling only those related to. In the device for capturing and displaying a stereoscopic image according to any one of claims 1 to 12.
A device for both imaging and displaying a stereoscopic image, which is characterized in that imaging and display can be performed at the same time. A head mount device using the device for capturing and displaying a stereoscopic image according to claims 1 to 13.
Surface on which the pixels are arranged faces the object direction at the time of shooting, as the time display can change the orientation of the imaging and display sharing device of the stereoscopic image so as to face the eye surface on which the pixels are arranged by the observer A head mount device characterized by the fact that it has been done. A head mount device using the device for capturing and displaying a stereoscopic image according to claims 1 to 13.
From the first half mirror arranged in front of the field of view, the second half mirror arranged at an angle of approximately 45 ° between the first half mirror and the human eye, and the second half mirror. It has a device for capturing and displaying the stereoscopic image, which is arranged at a position where the reflected light of the above-mentioned light enters.
A device for capturing and displaying a stereoscopic image by passing light from a subject incident through the first half mirror through the second half mirror and guiding it to the human eye, and reflecting it by the second half mirror. The stereoscopic image displayed by the device for capturing and displaying the stereoscopic image is reflected by the second half mirror and the first half mirror, and further passed through the second half mirror to pass a person. A head-mounting device that is designed to lead to the eyes of a person. In the head mount device according to claim 15,
A head mount device, wherein the first half mirror or the second half mirror is a curved half mirror. In the head mount device according to claim 15 or 16.
A head mount device characterized in that the system including the first half mirror and the second half mirror does not have a lens effect on transmitted light. In the head mount device according to claim 15 or 16.
A head mount device, wherein the system including the first half mirror and the second half mirror has a visual acuity correction effect on the transmitted light.

Images