WO2003098942A1

WO2003098942A1 - Information processing apparatus, information processing system, and dialogist displaying method

Info

Publication number: WO2003098942A1
Application number: PCT/JP2003/006155
Authority: WO
Inventors: Yoshihiko Kuroki
Original assignee: Sony Corporation
Priority date: 2002-05-21
Filing date: 2003-05-16
Publication date: 2003-11-27
Also published as: JP2004048644A; US7528879B2; KR20040106575A; CN1672431A; KR100976002B1; US20050175257A1; TW200307460A; TWI223561B; EP1507419A1; EP1507419A4

Abstract

An information processing apparatus has an image display part for displaying a desired image in accordance with an image signal; and imaging parts provided on right and left sides of the image display part. An image obtained from the right side and an image obtained from the left side are combined and displayed, as a desired image, on an image display part of the other end’s terminal or the like. This realizes displays in which speakers can have eye contacts with each other. In this way, there can be provided information processing apparatus, such as a mobile communication terminal, that allows dialogues with speaker’s natural eye contacts realized.

Description

Description Information processing apparatus, information processing system, and interlocutor display method

The present invention relates to an information processing apparatus such as a mobile communication terminal capable of displaying an image of a communication partner. The present invention relates to an information processing apparatus, an information processing system, and a method for displaying an interlocutor using a mobile communication terminal or the like for displaying the other party's line of sight according to the terminal user. Background art

In recent years, mobile communication terminals such as mobile phones have become extremely popular. In addition to voice communication functions, these mobile communication terminals are capable of sending and receiving e-mail, accessing the Internet, Various functions, such as receiving still images and moving images captured by the other party's camera, have become available. In recent years, not only mobile phones, but also two-way video communication systems such as video conference systems and video phones that connect remote locations with a video-voice communication network have become widespread.

Some mobile phones and two-way video communication systems are configured so that an image of a communication partner is displayed on an image display device arranged near a camera that images a user's face and the like. A user of such a mobile phone or a two-way video communication system usually proceeds with a conversation while watching an image such as a face of a call partner displayed on an image display device.

By the way, when an imaging device such as a camera is mounted on a relatively small device such as a mobile phone, for example, the size of the image display unit is 50 mm. Even if it is as small as four sides, if the imaging device is arranged in the left and right areas or the upper and lower areas of the image display unit, the line of sight of the captured face image indicates the other party As long as you look at the image display, you will not face the imaging device. As a result, in such a device, a face image that does not match the line of sight is displayed on the image display unit. Therefore, in such a device, since the user talks with the other party with an image that does not match the line of sight, there is a problem that an unnatural feeling cannot be denied and the sense of reality is lost as the impression of the dialogue.

Specifically, such a defect can be described with reference to FIGS. 27A, 27B, 28A, and 28B. FIG. 27A, FIG. 27B, FIG. 28A, and FIG. 28B are diagrams showing examples in which the viewing directions do not match on the image display unit of the terminal. In these examples, the camera is located 65 mm from the center of the image display, the user's face is positioned at a distance of about 25 cm, and the user looks at the center of the image display. The image is shown.

Figure 27A and Figure 27B show the appearance of the image displayed on the image display unit when the camera is placed in one of the left and right areas of the image display unit, which is about 50 mm square. It is a figure which shows typically. FIG. 27A is a camera image when the camera is arranged on the left side toward the image display unit, and FIG. 27B is a camera image when the camera is arranged on the right side toward the image display unit. In this way, all camera images have an unnatural dialogue screen because their eyes are not directed toward the user from the image display unit.

Similarly, FIGS. 28A and 28B are displayed on the image display section when the camera is arranged in one of the upper area and the lower area of the image display section of about 50 mm square and picked up. It is a figure which shows the mode of an image typically. Figure 28A shows a camera image when the camera is placed above the image display section. B is a camera image when the camera is arranged below the display screen. In this case as well, the camera image does not look directly at the user from the image display unit, resulting in an unnatural dialog screen.

In order to eliminate such unnaturalness, various gaze matching technologies have been devised.

As such a technology, a micromirror of a substantially flat plate shape is arranged on the surface of the image display unit, and an image displayed on the image display unit is displayed through the micromirror and displayed by the user. In some cases, an image is captured by receiving reflected light reflected on the surface of a minute half mirror with an imaging camera. In such a technique, since the line of sight of the user who looks at the image display unit and the optical axis of the incident light of the minute half mirror before reflection are matched, it is possible to perform a display that matches the line of sight. .

Further, as another technique, for example, there is a technique in which an image display unit having a structure capable of transmitting light is provided, and a camera is arranged behind the image display unit for a user. In this technique, the image display unit repeats the display state and the transmission state in a time-division manner, and when the image display unit is in the transmission state, an image of the user is taken. In this technique, a required video signal is sent to the image display unit during a period different from the transmission state, and an image of the other party is displayed. Even in such a technique, since the light emission direction of the image display unit and the optical axis of the incident light transmitted to the camera arranged on the back side of the image display unit match, the display is performed with the eyes aligned. It becomes possible.

Further, as another technique, a display / imaging device as described in Japanese Patent Application Laid-Open No. 4-166790 is known. In this display / imaging device, minute holes are provided over the entire surface of the image display unit, the end of the optical fiber faces each of the minute holes, and the other end of the optical fiber is connected to the camera. It is configured to connect. In this display and imaging device, Since the relationship between the end of the optical fiber 1 facing the hole and the image display unit does not deviate in position, it is possible to perform display in line of sight. The conventional techniques described above are realized by aligning the optical axis of the image display unit with the optical axis of the imaging device. Further, as the technique of adjusting the line of sight, for example, "Yamaguchioka IJ, et al. Like the technology described in "Proposal of Eye-gaze Adjustment Technique in Video Conference", Proceedings of the 6th Image Sensing Symposium, p.267-p.272, 2000 Also known are those that make full use of tag raftics

Furthermore, recently, an attempt has been made to realize eye-gaze matching by stereoscopic vision (for example, see Japanese Patent Application Laid-Open No. H10-75432). Japanese Patent Application Laid-Open No. H10-747532 discloses a housing placed on a table provided with an imaging unit composed of a camera and an image display unit. A three-dimensional video phone is disclosed, which is configured using a three-dimensional liquid crystal display element without glasses of the split-type system, and in which force lenses are arranged at right and left positions of the image display unit. Japanese Patent Application Laid-Open No. Hei 10-747532 discloses that by selectively combining and fusing images captured by two cameras provided at left and right positions of an image display unit, It states that a pseudo-stereoscopic frontal face image can be obtained, and that conversation with eyes can be performed.

By the way, in the above-described various technologies realized by aligning the image display unit with the optical axis of the imaging device, the imaging device is arranged with some positional relationship with respect to the image display unit itself.

However, in the above-described technology using the minute half mirror, it is necessary to dispose the imaging device in the direction of reflection of the light reflected on the surface of the minute half mirror. In this technology, it is necessary to arrange a camera behind the image display unit. Therefore, in order to realize these technologies, it is necessary to hold a camera or the like in a large size as a whole device, and it was impossible to realize such technologies with, for example, a mobile phone.

In an apparatus for attaching an optical fiber to a micro hole on an image display unit as described in the above-mentioned Japanese Patent Application Laid-Open No. 4-166690, a micro hole is provided in the image display unit, It was not easy to assemble the ends of the fibers together, and there was a problem that the price of the product would rise significantly.

Furthermore, in the technology described in "Proposal of Eye Adjustment Technique in Video Conference" described above, in which the eyes of the other party are synthesized by making full use of computer graphics, the above-mentioned question depends on the mounting position of the imaging device. Although there is no problem, the current state of the art of computer graphics is still far from the actual video, and has not yet wiped out the unnaturalness of the line of sight of the other party.

Furthermore, in the stereoscopic videophone described in the above-mentioned Japanese Patent Application Laid-Open No. 10-75432, when the size of the image display section is, for example, about 14 inches, Since the distance in the left-right direction is about 30 cm, when the two cameras are provided at positions that do not interfere with the image display unit, the displacement between the two captured images becomes large. Therefore, in this television phone, if stereoscopic display is performed using these two images as they are, the parallax will be too large, and it will not be possible to fuse as a double image. Even if it could be done, the display would put a burden on the eyes of the user, and there was a problem that it would cause fatigue.

The present invention has been made in view of such circumstances, and in a portable information processing apparatus such as a mobile communication terminal, an information processing apparatus that realizes a natural eye-gaze conversation with a partner is provided. System and interlocutor display The aim is to provide a method. The present invention also provides an information processing apparatus, an information processing system, and an interlocutor display for realizing a conversation in which a natural gaze is matched with a partner while avoiding a problem caused by a parallax between two captured images being too large. The aim is to provide a method. Disclosure of the invention

An information processing apparatus according to the present invention that achieves the above object is a portable information processing apparatus that performs a dialogue with a video, and an image display unit that displays a required image according to an image signal; An image display means is provided on each of the left and right sides of the image display means.

In such an information processing apparatus according to the present invention, by providing the imaging means on each of the left and right sides of the image display means, the right imaging means captures the image viewed from the right front of the user as an image. The image pickup means captures a state viewed from the front left of the user as an image. The image taken from the left side and the image taken from the right side are displayed together as required image display on an image display means such as a terminal on the other side. Then, the viewer of the image display means sees both images at the same time, and the discrepancy between the left and right gaze directions is grasped in a corrected form. Therefore, a display can be realized in which the eyes are aligned without particularly aligning the optical axis of the imaging means with the optical axis of the image display unit.

Further, an information processing apparatus according to the present invention that achieves the above object is a portable information processing apparatus that performs a dialogue with a video, comprising: a portable housing; an image display unit that displays a desired image in response to the signal, the _t casing a surface of which is characterized by comprising an imaging means provided on each of the right and left sides of the image display unit such In the information processing apparatus according to the present invention, It has a structure in which the device is mounted in a portable housing, and the imaging means mounted on each of the left and right sides is formed on the surface of the housing, so it must have a thin structure as a whole. The information processing device can be configured with a small and lightweight housing.

Further, an information processing apparatus according to the present invention that achieves the above object is a portable information processing apparatus that performs a dialogue with a video, and includes a plurality of pixels that perform display based on a left-eye signal and a right-eye image. It is characterized by comprising image display means in which a plurality of pixels for performing display based on signals are mixed, and image pickup means provided on each of the left and right sides of the image display means.

According to such an information processing apparatus according to the present invention, by providing the image pickup means on each of the left and right sides of the image display means, it is possible not to adjust the position of the image display means in particular, for example, to avoid a positional relationship of overlapping. Since the imaging means is mounted, a small and thin structure as a whole is realized. Since the image display means includes a plurality of pixels for performing display based on the signal for the left eye and a plurality of pixels for performing display based on the signal for the right eye, no special device such as polarized glasses is required. It is possible to proceed with the conversation while keeping the gaze with the other party.

In particular, the information processing apparatuses of the three forms according to the present invention as described above each generate a new image in which parallax is interpolated, based on the two images captured by the imaging unit. It is preferable that an image processing unit is provided, and two new images generated by the image processing unit are displayed on the display surface of the image display unit. '

With this, in the information processing apparatus according to the present invention, it is possible to avoid a problem in displaying images due to an excessively large parallax between two images captured by the imaging unit. You can optimize the stereoscopic display to match the line of sight with the other party, It can be a natural image.

Still further, the information processing system according to the present invention for achieving the above-mentioned object has an image display means capable of displaying an image including a face portion of a call partner, and has image pickup means on both left and right sides of the image display means. A plurality of portable information processing terminals that are configured to include means for performing a dialogue with a video are provided, and the information processing terminals can communicate with each other. In such an information processing system according to the present invention, the imaging means is provided on each of the right and left sides, so that the imaging means on the side has a view from the right front of the user, and the imaging means on the left has the user The images viewed from the left front of are captured as images. The image taken from the left side and the image taken from the right side are displayed together by the image display means of the information processing terminal on the other side. Then, the viewer of the image display means sees both images at the same time, and an image in which the line of sight matches can be obtained. In particular, in the information processing system according to the present invention, each of the information processing terminals is configured to generate a new image in which parallax is interpolated based on two images obtained by the imaging unit. It is preferable that the information processing terminal has a processing means, and two new images generated by the image processing means are displayed on a display surface of the image display means of the information processing terminal of the other party.

With this, in the information processing system according to the present invention, it is possible to avoid a problem in displaying images due to an excessively large parallax between two images captured by the imaging unit, and therefore, the communication partner It is possible to optimize the stereoscopic display for matching the eyes with the eyes, and to make the images easy to see and natural.

In addition, the method for displaying an interlocutor according to the present invention that achieves the above-described object includes a pair of imaging devices provided on both left and right sides of an image display unit of a portable terminal. A video capturing step of capturing the video of the user by the means, and a display step of causing the image display means of the terminal of the other party to display the user's line of sight from the viewpoint of the other party.

In such a method for displaying an interlocutor according to the present invention, a video of a user is captured by a pair of imaging means, and a signal relating to a left image and a signal relating to a right image of the user are obtained. Become. These signals are sent to the image display means of the terminal of the communication partner, and the pixels of the left image and the pixels of the right image are mixed and displayed, for example, so that the user's line of sight coincides with the viewpoint of the communication partner. Can be displayed.

In particular, the interlocutor display method according to the present invention includes an image processing step of generating a new image in which parallax is interpolated based on the two images captured in the video capturing step. It is desirable that two new images generated in the image processing step be displayed on the display surface of the image display means.

Accordingly, in the interlocutor display method according to the present invention, it is possible to avoid a problem in display due to an excessively large parallax between the two images captured by the imaging unit. It is possible to optimize the stereoscopic display to match the line of sight with the other party, and to make it easier to view and natural images.

Further, an information processing apparatus according to the present invention that achieves the above-described object is an information processing apparatus that performs a dialogue with a video, and includes an image display unit that displays a required image in accordance with an image signal; Imaging means provided on each of the right and left sides of the means, and image processing means for generating a new image with parallax interpolated based on the two images obtained by the imaging means. The display screen of the image display means displays two new images generated by the image processing means _c. In such an information processing apparatus according to the present invention, imaging means are provided on both left and right sides of the image display means, and a new parallax-interpolated image is obtained based on the two images obtained by the imaging means. By generating an image and displaying these two images on the image display means, it is possible to realize a display that matches the line of sight without aligning the optical axis of the imaging means and the optical axis of the image display unit. Optimizing the stereoscopic display to match the line of sight with the other party, because it is possible to avoid the problem of display due to too large a parallax between the two images obtained by the imaging means. And a natural image that is easier to see.

Furthermore, an information processing apparatus according to the present invention that achieves the above-mentioned object is an information processing apparatus that performs a dialogue with a video, wherein the information processing apparatus is mounted on a surface of the housing, and required according to an image signal. Image display means for displaying the image of the image display means, imaging means provided on the left and right sides of the image display means on the surface of the housing, and two images obtained by the imaging means. Image processing means for generating a new image in which parallax is interpolated, and two new images generated by the image processing means are displayed on the display surface of the image display means. It is characterized by that.

Such an information processing apparatus according to the present invention has a structure in which the above information processing apparatus is mounted on a portable housing, and the imaging means mounted on each of the left and right sides is a surface of the housing. Since it is formed above, it is possible to take a thin structure as a whole, and the information processing device can be configured with a small and lightweight housing.

An information processing apparatus according to the present invention that achieves the above object is an information processing apparatus that performs a dialogue with a video, and includes a plurality of pixels that perform display based on a left-eye signal and a display based on a right-eye signal. Image display means in which a plurality of pixels for performing Image capturing means provided, and image processing means for generating a new image with parallax interpolated based on the two images captured by the image capturing means, and provided on a display surface of the image display means. Is characterized in that two new images generated by the image processing means are displayed.

According to such an information processing apparatus according to the present invention, the imaging means is mounted without adjusting the position of the image display means, for example, without a positional relationship such that they are superimposed on each other. This structure is realized. Further, the image display means includes a plurality of pixels performing display based on the signal for the left eye and a plurality of pixels performing display based on the signal for the right eye, and is obtained by being imaged by the imaging means. The newly generated image is displayed by interpolating the parallax based on the two images, so the stereoscopic display to match the line of sight with the other party is optimized without the need for special equipment such as polarized glasses. This makes it possible to proceed with the conversation while keeping the gaze with the other party under a more natural image that is easier to see.

Further, the information processing system according to the present invention that achieves the above-mentioned object includes, in addition to providing image display means capable of displaying an image including a face part of a call partner, image pickup means on each of the left and right sides of the image display means. A plurality of information processing terminals for performing a dialogue with video, each of the information processing terminals having a new parallax-interpolated based on two images captured by the imaging means. Image processing means for generating a unique image, and when communicating between the information processing terminals, the two new images generated by the image processing means are transmitted to the information processing terminal of the other party. The image is displayed on the display surface of the image display means.

In such an information processing system according to the present invention, the image pickup means is provided on each of the left and right sides, so that the right image pickup means looks from the right front of the user, and the left image pickup means shows the user's view. The state seen from the front left is Each is captured as an image. The image taken from the left side and the image taken from the right side are displayed together by the image display means of the information processing terminal on the other side. At this time, the image display means displays a newly generated image by interpolating the parallax based on the two images obtained by the image pickup means, so that it is possible to match the gaze with the other party. The 3D display can be optimized, and an image that is extremely easy to see and has a natural gaze direction can be obtained.

Still further, the method for displaying an interlocutor according to the present invention, which achieves the above-mentioned object, comprises: An image processing step of generating a new image with parallax interpolated based on the two images captured in the video capturing step, and an image display means of the other party's terminal displaying the image in the image processing step. And displaying the two new images so that the user's line of sight coincides with the viewpoint of the other party.

In such an interlocutor display method according to the present invention, a user's video is captured by a pair of imaging means, and a new image in which parallax is interpolated based on the captured two images is generated. The signal concerning the left image and the signal concerning the right image of the user are obtained. By displaying these signals on the image display means of the other party's terminal while mixing the pixels for the left image and the pixels for the side image, for example, the three-dimensional display to match the line of sight with the other party is optimized. This makes it possible to match the line of sight of the user with the viewpoint of the other party and display the same. BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a schematic diagram of an example of the information processing device of the present invention. 2A to 2C are schematic diagrams illustrating the principle of an example of the information processing device of the present invention.

FIG. 3 is a block diagram showing a schematic circuit configuration of an example of the information processing apparatus of the present invention.

FIG. 4 is a diagram showing a use state of a system using the information processing device of the present invention.

FIG. 5 is an explanatory diagram (X direction) of an emission angle of a light beam in the image display unit of the information processing device of the present invention.

FIG. 6 is an explanatory diagram (y-direction) of an emission angle of a light ray in the image display unit of the information processing device of the present invention.

FIG. 7 is a diagram illustrating a mounting position of a camera on an image display unit of an example of the information processing apparatus of the present invention.

FIG. 8 is a schematic diagram illustrating a configuration of a pixel of an image display unit of an example of the information processing device of the present invention.

9A to 9E are diagrams illustrating an example of an array pattern of left-eye pixels (L) and right-eye pixels (R) of an image display unit of an example of the information processing apparatus according to the present invention. FIG. 10 is a schematic diagram showing a configuration near a pixel of an image display unit of an example of the information processing apparatus of the present invention. .

FIG. 11 is a cross-sectional view of an image display unit of an example of the information processing apparatus according to the present invention, and is a cross-sectional view in a case where the axes of the light emitting element and the microphone aperture lens are not shifted.

FIG. 12 is a cross-sectional view of an image display unit of an example of the information processing apparatus of the present invention, and is a cross-sectional view in a case where the axes of the light emitting element and the microphone aperture lens are misaligned.

FIGS. 13A and 13B are diagrams showing the spread of light rays by the light emitting element and the microlens as an example of the information processing apparatus of the present invention. FIG. 13A shows the axis of the light emitting element and the microlens. Figure 13B shows the spread of the light beam when the axis of the light emitting element and the microphone aperture lens are misaligned. It is the figure which simulated each of these.

FIGS. 14A to 14C are graphs showing optical calculation results on the size and the positional relationship between the microlens and the light-emitting element as an example of the information processing apparatus according to the present invention. FIG. Fig. 14B is a graph showing the result of optical calculation of the element diameter and the spread angle of the light beam. Fig. 14B is a graph showing the result of calculating the position of the lens and the optical axis direction and the spread of the light beam. FIGS. 15A and 15B are graphs showing calculation results of an in-plane direction position of a light emitting element and an emission angle of a light ray. FIGS. 15A and 15B show microlenses used as an example of the information processing apparatus of the present invention. FIG. 15A is a diagram showing an example, FIG. 15A is a front view, and FIG. 15B is a cross-sectional view.

FIG. 16 is a conceptual diagram for explaining the concept of image processing as a function of an example of the information processing apparatus of the present invention, and is a diagram for explaining a state of image processing for finding an image closest to a template from an image. is there.

FIG. 17 is a diagram showing parameters for parallax.

FIGS. 18A and 18B are diagrams showing an image display section of another example of the information processing apparatus of the present invention, and FIG. 18A is a cross-sectional view showing the vicinity of a light emitting element and a cone type micro lens. FIG. 18B is a perspective view of the cone-type microlens.

FIG. 19 is a schematic view illustrating parameters for explaining a relationship between light emission and an emission angle when a microlens or the like is used.

FIG. 20 is a schematic diagram illustrating the relationship between the partial area ratio to the global area and the exit angle Φ when a micro lens or the like is used.

FIG. 21 is a cross-sectional view showing an image display unit of still another example of the information processing device of the present invention.

FIGS. 22A and 22B are schematic diagrams showing an experiment in which the impression of an image whose gaze direction and the gaze direction were matched was examined. FIG. 22B is a diagram when a monaural image is obtained.

FIGS. 23A and 23B are graphs showing the results of the experiments of FIGS. 22A and 22B, and FIG. 23A is a diagram when a stereoscopic image is obtained. B is a diagram when a monaural image is obtained.

FIG. 24 is a conceptual diagram illustrating the concept of interpolating parallax in the information processing apparatus of the present invention, and is based on two images captured by two cameras provided on the left and right sides of the image display unit. FIG. 8 is a diagram for explaining how to generate a new image with small parallax as if it were obtained by being virtually imaged by two virtual cameras provided at a smaller interval than the two cameras.

FIG. 25 is a conceptual diagram illustrating the concept of image processing as a function of an example of the information processing apparatus according to the present invention, in which two images captured by two cameras provided on the left and right sides of the image display unit are shown. FIG. 7 is a diagram for explaining a state of image processing for obtaining corresponding points of the image processing.

Figure 26 is a diagram schematically showing an example of an image generated by moving the image by half the number of pixels of the amount of displacement indicating parallax, taken from the center of the two cameras that took the two images FIG. 8 is a diagram for explaining how an image equivalent to the generated image is generated.

FIGS. 27A and 27B are diagrams schematically showing a display screen when a camera is arranged on one of the left and right sides of a screen having a predetermined angle, and FIG. 27A is a case on the left side. FIG. 27B is a diagram of the case on the right side.

FIGS. 28A and 28B are diagrams schematically showing a display screen when a camera is arranged on one of the upper and lower sides of a screen having a predetermined angle, and FIG. 28A shows an upper side of the screen. FIG. 28B is a diagram of the lower case. BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings.

This embodiment relates to a mobile communication terminal capable of displaying a face image of a call partner. As shown in FIG. 1, the mobile communication terminal 10 as an information processing apparatus has a housing 12 having a size that can be held by a user with one hand, and the longitudinal direction of the housing 12 is a vertical direction. It is configured to enable dialogue with An image display section 11 having a size of about 10 mm square to about 100 mm square is provided on the front side of the housing 12 so that its display surface is exposed to the outside. On the left and right sides of the image display section 11, a left camera 13L as an imaging section and a right camera 13R are provided. On the lower side of the image display section 11, a minor monitor 14 for monitoring the image of the face of the user is provided, and on the side of the minor monitor 14, the face of the user is displayed. An indicator 17 is provided for notifying by flashing or the like when a large departure from the imaging region has occurred. In addition, a button 15 and a key input unit 16 are formed below the minor monitor 14. In the portable communication terminal 10, the operation of the button 15 and the key input section 16 enables the dialing of a telephone number, the input of characters of an e-mail, the function control of the portable communication terminal 10, and the like. Become.

The left camera 13L and the right camera 13R as imaging units are disposed on the left and right sides of the image display unit 11, respectively. The left camera 13L is provided so as to take an image from a slightly left front of the user using the mobile communication terminal 10, and the right camera 13R is provided to take an image from a slightly right front of the user. It is provided as follows. The left camera 13 L and the right camera 13 R are each composed of, for example, an optical system in which an M〇S (Metal-Oxide Semiconductor) image sensor or a CCD (Charge Coupled Devices) image sensor and a lens are combined, Specifically, in the mobile communication terminal 10, the left camera 13 L and the right camera For example, a small and lightweight solid-state imaging device is used as the 13R.

The image display unit 11 is configured by mixing a plurality of pixels performing display based on the left-eye signal and a plurality of pixels performing display based on the right-eye signal. In particular, the image display section 11 has emission means for independently emitting light to both eyes of the user. Here, a light emitting unit that generates required light based on a left-eye signal or a right-eye signal as an emission unit, and an emission angle that controls light from this light-emitting unit to emit in a predetermined angle direction And a control unit. The light emitting section is composed of a plurality of light emitting diodes arranged, and the emission angle control section is composed of a micro lens array in which a plurality of micro lenses are arranged in a matrix. Note that the structure of the emitting means here is merely an example.

It is also possible to use a structure using an EL (Electronic Luminescent) device, a plasma display device, or the like, and may have a structure such as a transmission type or a reflection type. Further, the injection angle control unit can also have various structures as described later. A method of mixing a plurality of pixels performing display based on the left-eye signal and a plurality of pixels performing display based on the right-eye signal will be described later.

First, a technique for sending the signals of both eyes to one display unit to adjust the gaze will be described. In general, there have been studies in the field of videophones and the like on the eyes detected and the eyes allowed in which the eyes are inconsistent. For example, see "Sato et al., No. 1998," And "South," 4.2 Video Telephone ", Journal of the Institute of Electronics and Communication Engineers, 1 1 / '73 Vol. 56, No. 11 and pi 485-149 0 "are known. These documents show considerations and experimental results on gaze coincidence in videophones, and state that "the detection criteria that can detect the discomfort of the gaze as human sensibility are about 2 to 3 degrees from the center. It is an extremely narrow range, and there is little difference between the horizontal and vertical directions. " And these documents state that the permissible eye as the limit of the permissible range for a videophone is about 4.5 ° in the horizontal direction, 12 ° in the vertical direction, and 8 ° in the vertical direction. °, and the range allowed in the horizontal direction is particularly narrow, and people are sensitive to the left and right eyes. ”

The mobile communication terminal 10 shown as an embodiment of the present invention is a device that can compensate for this relatively easily perceived shift in the line of sight between left and right to realize realistic communication. 2A to 2C are illustrations that briefly illustrate the concept. FIG. 2A is an image from the left camera 13L. Since the left camera 13L is located on the left side of the center, the user's line of sight appears from the right. FIG. 2B is an image from the right camera 13. Since the right camera 13 R is located on the right side of the center, the user's line of sight appears from the left. Then, FIG. 2C shows a case where a plurality of pixels performing display based on the signal for the left eye and a plurality of pixels performing display based on the signal for the right eye are mixed and independently for both eyes of the user. 3 shows a display screen when an image is displayed using the image display unit 11 having a mechanism for emitting light. That is, the mobile communication terminal 10 emits light toward the left eye of the user from each pixel of the image display unit 11 to which the signal for the left eye is sent exclusively, and emits the emitted light. Reaches the left eye of the user. Similarly, the mobile communication terminal 10 emits light toward the right eye of the user from each pixel of the image display unit 11 to which the signal for the right eye is sent exclusively, and the emitted light is used. Reaches his right eye. As a result, the user sees a different image between the two eyes, but the user's cerebrum has the function of synthesizing and seeing it, and as a result, as shown in Fig. This makes it possible to see the screen as if it were displayed, and a natural telephone call with the gaze of the eyes was realized. In addition, The image may be a still image or a moving image. Also, the mobile communication terminal 10 can transmit and receive real-time video if there is enough transmission and reception bandwidth, but it is configured to transmit and receive compressed or thinned-out image data. Is also good.

FIG. 3 is a block diagram showing a circuit configuration example of the mobile communication terminal 10. The control circuit 21 includes a CPU (Central Processing Unit), a video memory, and a required image information processing circuit, and the control circuit 21 also performs signal processing for preventing camera shake as described later. The data processed by the control circuit 21 can be transmitted by the transmission / reception unit 28, and the data received by the transmission / reception unit 28 is transmitted to the control circuit 21. In the key input section 27, for example, a numeric keypad and a function setting button are arranged, and further, a jog dial, an input pad, other extended function devices, and the like may be arranged. A signal from the left camera 24L and a signal from the right camera 24R are independently supplied to the control circuit 21. In normal use, the signal from the left camera 24 L includes video data captured from the user's slightly forward left, and the signal from the right camera 24 R captures an image from the user's slightly forward right. Includes video and overnight. In the mobile communication terminal 10, signals from the left camera 24 L and the right camera 24 R are transmitted to the transmission / reception unit 28 via the control circuit 21, and transmitted from the transmission / reception unit 28. Received by other mobile communication terminals. In the mobile communication terminal 10, it is possible to perform a call by transmitting and receiving such data overnight. In particular, in the present embodiment, as shown in FIG. Being able to see it means that a natural call can be made with the eyes aligned.

The signals from the left camera 24L and the right camera 24R are sent to other portable communication terminals, respectively, and can be projected on a small monitor screen via the control circuit 21. That is, the left camera 24 L and the right camera The signals from 24 R are sent to the minor monitor 25 via the control circuit 21 and the driver 26, respectively. This allows the portable communication terminal 10 to see its own face while talking on a relatively small screen such as the minor monitor 14 shown in FIG. If you want to talk on a relatively small screen such as a main monitor 14 without checking your face, depending on the angle of the mobile communication terminal 10, the left camera 24 L and the right camera 2 At the center of the 4R, it may not be possible to capture your own face, and the face may be out of the camera's imaging range. Therefore, in the mobile communication terminal 10, by making a call while checking the image from the minor monitor 25, it is possible to proceed with a conversation in which the communication partner and the natural gaze coincide with each other. The minor monitor 25 is a device that prompts visual confirmation, but in the mobile communication terminal 10, signals from the left camera 24 L and the right camera 24 R are sent to the control circuit 21, respectively. As a result, steady image processing such as camera shake prevention processing may be performed, and if an appropriate face cannot be displayed, the indicator 29 may be made to blink, and the minor monitor Control may be performed so as to display this on the screen of the main monitor 22 or the main monitor 22.

The signal from the other party is sent from the transmission / reception unit 28 to the control circuit 21, divided into a left-eye signal LE and a right-eye signal RE, and output from the control circuit 21. The left-eye signal LE is sent to the left-eye driver 23 L for driving the left-eye pixel, and the right-eye signal RE is sent to the right-eye driver 23L for driving the right-eye pixel. Sent to driver 23 R. The main monitor 22 that displays the other person's face is driven by signals from the left-eye driver 23 L and the right-eye driver 23 R. As described above, on the main monitor 22, pixels for the left eye and pixels for the right eye are mixed, for example, even lines are for the right eye and odd lines are for the left eye. Independent pixels On the side of the main monitor 22, the function of synthesizing and viewing it on the cerebrum of the user works, and as a result, as shown in FIG. The screen can be viewed as if it were being displayed, and a natural call that matched the eyes was realized.

Note that the block diagram shown in FIG. 3 merely shows an example of the mobile communication terminal 10, and other circuit configurations are possible. For example, in the mobile communication terminal 10, only one of the signal LE for the left eye and the signal RE for the right eye may be sent to both drivers 23 L and 23 R by a button operation or the like. Also, it may be controlled so that the signal of the main monitor 122 and the signal of the minor monitor 25 are switched in accordance with a button operation, etc. It is also possible to display as follows. In the mobile communication terminal 10, not only the image but also the sound may be stereo simultaneously. Further, in the mobile communication terminal 10, signal reproduction for the left eye and signal reproduction for the right eye are performed only when necessary, and in a normal case, the screen is displayed with a signal that does not distinguish between left and right. Good. FIG. 4 is a diagram schematically showing a state in which a call is made using two mobile communication terminals 35 and 38 having the same configuration as the above-described mobile communication terminal 10. In FIG. 4, assuming that the user 33 and the other party 34 are communicating, the mobile communication terminal 35 carried by the user 33 has an image display section 36 and a pair of Cameras 37 L and 37 R are mounted. The image display section 36 shows the face of the other party 34, and the face image is a natural-shaped image whose eyes are aligned according to the principle described above with reference to FIGS. 2A to 2C. It is said. On the other hand, the portable communication terminal 38 held by the other party 34 has an image display unit 39 and a pair of cameras 40L and 40R. The image display section 38 shows the face of the user 33, and similarly, the face image has a line of sight according to the principle described above with reference to FIGS. 2A to 2C. It is considered to be a natural shaped image that fits.

Next, a structure for controlling light emitted from the light emitting unit of the image display unit in the portable communication terminal according to the present embodiment to emit light in a predetermined angle direction will be described with reference to FIGS.

FIG. 5 is a diagram when the image display unit of the mobile communication terminal shown as the present embodiment is viewed from above.

In FIG. 5, a region bl indicates a range in which light emitted or reflected from the pixel on the left end of the display surface of the image display unit is strong enough to be seen. Similarly, the region b2 is a range of light emitted or reflected from the rightmost pixel on the display surface of the image display unit. The angle Q continuously changes between the two so that they substantially overlap each other at a distance L apart from the display surface of the image display unit. The distance is a dimension normally assumed as the distance to look at the display surface. Here, this distance L is set to 250 mm, which is called the clear vision distance of a person. The distance L ₂ is the assumed dimensions further consideration to see stretched out arm. Here, this distance L ₂ is assumed to be 400 mm. Further, black dots b 3 and b 4 indicate the positions of the left eye E _L1 and the right eye E _{R 1 at} the distance L of the viewer of the display surface of the mobile communication terminal, respectively. Furthermore, black points b 3 ′ and b 4 ′ indicate the positions of both eyes at a distance L ₂ , respectively.

As is clear from FIG. 5, the light in the regions indicated by the regions bl and b2 does not enter the right eye but can be seen only by the left eye. Similarly, if the light angle is reversed left and right, pixels that can be seen only by the right eye can be set. Therefore, by displaying these images for the left and right eyes for each line and for each pixel with respect to the entire display pixels, stereoscopic vision becomes possible.

The distance D is the horizontal size of the display surface of the image display unit. Usually, the width of the image display section of a portable device is about 20 mm to 80 mm due to a requirement that the image display section can be held in a hand. Here is the distance An example is shown where D is 40 mm. 9.5 7 ± 5 °) and 0 ₂ (= 0.43 ± 5 °) are the design reference values at that time, respectively, at the position of L 丄 = 250 mm, at the center of both eyes The values are designed so that the light spread is 10 ° so that the light for both eyes is not mixed as much as possible and there is no gap, so that the light reaches the eyes sufficiently. The design value even distance L ₂ 4 0 0 mm as described above, it is sufficiently value light reaches the eye. In this setting, the distance at which a part of the display surface becomes theoretically invisible in this setting is geometrically the shortest at 202 mm and the longest at 657 mm. The divergence angle of this ray may be larger than 0.10 ° if the light can be separated into both eyes. However, if this spread is large, it is necessary to increase the exit angle of light rays, which makes optical design difficult. Also, since the object of the present invention is a device that assumes that the user is an individual, it is not necessary to spread the light more than necessary in terms of privacy protection and energy consumption reduction. There are advantages.

Similarly, in the vertical direction, control is performed so that the light beam reaches the position of the eye as shown in FIG. In this vertical direction, the angle Φ is used as the angle parameter for the exit direction, but it is assumed that there is no positional difference for both eyes, and the black points b 3 and b 4 are at the same position

The black spots b 3 ′ and b 4 ′ are at the same position (L ₂ ). With such a design method, it is possible to configure an image display unit capable of displaying images with matching eyes, but this can be further generalized by the following equations. First, comparison between the distance D x _max from the center of the display surface of the half of the distance E _{R 1} and image display radical 113 of binocular interval until the end is performed. That is, the comparison as shown in the following equation (1) is performed. if marauder < ^E if Dx _max > E _R1

That is, when the above condition (1) is met, the distance D x _max from the center to the edge of the display surface of the image display unit is shorter than the distance E _{R 1} which is half the binocular distance. In this case, the size of the display surface of the image display unit is small. In this case, the exit angle of the light beam is set as shown in the following equation (2). =-n- ¹ ()

Θ 0 —10 A _min is the center (yz plane) symmetry η _ίά = ( _-tan - ¹ (Dy / L,)-tan ' ^J (DX / L ₂ )) / 2

Φτηιά '

(2)

In the above equation (2), the distance 1 ^ is a dimension normally assumed as a distance for viewing the display surface, and is, for example, a clear visual distance of a person. The distance L ₂ is the assumed dimensions further consideration to see stretched out arm. Furthermore, the distance Dx is the distance in the X direction (horizontal direction), and the distance Dy is the distance in the y direction (vertical direction). In other words, the calculation can be performed assuming that the position of the pixel for which the angular direction of the ejection is found exists in (D x D y). On the other hand, if the above condition (1) applies to the lower condition, the distance D x _max from the center to the end of the display surface of the image display unit is longer than the half distance E _{R 1} of the binocular distance In this case, the size of the display surface of the image display unit is large. In this case, the exit angle of the light beam is set as shown in the following equation (3).

¾ _max A _min is center (plane) symmetric

η _ίΛ

)) / 2

Φ = +5

Ψτπ3Χ rni'd-

(3) In the above equation (3), the setting of each parameter is the same as that of each parameter in the above equation (2).

By using each of the above equations (1) to (3), it is possible to obtain the angle direction of the light beam emission at an arbitrary position (Dx, Dy) on the image display unit. If the correspondence between the pixel and the right or left eye is random, the emission direction can be determined by calculating each of the pixels using the above equations (1) to (3). It is. For example, when the pixels for the right eye and the left eye are on a line-by-line basis, several points are extracted on the line, and the extracted points are expressed by the above equations (1) to (3). It is also possible to calculate using each of these, and for each point between the extraction points, set the data on the exit angle by a method such as linear interpolation. Next, the position of the camera in the mobile communication terminal shown as the present embodiment will be described with reference to FIG.

The image display section 31 has a structure in which pixels are arranged in a matrix, and its outer shape is substantially rectangular. Although at least one camera is provided on each of the right and left sides, more cameras may be provided, for example, different types of cameras may be provided. One of the left and right cameras is a normal camera, and the other can be combined with a relatively simplified camera for synthesizing the eyes.

The position of each camera includes a position within a predetermined range from the left and right ends of the substantially rectangular image display unit 31 in the horizontal direction and from the upper and lower ends of the left and right ends, for example, in FIG. The camera can be placed in the areas indicated by the areas 32L and 32R. Region 3 2 L, 3 2 R shown in FIG. 7 is a side horizontal left right end of the image display unit 3 1 includes a strip-shaped region having a width H _2, the upper and under end of the substantially rectangular shape The distance from the corner of the image display unit 31 is, for example, within a radius r. In the height direction, it also includes an area that extends above the substantially rectangular image display unit 31 by a distance of 11 inches. As described above, in the vertical direction, there is a relatively high degree of freedom in the position of the camera, but this is because the above-mentioned permissible eyes are not strict in the vertical direction. When is small, the position does not necessarily have to be right next to the image display unit 31, and the same applies to the lower side.

The width H _2, is not particularly limited, for example, when the previous radius r about 2 0 mm likewise the width H ₂ may be about 2 0 mm. It is preferable that the camera position as described above is fixed to the image display unit 31 because each of the optical systems (not shown) is provided. However, the camera itself can be freely moved in and out of the side of the image display unit 31. In addition, both or one of the pair of cameras may be attached at the time of imaging. It may be a structure that can be used. A glass or plastic lens is attached to the front end of the camera, but it can be covered with a cover when not in use to prevent scratches.

Next, an example of the structure of the image display unit will be described with reference to FIGS.

FIG. 8 is an enlarged view of a display pixel, and a region 51 indicated by a substantially square block corresponds to one pixel. Four light-emitting elements 52 R, 52 G, 52 B, and 52 G are arranged in the area 51 of each pixel so as to occupy the positions of the four points of the dice. The light emitting elements 52 R, 52 G, 52 B, 52 G are composed of semiconductor light emitting elements such as light emitting diodes. The light-emitting element 52R is an element that emits red light, the light-emitting element 52G is an element that emits green light, and the light-emitting element 52B is an element that emits blue light. The light-emitting elements 52G that emit green light are more easily resolved by human eyes than elements that emit other light-emitting colors, so the green light-emitting elements 52G are densely arranged. By doing so, it is possible to give a uniform impression. In addition, instead of these light emitting elements, a transmissive display element having a color filter such as a color liquid crystal may be used as an element constituting the image display unit, or some kind of reflective display element. You may.

In order to configure an image display unit capable of performing required stereoscopic viewing by emitting light beams to the left and right eyes using the pixel region 51 as described above, images must be distributed for each line and each pixel. Just set it.

9A to 9E are diagrams showing examples of left and right video distribution patterns for realizing stereoscopic vision. In each figure, "L" is a pixel that generates light according to data for the left eye, and emits a light beam toward the left eye. On the other hand, “R” is a pixel that generates light based on data for the right eye, and emits a light beam toward the right eye. In each figure, Only 4 pixels in width and 4 pixels in height are extracted and shown. FIG. 9A shows a pattern in which pixels for the left eye indicated by "L" and pixels for the right eye indicated by "R" are alternately arranged for each horizontal line. FIG. 9B shows a pattern in which pixels for the left eye indicated by "L" and pixels for the right eye indicated by "R" are alternately arranged for each vertical line. FIG. 9C shows a checkered pattern. This figure shows a pattern in which pixels for the left eye and pixels for the eye appear alternately.One pixel for the left eye and one pixel for the right eye appear alternately on a horizontal line, one pixel at a time. The horizontal line indicates a pattern in which a similar pattern appears shifted by one pixel in the horizontal direction. FIG. 9D shows a case where a checkered pattern is formed for each pixel size of 2 × 2 pixels, similar to FIG. 9C. In addition, in FIG. 9E, two pixels for the left eye and pixels for the right eye appear alternately every two pixels in one horizontal line, and a similar pattern appears horizontally in the current horizontal line and the next horizontal line. This figure shows a pattern that appears to be shifted by each pixel. It should be noted that the patterns shown in FIGS. 9A to 9E are examples, and other patterns can be formed. The entire surface can be formed with the same pattern.However, for example, different patterns may be used on the center side of the image display area and near the periphery. May be arranged. Also, instead of assigning the left-eye pixels and right-eye pixels to a regular pattern, the wiring for the left-eye pixels and the right-eye pixels is assigned in an irregular pattern although the wiring is complicated. You may.

Next, as an example of the structure of the image display unit, an example in which microlenses, which are minute lenses, are arranged on a light emitting element will be described.

FIG. 10 is a diagram showing a state in which four light emitting elements 63G, 63B, 63R composed of semiconductor light emitting elements such as light emitting diodes and semiconductor lasers are arranged for one pixel 61. It is. Light emitting element 63 R emits red light The light emitting element 63 G is an element that emits green light, and the light emitting element 63 B is an element that emits blue light. As described above, the green light-emitting element 63 G emits green light more easily than the other light-emitting elements. By arranging them densely, it is possible to give a uniform impression.

On the surface side of the image display section, which is the light emission side of such light emitting elements 63G, 63B, 63R, minute microlenses 62 made of a spherical transparent body are arranged. The microlens 62 is an emission angle control unit that controls the light from the light emitting elements 63G, 63B, and 63R so as to emit the light in a predetermined angle direction to the left eye or the right eye. It is formed of a transparent synthetic resin such as polymethyl methacrylate) or glass. The shape of the microlens is not limited to a sphere, but may be a cone, a pyramid, or a rectangle. Also, in the image display section, it is also possible to assemble the unit by integrating the micro aperture lenses in a matrix on the holding plate and assemble them by positioning each micro lens individually. You may comprise. Since each microlens 62 has a function of controlling the light emission angle, an opening for each light-emitting element pointing to one of both eyes of the user is provided on a shielding plate or the like for that purpose. Alternatively, a method of shifting the position of the microlens 62 from the optical axis of light from the light emitting elements 63G, 63B, 63R is also possible.

FIGS. 11 and 12 are schematic cross-sectional views for explaining an example in which the emission direction of the light beam is controlled by the position of the microlens 62. FIG. FIG. 12 shows an example in which the axes of the microlens 62 and the microlens 62 coincide with each other. FIG. 12 shows an example in which the axes of the light emitting element 63G and the microlens 62 are misaligned. In FIG. 11, the size of the light emitting element 630 is indicated by ^. ! ^ Is about 30 im, which is representative of a light emitting element 63 G that emits green light. In the figure, the z-axis direction is the normal direction of the display surface of the image display unit, and here, the z-axis direction is the light emission direction. The light-emitting element 63G is, for example, a GaN-based light-emitting diode, and a blue light-emitting diode can also be constituted by a GaN-based semiconductor or the like. Further, the light-emitting element 63R that emits red light may be composed of a GaAs-based compound semiconductor or the like. Each light emitting element 63G is adhered on a support substrate 65, and is arranged in a matrix at a pitch of about 300m to 300m.

On the support substrate 65, a molded holding member 66, which functions as a holding member for the microlenses 62 and also functions as a shielding plate for restricting light from the light emitting element to an appropriate angle, is provided. . The molded holding member 66 has an opening corresponding to the position of each light emitting element. From the opening, the diameter is enlarged with a substantially frustoconical shape, and a micro end is provided at the end opposite to the light emitting element. The black lens 62 is configured to be fitted. The molded holding member 66 and the microlens 62, and the molded holding member 66 and the support substrate 65 are adhered and fixed to each other. The microlenses 62 are connected and held by a holding portion 64 that holds the lens at the maximum diameter portion. The diameter _LENS of each microlens ₆₂ is set to about ₃₀₀ m here. ing. With such a configuration, a gap is formed at the bottom of the microlens 62 attached to the opening of the molded holding member 66 by a distance d between the microlens 62 and the light emitting element 63G, and light passes through the gap. It passes through and is introduced into the micro lens 62.

FIG. 12 shows an example in which the axes of the light emitting element 63G and the microlens 62 are shifted as described above. In FIG. 12, the light emitting element 63 G is positioned at a distance from a line passing through the center of the microlens 62 and parallel to the z-axis. It is arranged at a position shifted by the distance y. If the light-emitting element 63G is arranged at such a position shifted by the distance Δy, light emitted from the light-emitting element 63G is bent from the misalignment of the axis of the microlens 62. From the setting of such a positional relationship, light can be emitted in the direction toward the right eye and the left eye. The method of shifting the axis of the light emitting element 6 3 G and the micro lens 6 2 includes a method of shifting the position of the light emitting element 6 3 on the support substrate 65, a method of shifting the position of the micro lens 62, and a method of shifting the micro lens 6. In a structure in which the lens 2 is attached to the opening of the molding and holding member 66 for each lens, a method of shifting the position of the microlens 62 by shifting the position of the opening of the molding and holding member 66 may be used. In the example of FIG. 12, the method of shifting the position of the microlens 62 is adopted, and the center of the opening of the molded holding member 66 is fixed so that the axis of the microlens 62 does not match. I have. In FIG. 12, only the displacement in the y direction is shown, but not only in the y direction but also in the z direction and the X direction in order to enable light to be emitted in the direction toward the right and left eyes. The deviation may be included.

Fig. 13A simulates the spread of light rays when the axes of the light-emitting element 63G and the microlens 62 are coincident. Fig. 13B shows the light-emitting element 63G and the microlens. This is a simulation of the spread of light rays when the axis is shifted from 62. That is, FIG. 13A shows the relationship between the light emitting element having the structure of FIG. 11 and the microlens, and FIG. 13B shows the relationship between the light emitting element having the structure of FIG. 12 and the microlens. It shows the relationship. When the axes of the light-emitting element 63G and the microlens 62 match, as shown in Fig.13A, the center is about the z-axis, which is the normal direction of the display surface of the image display section. When the axes of the light emitting element 6 3 G and the microlens 62 are misaligned, an angle is given to the light emitting direction as shown in FIG. Slightly diagonally above The light rays will be emitted toward the other side. In this calculation example, the material of the microphone aperture lens is PMMA, the size is 300 m in diameter, the size of the light emitting element is 30 m in diameter, and the distance between the light emitting element and the microlens is Is set to 50 m, FIG. 13A shows the case where Δy = 0, and FIG. 13B shows the case where Δγ−15 m. When the material of the microlens is PMMA, the refractive index changes according to the wavelength. The following table shows the relationship. The calculation is performed using the data shown in the following table.

PMMA refractive index

Further, for the size and positional relationship between the microlens and the light emitting element, optical calculations as shown in FIGS. 14A to 14C are performed.

First, the graph shown in FIG. 14A shows the result of optical calculation of the diameter of the light emitting element and the spread angle of the light beam. The conditions were the same as in the case shown in Fig. 13A, the material of the microlens was PMMA, the size was 300 m in diameter, and the distance between the light emitting element and the microphone aperture lens was 50 ^ m. The size of the light emitting element is changed. Based on this result, Here, if the divergence angle of the light beam is set to the aforementioned 10 °, the size of the light emitting element is preferably about 30 m in diameter. Since the comparison between the size of the microphone aperture lens and the size of the light emitting element is relative, for example, the size ratio between the micro lens and the light emitting element is set to 30: 1 to 5: 1. Is preferred.

The graph shown in Fig. 14B shows the result of calculating the effect of the distance between the light emitting element and the microlens on the spread angle of the light beam. The conditions are the same as in the case of Fig. 14A. The material of the microlens is PMM A, the size is 300 mm, the size of the light emitting element is 3 Om, and the light emitting element and the micro lens are And the distance d is variable. From these results, it can be seen that it is preferable to set the distance between the light emitting element and the micro-aperture lens to about 50 im in order to set the spread angle of the light beam to 10 °. The graph shows the calculation result of the distance A y (= Δχ) indicating the lateral shift between the light emitting element and the microlens and the light emission angle. The upper and lower angles mean the range of 0 in the light spread area shown in FIG. The center is a plot of the center of those angles. From the explanation of Fig. 5, it is necessary to change from 0.43 ° to 19.57 ° depending on the position of the display surface in order to distribute images to the left and right eyes. From the graph shown in Fig. 14C, it can be seen that in order to satisfy this condition, the distance Δy should be changed linearly from approximately 0 to 35 m. Such a linear calculation can be approximately given by the following equation (4).

Θ = V ^ ± 5

3.5 '

Θ: Injection angle

Vy: relative position of the light emitting element in the horizontal direction with respect to the lens

. . . (Four) From the above optical calculations, it is confirmed that dedicated images can be displayed on the left and right eyes using the microlenses. Here, an example of the optical design in the left-right direction has been described, but the same design is also possible in the vertical direction. In addition, here, the size of the element assuming a self-luminous device such as a light emitting diode was discussed. For example, when a liquid crystal, an organic EL display device, a plasma display, or the like is used, the size of the opening is simply controlled. If there is a concern that the amount of light will decrease by simply limiting the light, it is also possible to collect the light from the pixel once with a lens or the like, set the target light emission size, and then apply the above method . The advantages of using a microlens include, as described above, high light use efficiency and low power consumption, as well as reflection of external light obliquely entering the eye. High contrast, high image quality, and the lens effect increases the apparent size of the pixels, which reduces the apparent pixel spacing and visually separates the pixels. For example, it is possible to prevent the image from being seen visually, and to obtain a continuous image with a relatively small number of pixels.

Next, an example of a microlens will be described with reference to FIGS. 15A and 15B.

FIG. 15A is a front view of the microlens, and FIG. 15B is a cross-sectional view of the microlens. The microlenses shown in FIGS. 15A and 15B have a structure in which the individual microlenses formed of substantially transparent spheres are held by a substantially flat holding member at the maximum diameter portion. They are arranged densely. The diameter of each micro lens is, for example, about 300 m. It is also possible to form an image display section by bonding such individual microphone aperture lenses to an array of light emitting elements while holding them on a substantially flat holding member, and it is possible to position individual micro lenses. Since it becomes unnecessary, the manufacturing cost of the mobile communication terminal can be reduced. Next, as an example of the image processing applied to the present embodiment, a method of keeping the position of the eye stable will be described.

As shown in FIG. 1 earlier, the cameras 13 L and 13 R that capture the face are respectively installed on both sides of the image display section 11, and the minor monitor 14 has its own for confirmation. Since the face is displayed, it can be adjusted to some extent so that the face is within the imaging range of the camera. However, the positional relationship between the displayed image of the face of the interlocutor and the camera usually fluctuates greatly when it is hand-held. The shift in the direction of the line of sight does not become extremely large as long as the face of the interlocutor is almost included on the display surface.However, in order to further match the line of sight and reduce the blur of the screen, It is preferable to provide a function for stabilizing the position.

The method of stabilizing the eye position is to provide a margin in the area imaged by the cameras 13L and 13R, and to image an area one size larger than the face. Then, on the display side, the position of the eye of the interlocutor is adjusted by image processing so as to be on a line connecting the cameras 13 L and 13 R and their centers approaching each other, and then displayed. Here, as a method of detecting the position of the eyes from the face image, a widely known image recognition method can be used. As an example, a method based on correlation detection will be described. FIG. 16 is a conceptual diagram illustrating the process of finding the closest to the template from the image. This correlation detection uses a calculation formula of a correlation value represented by the following equation (5). Correlation coefficient covariance variance

(5) In the above equation (5), the coordinate value (i, j) at which the correlation coefficient c; j becomes maximum gives the matching position. In the above equation (5), g is a template image. In this case, images of a standard eye, nose, eyebrows, mouth, etc. are registered in the memory in advance. F is the target display image.

In addition, generally known auto focus and auto tracking methods can be introduced. First, the cameras 13L and 13R are installed at a standard distance (for example, L1 = 250mm) so that the orientation and focus are in focus. Therefore, the distance to the face is determined from the difference between the images captured by the left and right cameras. This is due to the generally known geometric calculation of disparity. FIG. 17 illustrates parameters related to parallax, and the calculation is performed using the following equation (6). Absolute parallax = convergence angle: y _f , y _f

parallax

-5D _F ) where D _F , D _P »E

(6) From the distance calculated using the camera 13 L and 13 R and the calculation formula, the standard The size of the template is adjusted to match the apparent size of the typical face. This size change can be done with a simple matrix calculation. Such a matrix calculation is represented, for example, by the following equation (7). Normalization matrix Scaling matrix

1/0 0 0a 0 0 0 '

0 0 0

MTRX • 0 b 0 0

0 0 1 0 0 0 1 0

0 0 0 1 0 0 0 1 Movement matrix Ζ Axis rotation matrix

X axis rotation matrix y axis rotation matrix

Viewpoint coordinate transformation matrix Perspective transformation matrix Normalized inverse matrix

(7) Instead of such a calculation, it is also possible to register in advance a multi-level difference in size in the memory and use it as a look-up table.

By using such a method, a template of a face part of appropriate size can be used. Then, the input screen is searched using the template, and the position of the face eye is grasped by finding the position having the largest correlation value. Then, the screen is translated or rotated, or enlarged or reduced so that the position of the eye overlaps as much as possible in the left and right images and the center thereof approaches the position corresponding to the center of the camera on the display screen. Also at this time, it is possible to perform appropriate display by using a matrix as shown in the above equation (7) for converting the image. Next, another example of the structure of the image display unit will be described with reference to FIGS. 18A and 18B.

FIG. 18A and FIG. 18B are examples of an image display unit using a cone type lens. FIG. 18A is a cross-sectional view of the image display unit, and FIG. 18B is a lens. It is a perspective view of. In the image display unit, light-emitting elements 74 and 75 such as light-emitting diodes are arranged on a support substrate 77. The light emitting element 74 is formed at a substantially central portion of a gap 76 formed by the molding and holding member 3 so that the direction of the emitted light beam is substantially the z direction, and the light emitting element 75 emits light. It is formed by the molding and holding member 73 to shift the direction of the light beam in the y direction, and is formed by shifting in the y direction from a substantially central portion of the gap 76. The molded holding member 73 is a member formed by molding a required synthetic resin, functions as a holding member for the cone-shaped micro lenses 71 and 72, and restricts light from the light emitting element to an appropriate angle. It also functions as a shield plate for

In the image display section shown in FIGS. 18A and 18B, the micro lenses 71 and 72 have cone shapes, and the large-diameter end on the bottom side has light emitting elements 74 and 7. It is configured so as to face the space 5 via a gap 76, and is arranged such that the narrowed end faces 78, 799 having a small diameter are on the display surface side. In particular, since the axial direction of the microlenses 7 1 and 7 2 coincides with the light emission direction, if the micro lenses 7 1 and 7 2 are inclined in the direction in which light rays should be emitted, the desired light emission The angle can be adjusted. The light from 4, 7 5 can be collected and emitted from the end faces 7 8, 7 9. By using an image display unit with such a structure, micro lenses 71

72 and the molded holding member 73 can be produced by plastic molding technology. Therefore, in such an image display unit, mass production of products at low cost is realized.

Next, with reference to FIG. 19 and FIG. 20, the usefulness of emitting the emitted light with the angle of the emitted light focused within 10 ° will be described.

Figure 19 shows the sphere area S with respect to the angle Φ from the origin, and illustrates the parameters over time. r is the radius of the virtual sphere, and h gives the diameter of the portion where the ray of angle Φ intersects the virtual sphere. Also, S is the sphere area cut off by a cone according to the angle s _rati . Is the ratio of the spherical area S to the global area. FIG. 20 shows the angle φ calculated from the equations shown on the left side of FIG. 19 and the sphere area ratio S _rat ; FIG. In particular, when the angle <角度> force is 10 ° (0.174 rad), the sphere area ratio is S _rat ;. May be an extremely small value of 0.001 920 265. In other words, compared to the case where the hemisphere emits light evenly, the angle of the emitted light beam is narrowed down to within 10 ° and emitted, so that the amount of light of 1 Z 2 63 is sufficient. It is suggested that a good image can be obtained by controlling the angle of the emitted light beam without increasing the amount of light. In addition, this means that the contrast can be increased when driving the light emitting device with the same power. By using the mobile communication terminal described in this embodiment mode, a clear line of sight can be obtained. An interactive screen will be displayed.

FIG. 21 is a cross-sectional view illustrating still another example of the structure of the image display unit. In FIG. 21, light emitting elements 83 and 84 such as light emitting diodes are mounted on a substrate 82, and these light emitting elements 83 and 84 are arranged in a matrix. Micro-diffraction plates 85 L and 85 R are formed on the front side of the substrate 82. Transparent substrate 81 is attached. The micro-diffraction plates 85 L and 85 R have the function of bending the light beams emitted from the light emitting elements 83 and 84 by the diffraction phenomenon, respectively. And sent to the user. Specifically, the micro-diffraction plate 85L controls the exit angle of the light beam so that it can be seen by the left eye of the user, and the micro-diffraction plate 85R can emit the light beam so that it can be seen by the right eye of the user. Control the angle. By such control, the function of synthesizing and viewing it in the cerebrum of the user works, and as a result, as shown in Fig. 2C, the screen that matches the eyes is displayed. This will enable a natural call that matches the gaze.

In the above description, a method of mixing a plurality of pixels performing display based on the left-eye signal and a plurality of pixels performing display based on the right-eye signal in the image display unit is spatially mixed. Although described using an example, the present invention is not limited to this. Display based on the left-eye signal is performed by switching between display based on the left-eye signal and display based on the right-eye signal in a time-division manner. A plurality of pixels and a plurality of pixels performing display based on the right-eye signal may be mixed.

Next, referring to FIGS. 22A, 22B, 23A, and 23B, a brief description will be given of the result of examining the impression of an image in which the gaze direction matches the gaze direction.

Fig. 22A and Fig. 22B are diagrams showing the state of an experiment for examining the impression of gaze direction and gaze coincidence, and Fig. 22A is a diagram when images are captured from the left and right cameras. 22B is a diagram when a camera is assumed to be at the center of the virtual screen. In the case of FIG. 22A, the cameras are on both sides of the virtual screen and have the structure of the mobile communication terminal described as the present embodiment. The distance between the pair of force lenses is 65 mm, which is the same as the distance between both eyes. On these virtual screens, 7 points of interest are set in the vertical direction and 7 points in the horizontal direction. The results of examining the impression at that time are shown in FIGS. 23A and 23B. Note that the distance between the gazing points is 12.5 mm, and this distance is equivalent to an angle of 2.56 °. The subject looked at the point of gaze at a position 280 mm from the virtual screen.

There is not much difference between the case where images are captured from the left and right cameras as shown in Fig. 23A and the case where the camera is at the center of the virtual screen as shown in Fig. 23B. It can be seen that by capturing images from the left and right cameras, the eye-gaze matching effect equivalent to arranging the cameras in the center is obtained. Furthermore, when capturing images from the left and right cameras, there is a strong tendency to give the impression that the eyes are aligned even when the gazing point is changed in the vertical direction, compared to when the camera is at the center of the virtual screen. In other words, by using the mobile communication terminal described in the present embodiment, it is possible to obtain a suitable sense of reality due to eye-gaze matching when proceeding with a dialogue.

As described above, the mobile communication terminal described as the present embodiment has a camera on each of the left and right sides of the image display unit, so that it is possible to interact with the other party while keeping their eyes on the same, thereby providing a sense of presence. It is possible to have a dialogue. In mobile communication terminals, higher light use efficiency enables lower power consumption, and high contrast images can be viewed even in bright outdoor environments. Furthermore, a mobile communication terminal has a structure in which imaging devices are arranged on both left and right sides of a display screen, so that it can be downsized and is extremely useful as a portable device. Next, a further improved version of the above-described portable communication terminal will be described.

In the mobile communication terminal described here, cameras are provided on each of the left and right sides of the image display unit, and three-dimensional display for matching the line of sight with the other party is performed based on the two images captured by the cameras. To do, By interpolating the parallax based on the two images captured by this camera, it is possible to optimize the stereoscopic display for matching the line of sight with the other party to make the image easier to see. .

That is, if the difference between the images captured by the two cameras provided on the left and right sides of the image display unit is large, the parallax becomes too large, so that the fusion cannot be performed as a double image, or Even if the image is formed, it is difficult to see it, which may cause fatigue of the user. Mobile communication terminal described ,, here contrast, as shown in FIG. 2 4, two provided on the left and right sides of the image display unit of the camera R _R, 2 two images captured by R _L A _R , A _L , the parallax is small as if the image was obtained by two virtual cameras V _R , _VL provided at a smaller interval than the two cameras R _R , _RL By generating new images B _R and B _L , the parallax can be set to an appropriate size, thereby realizing an extremely easy-to-see natural image display.

Generally, the disparity is defined as the relative disparity as the difference in the angle of convergence as shown in the above equation (6), but here, for simplicity, the two cameras provided on the left and right sides of the image display unit are used. Therefore, the difference between the corresponding points of the two images captured is treated as the number of pixels. In a mobile communication terminal, an image recognition method based on correlation detection can be used to find a corresponding point between two images. That is, in a mobile communication terminal, as shown in FIG. 25, a predetermined area such as a contour of a face with respect to a background and positions of eyes and nose from an image taken by a camera provided on the left side of an image display unit. Is extracted as a template image g, and a target image に corresponding to the template image g is searched for in the image R captured by the camera provided on the side of the image display unit. Then, the number of pixels having a shift amount between the two is obtained. For example, in a mobile communication terminal, the image processing means shown in FIG. The control circuit 21 extracts an area of 2 u × 2 v with the center coordinate value (i, j) from the image L as a template image g. , P and q are variable, and a search is made for a _2uX by _2v region where the center coordinate value is (i + P, j + q), and by finding a region where the correlation coefficient _pq is maximum. Find the target image ί corresponding to the template image g. At this time, in the mobile communication terminal, since the imaging target is mainly a face image, the search range of the target image f corresponding to the template image g is determined according to the position of the template image g in the image L. Can be limited in advance, and processing can be performed efficiently. Note that this correlation detection uses the equation for calculating the correlation value expressed by the following equation (8).

COV, Re-, '(, g, f)

Correlation coefficient

Covariance ∞ ,,) =-g

if—one / ') variance _var )-(s _m , _n -g') ² variance va _{ri + w + 9} (/) = (f _{m + P} , _{n + q} -rr

(8) In the above equation (8), g ′ indicates the average value of the template image ′ and f ′ indicates the average value of the target image f. (P, q) where が_pq is the maximum indicates the number of pixels of the displacement amount of the target image f with respect to the template image g, and corresponds to the parallax. . Therefore, in the mobile communication terminal, an image of an arbitrary parallax can be generated from one of the images R and L by adjusting the amount of the shift. For example, in the case of a mobile communication terminal, the displacement amount indicating parallax is 1 When an image is generated by moving the image L by the number of pixels of (p / 2, Q / 2) / 2, the image is captured from the center of the two cameras that captured the images R and L, as shown in Fig. 26. Image, that is, an image equivalent to an image of a subject viewed from the front can be generated.

In the case of a mobile communication terminal, if there is no pixel to be drawn at the original position where the pixel is moved, the pixel is filled with a pixel obtained by interpolating from the pixel such as the left and right or the left and right and up and down of the pixel. Deficiency can be avoided. Also, in a mobile communication terminal, when there is a large parallax between the images R and L, if there is a hidden portion called so-called occlusion that appears only in one image and does not appear in the other image, In some cases, an appropriate point of correspondence cannot be found. However, this occlusion occurs even in situations where people are looking at the natural world, and if it is equal to this degree, it will hardly cause any discomfort.

In the mobile communication terminal, the process of calculating the number of pixels of the shift amount of the target image 対応 corresponding to the template image g is preferably performed on the entire image range, and the predetermined pixel based on the number of pixels of the obtained shift amount is determined. Two new images are generated by moving two images R and L by the number. As a result, the mobile communication terminal generates two new images with small parallax as if they were obtained by two virtual cameras provided at a smaller interval than the images R and L. be able to. In the mobile communication terminal, by displaying these two newly generated images on the image display unit, the user can reduce the distance between cameras and reduce the parallax. You can see two images, and you can feel as if a stereoscopic image with very easy-to-view high-quality eyes was displayed.

In particular, in a mobile communication terminal, the parallax required from the two images L is reduced and set to an appropriate arbitrary ratio, so that the left and right This makes it possible to reduce the displacement of the visible image of the subject and to provide a more easily viewable three-dimensional display.

Also, parallax interpolation technology can be used to increase the number of viewpoints. That is, in a mobile communication terminal, usually, only two viewpoint images can be generated from two cameras, but based on these two images, for example, the parallax becomes 1/2, respectively. By generating an interpolated image as described above, an image of four viewpoints can be generated. Also, in a mobile communication terminal, by interpolating so that the parallax is reduced to a predetermined amount, a multi-viewpoint image can be generated, and the obtained images can be obtained by using a lenticular lens or the like. By appropriately performing stereoscopic display, it is possible to reduce a so-called flicking phenomenon in which an image changes rapidly due to a difference in viewpoint position, and it is possible to perform higher quality stereoscopic display.

As a method of searching for the target image f corresponding to the template image g and obtaining the corresponding points of the two images, any method other than the correlation detection can be applied. For example, under the condition that the aperture values of two cameras are not affected and the brightness values of the two images captured by each camera hardly change, a so-called difference value between the brightness values of the two images is used. The method of Sum of Difference can be used, and the so-called residual sum of squares (SSD) method can also be used. However, under the condition that the brightness values of the two images captured by each camera are different, the above-described correlation detection method gives the most accurate result in order to normalize these two images. It goes without saying that it can be obtained.

Further, the mobile communication terminal that performs the process of interpolating parallax based on two images obtained by capturing images by the cameras provided on each of the left and right sides of the image display unit may be the transmitting side. The receiving side may be used.

As described above, the mobile communication terminal described as this embodiment has Interpolation of parallax based on two images obtained by cameras provided on the left and right sides of the display unit optimizes the stereoscopic display to match the line of sight with the other party, Furthermore, since the image can be made natural and easy to see, it does not cause a situation where the parallax becomes too large to be able to fuse the image as a double image, and the fusion becomes so large as to cause fatigue of the user. It is possible to provide extremely excellent convenience without causing a situation in which the formed image is difficult to see.

The technique of interpolating parallax based on two images obtained by imaging with cameras provided on each of the left and right sides of the image display unit is not limited to mobile communication terminals. The present invention can be applied to any information processing device that performs display. Industrial applicability

As described in detail above, according to the present invention, it is possible to have a conversation with a mobile communication terminal while matching the line of sight with a communication partner, and a realistic conversation can be achieved. In addition, the higher light use efficiency enables lower power consumption, and high contrast images can be viewed even in bright outdoor environments. In addition, the structure in which imaging devices are arranged on both sides of the display screen enables miniaturization, which is extremely useful for a portable information processing device.

Further, according to the present invention, by generating a new image in which parallax is interpolated based on two images obtained by imaging by the imaging means, it is possible to optimize a stereoscopic display for matching a line of sight with a communication partner. In addition, since the image can be made more natural and easy to see, it does not lead to a situation where the parallax becomes too large to be able to fuse as a double image, and the fusion becomes so large as to cause fatigue of the user. It is possible to provide extremely excellent convenience without causing a situation in which the formed image is difficult to see.

Claims

The scope of the claims

1. A portable information processing apparatus for performing a dialogue with a video, comprising: an image display means for displaying a required image according to an image signal;

Imaging means provided on each of the left and right sides of the image display means.

An information processing apparatus characterized by the above-mentioned.

2. An image processing unit that generates a new image in which parallax is interpolated, based on the two images obtained by the imaging unit,

Two new images generated by the image processing means are displayed on the display surface of the image display means.

2. The information processing apparatus according to claim 1, wherein:

3. The information processing apparatus according to claim 2, wherein the image processing unit sets the parallax obtained from the two images captured by the imaging unit by reducing the parallax to an arbitrary ratio.

4. The image processing means obtains a corresponding point between the two images obtained by the image capturing means, and captures a predetermined number of pixels by the image capturing means based on the determined number of pixels of the corresponding point shift amount. Moving the two images that were obtained and generating two new images.

3. The information processing apparatus according to claim 2, wherein:

5. The image processing means obtains the corresponding point by performing a correlation detection between two images obtained by the imaging means.

5. The information processing apparatus according to claim 4, wherein:

6. The position where the image pickup means is provided includes a position within a predetermined range from a horizontal side of a left and right end of the substantially rectangular image display means and from upper and lower ends of left and right side ends.

2. The information processing apparatus according to claim 1, wherein:

7. The imaging means is constituted by a solid-state imaging device.

2. The information processing apparatus according to claim 1, wherein:

8. A portable information processing device for performing a dialogue with a video, comprising a portable housing,

Image display means mounted on the surface of the housing and displaying a required image according to an image signal;

Imaging means provided on the surface of the housing and on each of the left and right sides of the image display means.

An information processing apparatus characterized by the above-mentioned.

9. An image processing unit that generates a new image in which parallax is interpolated based on the two images obtained by the imaging unit,

9. The information processing device according to claim 8, wherein:

10. The information processing apparatus according to claim 9, wherein the image processing unit sets the parallax obtained from the two images captured by the imaging unit by reducing the parallax to an arbitrary ratio.

1 1. The image processing means obtains a corresponding point between the two images obtained by the imaging means and obtains a predetermined number of pixels by the imaging means based on the determined number of pixels of the shift amount of the corresponding point. Moving two images acquired and generating two new images

10. The information processing device according to claim 9, wherein:

12. The information processing apparatus according to claim 11, wherein the image processing unit obtains the corresponding point by performing a correlation detection between two images captured by the imaging unit.

13. The housing is formed in a size that can be held by a user with one hand, and the image display means has a horizontal width of about 10 mm from about 10 mm on the display surface.

100 m

The distance between the imaging means on the left and right sides is set to be larger than the horizontal width of the display surface.

9. The information processing device according to claim 8, wherein:

1. The position where the imaging means is provided includes a position within a predetermined range from the upper and lower ends of the right and left ends of the substantially rectangular image display means beside the left and right ends.

14. The information processing apparatus according to claim 13, wherein:

15. The position within the specified range from the upper and lower ends should be within a radius of about 20 mm.

15. The information processing apparatus according to claim 14, wherein:

16 6. A portable information processing device that performs a dialogue with a video, in which a plurality of pixels performing display based on a left-eye signal and a plurality of pixels performing display based on a right-eye signal are mixed. Image display means,

An information processing apparatus characterized by the above-mentioned.

17. An image processing means for generating a new image with parallax interpolated based on the two images obtained by the imaging means,

17. The information processing device according to claim 16, wherein: 18. The information processing apparatus according to claim 17, wherein the image processing unit sets the parallax obtained from the two images captured by the imaging unit by reducing the parallax to an arbitrary ratio. .

1 9. The image processing means obtains a corresponding point of the two images obtained by the imaging means and obtains a predetermined number of pixels by the imaging means based on the determined number of pixels of the shift amount of the corresponding point. Moving two images acquired and generating two new images

18. The information processing apparatus according to claim 17, wherein: 20. The image processing means obtains the corresponding points by performing a correlation check between two images obtained by the imaging means. The information processing device according to claim 19, characterized by:

2 1. The image display means has an emission means for emitting light independently to both eyes of the user.

17. The information processing device according to claim 16, wherein:

2 2. The above injection means

Light emitting means for generating required light based on the left-eye signal or the right-eye signal,

Emission angle control means for controlling the light from the light emitting means to be emitted in a predetermined angle direction.

21. The information processing apparatus according to claim 21, wherein:

2 3. The light emitting means is composed of a plurality of light emitting elements arranged,

The emission angle control means comprises an opening for each of the light emitting elements directed to one of the eyes of the user.

The information processing apparatus according to claim 22, characterized in that:

2 4. The light emitting means is composed of a plurality of light emitting elements arranged,

The emission angle control means comprises a microlens array in which a plurality of microlenses are arranged in a matrix.

The light emitting elements are arranged such that the positions of the light emitting elements and the corresponding microlenses in the in-plane direction of the microlens array are relatively shifted so as to independently emit light to both eyes of the user. thing

25. The ratio between the size of the microlens and the size of the light emitting element is in the range of 30: 1 to 5: 1.

25. The information processing apparatus according to claim 24, wherein: 26. The microlenses should be spherical, conical, pyramidal, or rectangular.

25. The information processing apparatus according to claim 24, wherein:

27. The light emitting means is composed of a plurality of light emitting elements arranged,

23. The information processing apparatus according to claim 22, wherein the emission angle control means is configured by arranging, in a matrix, micro-diffraction plates that emit diffracted light directed to one of the eyes of the user. .

28. The emission angle control means is configured to linearly interpolate the emission angle of each light emitting means between the end on the same side as one eye and the end on the opposite side in the horizontal direction of the image display means. What is provided

29. In the image display means, a plurality of pixels for performing display based on the signal for the left eye and a plurality of pixels for performing display based on the signal for the right eye are mixed with the display based on the signal for the left eye and the right based on time division. Switching between display based on ophthalmic signals

17. The information processing device according to claim 16, wherein: 30. An image display means for displaying an image including the face of the other party is provided, and image pickup means are provided on both left and right sides of the image display means. And a plurality of portable information processing terminals that carry out dialogue with video,

The information processing terminals must be able to communicate with each other

Information processing system characterized by

3 1. On the display surface of the above-mentioned image display means, the face of the other party must be displayed with their eyes aligned with the user.

The information processing according to claim 30, characterized in that:

3 2. Each of the information processing terminals has image processing means for making the position of the line of sight on the display surface substantially fixed on the display surface.

The information processing according to claim 30, characterized in that:

33. Each of the information processing terminals has image processing means for generating a new image in which parallax is interpolated, based on two images obtained by the imaging means.

31. The information processing method according to claim 31, wherein two new images generated by the image processing means are displayed on a display surface of the image display means of the information processing terminal of the other party.

3 4. An image capturing step of capturing a user's image by a pair of imaging means provided on each of the left and right sides of the image display means of the portable terminal, and the user's line of sight is displayed on the image display means of the other party's terminal. A display step of displaying the caller's eyes in a manner that they are matched with each other.

A method for displaying an interlocutor.

3 5. An image processing step of generating a new image with parallax interpolated based on the two images captured in the video capturing step,

Two new images generated in the above image processing process are displayed on the display surface of the image display means of the terminal of the other party

35. The method for displaying a dialogue person according to claim 34, characterized by the following.

36. The image display means is a mixture of a plurality of pixels performing display based on the left-eye signal and a plurality of pixels performing display based on the right-eye signal.

37. The image display means should have emission means for emitting light independently to both eyes of the user.

3 8. The injection means

A light emitting means for generating required light based on a signal for the left eye or a signal for the right eye; and an emission angle control means for controlling light from the light emitting means to emit in a predetermined angle direction.

38. The method for displaying a dialogue person according to claim 37, wherein:

3 9. The above terminals are

A portable housing,

On the left and right sides of the image display means on the surface of the housing Having imaging means provided

40. An information processing device for performing a dialogue with a video,

Image display means for displaying a required image according to the image signal;

Imaging means provided on each of the left and right sides of the image display means, and image processing means for generating a new image in which parallax is interpolated based on two images obtained by the imaging means. Prepared,

An information processing apparatus characterized by the above-mentioned.

4 1. An information processing device that performs a dialogue with a video,

A housing,

Imaging means provided on the surface of the housing and on each of the left and right sides of the image display means;

Image processing means for generating a new image in which parallax is interpolated, based on two images obtained by the imaging means,

An information processing apparatus characterized by the above-mentioned. 4 2. An information processing device that performs a dialogue with a video,

Multiple pixels for display based on left-eye signal and display based on right-eye signal Image display means comprising a mixture of a plurality of pixels performing

An information processing apparatus characterized by the above-mentioned.

4 3. An image display means for displaying an image including the face of the other party is provided, and an image pickup means is provided on each of the left and right sides of the image display means, and information processing for performing a dialogue with a video is provided. A plurality of terminals, each of the information processing terminals includes an image processing unit that generates a new image in which parallax is interpolated based on two images captured by the imaging unit,

When communicating between the information processing terminals, two new images generated by the image processing means are displayed on a display surface of the image display means of the information processing terminal of the other party.

Information processing characterized by

4 4. A video capturing step of capturing a user's video by a pair of capturing means provided on each of the left and right sides of the image display means of the terminal

An image processing step of generating a new image in which parallax is interpolated based on the two images captured in the video capturing step;

A display step of displaying the two new images generated in the image processing step on the image display means of the terminal of the other party so that the user's line of sight coincides with the viewpoint of the other party and displayed. To prepare Interviewer display method characterized by <