JPH08205197A - Human interface device - Google Patents

Abstract

PURPOSE: To provide the human interface device which is suitable for nonlinguistic transmission showing a feeling with expression. CONSTITUTION: This human interface device has a face-shaped screen 1 having a face shape, an image pickup means 4 which picks up an image of a sender 3, an image processing means 5 which extracts a face image as the part corresponding to the face from the image of the sender 3 picked up by the image pickup means 4, and normalizes the tilt, position, and size of the face image to the shape of the face-shaped screen 1, and a projection means 2 which projects the face image normalized by the image processing means 5 on the face-shaped screen 1. A receiver 6 can see the three-dimensional face image of the sender 3 projected on the face-shaped screen 1.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a human interface device suitable for non-verbal communication that expresses emotions through facial expressions.

[0002]

2. Description of the Related Art Generally, when a person conveys his / her intention to a person he / she directly faces, he / she carries out linguistic communication expressing his / her intention by words and non-verbal communication expressing his / her feelings by facial expressions. In some cases, the latter has a large amount of information.
Also, the contents expressed by the same word may have opposite meanings depending on the facial expressions.

On the other hand, when a person conveys his / her intention to a person he / she does not face directly, he / she communicates by letter like a letter and by voice like a telephone. Only verbal communication.

Conventionally, as a technique for performing linguistic communication and non-verbal communication to a person not directly faced, for example, a videophone as disclosed in JP-A-1-194585 is devised. Has been done. When a videophone is used, voice and video can be transmitted and received at the same time, so that linguistic communication and non-verbal communication can be performed, and smoother communication can be performed. In particular, when the conversation contents are emotional, such as when talking to a close person such as a friend, family, or lover, the effect becomes remarkable.

[0005]

However, in a videophone, since all the images captured by the video camera are transmitted to the other party, not only the images of human beings,
Even unnecessary information about the background will be sent. Therefore, the other person looks into one's life, and there is a sense that privacy is violated. As a countermeasure, it is possible to eliminate the background by image processing, but since it is an artificial act, it causes the other party to feel distrust of why it is hidden.

Further, the videophone has a problem of spatial distance as described below. That is, normally, when a person talks, it is said that the distance obtained by the videophone is constant and less than 1 meter, although the spatial distance is naturally used depending on the intimacy with the other party. This is when you talk to a close person,
It is a distance that is a little too far, and it becomes a factor that inhibits non-verbal transmission.

As described above, the technique of the videophone is not sufficient as an interface for non-verbal transmission.

[0008] Further, not only a device such as a videophone that transmits and receives images, but also a device that a person unilaterally presents information to the other party performs both linguistic transmission and non-verbal transmission. In some cases it is effective to be able to do so.

For example, when explaining a method of operating a device to an unspecified number of people, if both linguistic communication and non-verbal communication are performed, it will be possible for a beginner to give an easy-to-understand explanation. .

An object of the present invention is to provide a human interface device suitable for non-verbal communication.

Specifically, it is to realize a human interface device capable of stereoscopically displaying an image of a portion corresponding to the face of a person.

[0012]

In order to achieve the above object, a human interface device of the present invention has a facial shape screen having a facial shape, an image pickup means for picking up a person, and an image picked up by the image pickup means. An image processing means for extracting a face image corresponding to a face from a person image and normalizing the inclination, position and size of the face image with respect to the shape of the face shape screen; and the image processing means. And a projection means for projecting the face image normalized by the above on the face shape screen.

In addition to the above structure, a mannequin may be provided below the face-shaped screen.

In addition to the above configuration, the same unit as the above-mentioned image pickup means is formed, and a microphone that takes in sound from the outside while the image pickup means is picking up an image of a person and the same unit as the above-mentioned projection means are provided. When configured to have a speaker that outputs the sound captured by the microphone to the outside, both linguistic transmission and non-verbal transmission can be performed.

Further, the human interface device of the present invention is a face shape screen having a shape of a face, an image pickup means for picking up an image of a person, and a face corresponding to the face from the image of the person picked up by the image pickup means. Image processing means for extracting an image and normalizing the inclination, position, and size of the face image to the shape of the face shape screen, and the face image normalized by the image processing means via a communication network. A transmitting means for transmitting the facial image, a receiving means for receiving the facial image transmitted through the communication network, and a projecting means for projecting the facial image received by the receiving means onto the facial shape screen. There is.

It should be noted that the same unit as the above-mentioned image pickup means is constituted, and a microphone for taking in sound from the outside is further provided while the image of the person is picked up by the above-mentioned image pickup means, and the above-mentioned transmission means is the above-mentioned image processing means. The voice captured by the microphone is transmitted through the communication network together with the face image normalized by the above, and the receiving means transmits through the communication network together with the face image transmitted through the communication network. When the transmitted sound is received, the same unit as the projecting unit is configured, and a speaker for outputting the sound received by the receiving unit to the outside is further provided, linguistic transmission and non-verbal communication are performed. It is possible to carry out both of them.

Further, the human interface device of the present invention is a face shape screen having a face shape, an image pickup means for picking up an image of a person, and a face corresponding to the face from the image of the person imaged by the image pickup means. Image processing means for extracting an image and normalizing the inclination, position, and size of the face image to the shape of the face shape screen, and recording the face image normalized by the image processing means in a storage medium. The recording means and the projection means for projecting the facial image recorded in the storage medium by the recording means onto the facial shape screen are provided.

It should be noted that the same unit as the above-mentioned image pickup means is formed, and a microphone for taking in sound from outside while the image of the person is picked up by the above-mentioned image pickup means is further provided, and the above-mentioned recording means is the above-mentioned image processing means. The sound captured by the microphone is recorded on a storage medium together with the face image normalized by the above, and the same unit as the projecting unit is configured, and the projecting unit projects the face image on the face shape screen. While
When a speaker for outputting the sound recorded in the storage medium by the recording means to the outside is further provided,
It is possible to make both verbal and non-verbal communication.

In any of the human interface devices having any structure, the image processing means extracts a skin color portion of the image as a face image based on the color information of the image of the person imaged by the image capturing means. can do. Further, the image processing means extracts positions corresponding to images of two or more predetermined face portions from the extracted face images, and the face images are straightened based on these positions. , The inclination of the facial image is normalized, and one of the extracted two or more positions
The position of the face image is normalized so that one position corresponds to the position corresponding to the position on the face shape screen, and the face shape screen of the face image is calculated from the distance between the extracted two or more positions. The size of the facial image can be normalized so that the size of the facial image matches the size of the facial shape screen based on the obtained ratio.

[0020]

According to the human interface device of the present invention, when a human conveys his / her intention to a person he / she is not directly facing, a non-verbal communication that expresses an emotion by a facial expression by displaying a three-dimensional image of the face It can be performed.

Therefore, when the human interface device of the present invention is applied to a conventional device for non-verbal communication, smoother communication can be performed.

Further, since the background is not projected due to the shape of the face-shaped screen unlike the conventional videophone, the invasion of privacy does not occur. Furthermore, for users who see the image projected on the facial shape screen,
It is possible to give a sense of realism that the other party is near.

[0023]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram of the human interface device of the first embodiment.

In FIG. 1, 1 is a face-shaped screen,
Reference numeral 2 is a projection means, 3 is a sender, 4 is an imaging means, 5 is an image processing means, and 6 is a receiver.

The image of the sender 3 picked up by the image pickup means 4 is extracted by the image processing means 5 in a portion corresponding to the face, and the inclination, position and size of the face shape screen 1 are extracted.
After being normalized to the shape of, the image is projected on the facial shape screen 1 by the projection means 2, so that the recipient 6
The face of the sender 3 can be seen through the facial shape screen 1.

FIG. 2 is a diagram showing an image processing step of the image processing means 5.

First, the image processing means 5, when the image of the sender 3 is picked up by the image pickup means 4 as shown in FIG. 2A, the picked-up image is shown as shown in FIG. 2B. Then, a part corresponding to the face is extracted from the image, and subsequently, as shown in FIG. 2C, a predetermined 2 is extracted from the extracted image of the face.
The positions corresponding to the images of three or more face parts are extracted, and the inclination of the face is corrected with reference to these positions.

Here, a case where the image processing means 5 extracts the tip of the nose and the lower jaw is described as an example. Further, the reason why the image processing means 5 corrects the inclination of the face is that the image displayed on the face shape screen 1 is straight even if the sender 3 tilts the face, for example.

Then, as shown in FIG. 2 (d), the face image and the face shape screen are moved by translating the face image so that the extracted position of the nasal tip matches that of the face shape screen 1. After matching the position of 1,
From the distance between the tip of the nose and the lower jaw, the ratio of the facial image to the size of the facial shape screen 1 is calculated, and the size of the facial image matches the size of the facial shape screen 1 based on the calculated ratio. As described above, the face image is enlarged or reduced to be corrected.

At this time, the image processing means 5 can change the color tone of the image of the face so that the complexion looks healthy and change the shape of the face for display. .

The image of the face corrected in this way is
As shown in FIG. 2E, the image is projected on the facial shape screen 1 by the projection means 2.

Next, the configuration and operation of the image processing means 5 will be described in detail with reference to FIG.

FIG. 3 is a block diagram of the image processing means 5.

In FIG. 3, 30 is a background processing unit, 31 is a memory, 32 is a computing unit, 33 is a memory control unit, 34 is a control unit, 35 is an extraction unit, 36 is an input terminal, and 37 is an output terminal.

In the image processing means 5, the image picked up by the image pickup means 4 is transferred from the input terminal 36 to the background processing section 3.
0 and the extraction unit 35.

The extraction unit 35 extracts a portion corresponding to the face from the image input from the input terminal 36.

That is, since the portion corresponding to the face is a skin color, the control unit 34 outputs color information representing the skin color to the extraction unit 35, and the extraction unit 35 outputs the color of the image input from the input terminal 36. Based on the information, the flesh-colored portion of the image is regarded as the portion corresponding to the face and extracted.

The extraction unit 35 also detects the edges of the nose and lower jaw by detecting edges in the extracted image of the face, and outputs the positions of the extracted nose and lower jaw to the control unit 34. In addition, as for the method of extracting a specific object from a video image as described above, for example, Japanese Patent Application No. 2-
It is described in Japanese Patent No. 325643 and Japanese Patent Application No. 5-011732.

Further, the extraction unit 35 outputs a binary signal representing the extracted image of the face to the background processing unit 30.

On the basis of the binary signal output from the extraction section 35, the background processing section 30 outputs the video as it is for the portion corresponding to the face of the video input from the input terminal 36. For the other parts, the background is erased by not outputting the image or outputting a signal representing black, and only the image of the face is output.

The image of the face output from the background processing unit 30 in this manner is stored in the memory 31. The memory 31 is a frame memory that can store one frame of video.

The image of the face stored in the memory 31 is read by the memory control unit 33, the inclination of the face is corrected by the calculation unit 32, and the position and size are corrected, and then the output terminal 37. Is output from.

That is, the control unit 34 outputs the positions of the tip of the nose and the lower jaw to the calculation unit 32, and the calculation unit 32 uses the positions of the tip of the nose and the lower jaw output from the control unit 34 as a reference. Correct tilt. Further, the calculation unit 32
The image of the face is moved in parallel so that the position of the tip of the nose output from the control unit 34 matches the position of the screen of the shape of the face 1 to match the image of the face with the position of the screen of the shape of the face 1 and then the tip of the nose. The ratio of the facial image to the size of the facial shape screen 1 is calculated from the distance between the part and the lower jaw end, and the size of the facial image matches the size of the facial shape screen 1 based on the calculated ratio. As described above, the face image is enlarged or reduced to be corrected.

Note that these corrections are performed in order to match the inclination and size of the projected image on the facial shape screen 1 whose inclination and size are fixed, and are realized by affine transformation. To be done. The affine transformation is described, for example, on page 49 of "Industrial Image Processing" published by Shokodo in May, 1988.

Next, a concrete configuration example of the face shape screen 1 and the projection means 2 will be described with reference to FIGS.

FIG. 4 shows an example in which a projector 201 is used as the projection means 2 for projecting from the front onto the face-shaped screen 1 which is a head image made of a light-colored opaque material such as white.

In this case, a mannequin or a head of a plaster image can be used as the facial shape screen 101. However, considering the projection from the front, the recipient 6 can see the facial shape screen 101. The distance between the facial shape screen 101 and the projector 201 must be increased. Also, the facial shape screen 10
Since there is an optical path between 1 and the projector 201,
Care must be taken to prevent the recipient 6, another person, and another object from blocking the optical path.

FIG. 5 shows the projector 2 as the projection means 2.
No. 01 is used as the facial shape screen 1, and an example of using the facial shape screen 102 made of a semitransparent material and having the shape of the front half of the head is shown.

In this case, a mirror may be arranged between the projector 201 and the face-shaped screen 102 to bend the optical axis, if necessary. Further, the facial shape screen 102 is preferably a dark color such as black.

Further, instead of the face-shaped screen 102, a face-shaped screen 103 in which a transparent plate-shaped member is hollowed out into the face shape from the rear side may be used. In this case, it is necessary to make the facial shape on the rear surface of the facial shape screen 103 semitransparent.

In the example shown in FIG. 5, since the image is projected from the rear side, the receiver 6 does not block the optical path, and the projector 201 is arranged near the face-shaped screens 102 and 103 and is housed in one housing. However, there is a drawback in that the facial shape screens 102 and 103 can have only the shape of the front half of the head.

FIG. 6 shows the projector 2 as the projection means 2.
No. 01 is used as the facial shape screen 1, and the facial shape screen 104 having a hollow head image is used as the facial shape screen 1.

In this example, the projector 201 projects from below the face-shaped screen 104, and the mirrors 105A and 105B arranged in the face-shaped screen 104 project the light onto the front face of the face-shaped screen 104. In this case, by making the mirror 105A non-planar, it is possible to project on a wider area of the facial shape screen 104, but it is corrected beforehand by image processing so that the projected image is normally projected. You need to do it.

In the example shown in FIG. 6, the face-shaped screen 104 can have the shape of the entire head, and the projector 201 can be placed near the face-shaped screen 104 and housed in one housing. Is possible.

FIG. 7 shows an example in which a display device 202 provided with a liquid crystal, a cathode ray tube or the like is used as the projection means 2.

In this case, the face-shaped screen 106 having a face-shaped front surface, which is composed of optical fibers laminated vertically to the screen of the display device 202, is used as the display device 2.
A stereoscopic image can be obtained by arranging it in front of the screen of 02. At this time, since the human face is normally long, if the screen of the display device 202 is used vertically, the screen area of the display screen 202 can be efficiently used, and the resolution is increased. Also, by making the screen of the display device 202 a facial shape, a stereoscopic image can be similarly obtained.

In the example shown in FIG. 7, the display device 202 can be brought into close contact with the face-shaped screen 106 to achieve further miniaturization, but the face-shaped screen 106 can be formed only in the shape of the front half of the head. There is a drawback that.

In any of the above-mentioned examples, the face shape screen 1 does not need to reproduce the face modeling in detail, and it is better to omit the well-moving parts such as eyes and mouth and to make the surface flat to reproduce the image. Is good. This looks natural when the image of the sender 3 with the mouth closed is projected on the face-shaped screen 1 having the shape of a closed mouth, but when the image of the sender 3 with the mouth opened is projected, the image looks unnatural. Because it looks natural.

If the size of the face-shaped screen 1 is too large, the receiver 6 will feel a sense of pressure. Therefore, it is preferable that the size is smaller than the actual size.

Further, as will be described later, when the human interface device of this embodiment is applied as an additional means for transmitting a voice, if a speaker for generating a voice signal is arranged near the face-shaped screen 1, it will be realistic. The feeling increases.

The image pickup means 4 can be realized by a video camera or the like, and the image processing means 5 can be realized by a microprocessor or the like. further,
The path between the image pickup means 4 and the image processing means 5 or the path between the image processing means 5 and the projection means 2 may be realized by a simple cable, but as will be described later, a communication means is interposed. It is also possible to let.

As described above, according to the human interface device of the present embodiment, the receiver 6 can see the three-dimensional image of the face of the sender 3 displayed on the face shape screen 1. Non-verbal communication from the sender 3 to the receiver 6 can be effectively performed.

Next, an example in which the human interface device of this embodiment is applied to an intercom will be described with reference to FIG.

This is an example in which the face-shaped screen 1 is provided instead of the monitor of the intercom 107.

In the respondent 301, the customer 601 is the intercom 1
When a call is made by 07, when the user responds to the image pickup means 4 (here, the video camera 401),
An image of the face of the person 301 is displayed on the face-shaped screen 1 provided on the intercom 107, and in this state, the person 301 and the customer 601 can talk with each other.

Thus, for example, an office receptionist,
It is not necessary for the respondents 301, such as a department store guide and a drive-through order clerk, to be stationed in the place, and only when called, the customer 601 can smile and respond. In addition, the customer 601 is less likely to make a mistake in listening to the voice due to the linguistic transmission due to the non-verbal transmission performed by the respondent 301.

Next, an example in which the human interface device of this embodiment is applied to a clothing mannequin will be described with reference to FIG.

This is an example in which the face-shaped screen 1 is provided on the head of the clothing mannequin 108, and the image pickup means 4 (here, the video camera 402) is arranged near the clothing mannequin 108 placed in the store. An image of the face of the customer 302 captured by the video camera 402 is displayed on the face shape screen 1.

As a result, the customer 302 receives the clothing mannequin 1
Since it is possible to visually check the image when the user wears the clothes worn by 08, it is possible to consider purchasing.

An example of a human interface device in which a communication means is interposed in the path between the image processing means 5 and the projection means 2 will be described below as a second embodiment.

FIG. 10 is a block diagram of the human interface device of the second embodiment.

In FIG. 10, 1A and 1B are face shape screens, 2A and 2B are projection means, 3A and 3B are users, 4A and 4B are image pickup means, 5A and 5B are image processing means, and 7A and 7B are communication means. Is.

The configurations and operations of the image processing means 5A and 5B, and the specific configuration examples of the facial shape screens 1A and 1B and the projection means 2A and 2B are the same as those in the first embodiment.

User 3 imaged by imaging means 4A
As for the image A, the image of the face is extracted by the image processing means 5A, and the inclination, position and size are the face shape screen 1B.
After being normalized to the shape of, the communication means 7A transmits the data via a telephone line or the like. And the communication means 7B
Is received by the projection means 2B and projected on the face shape screen 1B by the projection means 2B, so that the user 3B can see the face of the user 3A through the face shape screen 1B.

On the other hand, from the image of the user 3B imaged by the image pickup means 4B, the image of the face is extracted by the image processing means 5B, and the inclination, position and size are normalized with respect to the shape of the face shape screen 1A. Then, the data is transmitted by the communication means 7B via a telephone line or the like. Then, it is received by the communication means 7A and is projected on the facial shape screen 1A by the projection means 2A, so that the user 3A
The face of the user 3B can be seen through the face shape screen 1A.

Next, an example in which the human interface device of this embodiment is applied to a videophone will be described with reference to FIG.

The example shown in FIG. 11A is a videophone 1.
Facial shape screen 1 instead of 09 display
In this case, the image pickup means 4 is the image pickup means of the videophone 109 (here, the video camera 403).
To use.

Further, as shown in FIG. 11B, the laminated fiber of the face-shaped screen 106 described with reference to FIG. 6 is perpendicular to the display (here, the liquid crystal display 404) of the videophone 109. By arranging the human interface device in front of the liquid crystal display 404, the human interface device according to the present embodiment can be applied to a conventional videophone.

As a result, for example, a person who has a close relationship, such as a single employee, his / her family, and a lover who has a long-distance relationship, can talk while feeling the other party closer.

Next, FIG. 12 shows an example in which the human interface device of this embodiment is applied to a video conference call.
Will be explained.

In the example shown in FIG. 12, the video conference telephone 11 is used.
This is an example in which a plurality of face-shaped screens 1 are provided instead of the 0 display. In this case, the image pickup means 4 is the image pickup means of the video conference telephone 110 (here, the video camera 4).
05) is used. As a result, the caller can hold a meeting while watching the powerful facial image of the other caller, and can have heated discussions.

By the way, in a videophone or a videoconference telephone, since the display and the video camera are usually arranged adjacent to each other, the optical axis between the display and the caller, and the video camera and the caller. The callers cannot face each other through the display because the optical axes between and do not match. As described above, if the front image cannot be obtained, there is a problem that the line-of-sight of both callers do not coincide with each other, resulting in excessive line-of-sight negotiations and a delay in conversation.

In order to solve this, it is conceivable to make the optical axes of the two coincide with each other by using a half mirror or the like. However, in the present embodiment, when the image is projected from the front,
As shown in FIG. 13A, based on the position of the facial shape screen 1, the optical axis between the facial shape screen 1 and the projection means 2 and the optical axis between the user 3 and the imaging means 4. If the projection means 2 and the image pickup means 4 are installed so that they are kept parallel to each other, a front image can be easily obtained on the facial shape screen 1. This is a facial shape screen 1
Even if the image of the user 3 facing in the direction is obliquely imaged, the image is projected on the face shape screen 1 at the same angle when the image is projected, so that the user 3 is corrected as if facing the front.

When projecting an image from the back,
As shown in FIG. 13B, based on the position of the facial shape screen 1, the optical axis between the user 3 and the imaging means 4 with respect to the straight line connecting the facial shape screen 1 and the user 3 And the face shape screen 1 with respect to the straight line
When the projection means 2 and the image pickup means 4 are installed with the straight line sandwiched so that the angles formed by the optical axes of the projection means 2 and the projection means 2 are the same, a front image can be obtained on the facial shape screen 1. .

Further, as shown in FIG. 14, after the image processing means 8 performs the same image processing as the image processing means 5 on the two images picked up by at least two image pickup means 4A and 4B, Further, one of the two images closer to the front image may be selected.

As shown in FIG. 15, the angle sensor 9 attached by the user 3 detects the direction in which the user 3 is facing, and a front image is obtained based on the detected direction. Alternatively, the projection means 4 may move on the moving table.

An example of the human interface device in which the image recording means is interposed in the path between the image processing means 5 and the projection means 2 will be described below as a third embodiment.

FIG. 16 is a block diagram of the human interface device of the third embodiment.

In FIG. 16, 1 is a facial shape screen, 2 is a projection means, 3 is a sender, 4 is an image pickup means, 5 is an image processing means, 6 is a receiver, and 10 is an image recording means.

The configuration and operation of the image processing means 5 and the concrete configuration examples of the face shape screen 1 and the projection means 2 are the same as those in the first embodiment.

After the image of the face taken from the image of the sender 3 picked up by the image pickup means 4 is extracted by the image processing means 5, its inclination, position and size are normalized with respect to the shape of the facial shape screen 1. Is recorded in the image recording means 10. The image recorded in the image recording means 10 is displayed on the facial shape screen 1 by the projection means 2 at an arbitrary time point, so that the receiver 6 can see the sender 3's face through the facial shape screen 1 at any time. it can.

Next, an example in which the human interface device of this embodiment is applied to an electronic memo device will be described with reference to FIG.

The example shown in FIG. 17 is an electronic memo device 111.
This is an example in which the facial shape screen 1 is provided instead of the display of No. 1 in this case, and in this case, the image pickup unit 4 uses the image pickup unit (here, the video camera 406) of the electronic memo device 111.

When the user operates the operation button 114 indicating recording, the video camera 406 captures the image of the user. At this time, the microphone 112 simultaneously inputs the content spoken by the user. A face image is extracted from the image captured by the video camera 406, and its inclination, position, and size are normalized with respect to the shape of the face shape screen 1, and then the internal semiconductor memory is used together with the sound input by the microphone 112. Recorded in.

In addition, the user operates the operation button 11 indicating reproduction.
4 is operated, the image recorded in the internal semiconductor memory is displayed on the shape screen 1 by the projection means 2 provided inside the electronic memo device 111, and the voice recorded in the internal semiconductor memory is output to the speaker. It is output from 113.

As a result, the user can convey the message to the other party who is not present by both linguistic communication and non-verbal communication, so that compared with the conventional electronic memo device. , The message will be transmitted correctly.

By the way, in all of the above-mentioned embodiments, the image picked up by the image pickup means 4 is projected onto the facial shape screen 1, but it is not the image picked up by the image pickup means 4 but by the computer. The generated graphic data is used for the facial shape screen 1
You may make it project on.

An example of a human interface device for projecting computer-generated graphic data on the facial shape screen 1 will be described below as a fourth embodiment.

FIG. 18 is a block diagram of the human interface device of the fourth embodiment.

In FIG. 18, 1 is a face shape screen, 2 is a projection means, 6 is a user, and 11 is a computer.

Specific examples of the structures of the facial shape screen 1 and the projection means 2 are the same as those in the first embodiment. Conventionally, the graphic data generated by the computer 11 is displayed on a display device such as a monitor display and presented to the user 6. However, in FIG. The image data is represented by the projection means 2 and is projected onto the facial shape screen 1.

Next, an example in which the human interface device of this embodiment is applied to an information processing terminal will be described with reference to FIG.

The example shown in FIG. 19 is an information processing terminal 115.
This is an example in which the face-shaped screen 1 is provided in addition to the monitor display 116.

On the face shape screen 1, the information processing terminal 1
By displaying a video image of the face explaining the operation method of No. 15, it becomes easier for the user to understand the content of the explanation than when the explanation is made only by voice or characters, and it is as if the person presenting the explanation is on the side. You can get a sense of security. Therefore, such an information processing terminal 11
It is preferable that 5 is used by an unspecified number of users such as an automatic teller machine and a game machine.

Further, in all the above-mentioned embodiments,
An image is displayed on the face shape screen 1 having a predetermined shape.
Instead of, a stereoscopic screen whose shape changes according to the image may be used.

An example of a human interface device using a stereoscopic screen will be described below as a fifth embodiment.

FIG. 20 is a block diagram of the human interface device of the fifth embodiment.

In FIG. 20, 3 is a sender, 2 is a projection means, 4 is an image pickup means, 5 is an image processing means, 6 is a receiver, 1
Reference numeral 2 is a stereoscopic screen, 13 is a stereoscopic information extraction means, and 14 is an actuator control means.

The structure and operation of the image processing means 5 are the same as those in the first embodiment.

The image of the sender 3 picked up by the image pickup means 4 is extracted by the image processing means 5 in the portion corresponding to the face, and the inclination, position and size are normalized with respect to the shape of the stereoscopic screen 12. .

At this time, the three-dimensional information extracting means 13 calculates the three-dimensional information which is the distance between each part of the face and the image pickup means 4 from the shadow of the image of the face, etc. And the actuator control means 14
The size of the face extracted by the three-dimensional information extraction means 13,
The shape of the three-dimensional screen 12 is changed so as to match the shape.

As a result, the image of the face whose inclination, position and size are normalized by the image processing means 5 is displayed on the three-dimensional screen 12 by the projection means 2,
The recipient 6 can see the sender 3's face through the stereoscopic screen 12. In this way, the 3D screen 1
2 makes it easier to adapt to a specific face than the face shape screen 1 having a predetermined shape, and it is also possible to display an image of a shape other than the face.

Next, a concrete configuration example of the three-dimensional screen 12 will be described with reference to FIGS. 21 and 22.

FIG. 21 shows an example in which the three-dimensional screen 12 is realized by the pin actuators 1201 assembled in a grid pattern.

Each of the pin actuators 1201 comprises a pin 1202 and a solenoid (or motor) 1203 which moves the pin 1202 back and forth.

Therefore, the actuator control means 14
By controlling the solenoid 1203 to move the individual pins 1202 back and forth, a three-dimensional shape can be created.
If an image is projected from the front onto 201, a stereoscopic image can be obtained.

Further, by sticking the stretchable film 1204 on the projection surface of the pin actuator 1201, the surface of the three-dimensional screen 12 can be made smooth.

FIG. 22 also shows an example in which the three-dimensional screen 12 is realized by the pin actuators 1201 assembled in a lattice pattern. Here, the pin actuator 1 is used.
Each of 201 is composed of a pin 1205 made of a light-transmitting material such as an optical fiber, and a solenoid (or motor) 1203 for moving the pin 1205 back and forth.

As described above, when the pin 1205 made of a light-transmitting material is used, the pin actuator 1201
You will be able to project images from behind. The same effect can be obtained by attaching the light emitting element to the tip of each pin.

Next, another example of the human interface device using the three-dimensional screen 12 will be described as a sixth embodiment.

FIG. 23 is a block diagram of the human interface device of the sixth embodiment.

In FIG. 23, 2 is a projection means, 3 is a sender, 4A and 4B are image pickup means, 6 is a receiver, 8 is an image processing means, 12 is a three-dimensional screen, 14 is an actuator control means, and 15 is a three-dimensional method. Extraction means 16 is a stereoscopic image processing means.

The three-dimensional information extracting means 15 is provided for the sender 3 imaged by at least two imaging means 4A, 4B.
Using the parallax of one image, the three-dimensional information that is the distance between the position of each part of the face and the imaging unit 4A, 4B from the position of each corresponding point in the two images and the angle of the imaging unit 4A, 4B. By calculating, the three-dimensional information representing the three-dimensional shape is extracted. Further, the image processing means 8 performs the same image processing as the image processing means 5 on the two videos of the sender 3 imaged by the at least two imaging means 4A and 4B, and then further performs the image synthesis processing. .

The three-dimensional image processing means 16 can process the three-dimensional information extracted by the three-dimensional information extracting means 15 and the face image combined by the image processing means 8.

The actuator control means 14 of the stereoscopic screen 12 is based on the stereoscopic information extracted by the stereoscopic information extracting means 15 (or the stereoscopic information after processing if processed by the stereoscopic image processing means 16). The shape is deformed, and the projection unit 2 projects the image of the face synthesized by the image processing unit 8 (or the image after processing when processed by the three-dimensional image processing unit 16) onto the three-dimensional screen 12.

As a result, if the stereoscopic image processing means 16 does not process the stereoscopic information, the shape of the stereoscopic screen 12 is transformed into a similar figure of the sender 3's face, and the stereoscopic image processing means 16 creates a stereoscopic image. When the information is processed, it is transformed into a shape different from the face of the sender 3.
In addition, the image of the face projected on the three-dimensional screen 12 is
If the face image is not processed by the stereoscopic image processing unit 16, it becomes the sender 3's face, and if the face image is processed by the stereoscopic image processing unit 16,
The face has a color different from the face of the sender 3.

Therefore, for example, when examining the cosmetic surgery, it is convenient to use the human interface device of the present embodiment because it is possible to obtain a facial image after surgery. Further, when considering the purchase of cosmetics, it is convenient to use the human interface device of the present embodiment because it is possible to obtain an image of the face after makeup.

Furthermore, the fifth embodiment and the sixth embodiment
In the embodiment, the example in which the image of the face is projected on the stereoscopic screen 12 is shown, but an image other than the image of the face can be projected.

An example of a human interface device for projecting an image other than the facial image on the stereoscopic screen 12 will be described below as a seventh embodiment.

FIG. 24 is a block diagram of the human interface device of the seventh embodiment.

In FIG. 24, 2 is a projection means, 6 is a user, 12 is a three-dimensional screen, 14 is an actuator control means, 17 is a database, and 18 is a three-dimensional forming means.

The database 17 stores three-dimensional object data composed of images and three-dimensional information about a three-dimensional object. The three-dimensional structure unit 18 extracts, from the three-dimensional object data stored in the database 17, three-dimensional object data in a predetermined range based on the position when viewed from an arbitrary viewpoint, and the actuator control unit. 14 deforms the shape of the stereoscopic screen 12 based on the stereoscopic information in the stereoscopic object data extracted by the stereoscopic composition unit 18, and the projection unit 2 images the stereoscopic object data extracted by the stereoscopic composition unit 18. Is projected on the three-dimensional screen 12.

Next, an example in which the human interface device of this embodiment is applied to a stereoscopic topographic map display device will be described with reference to FIG.

The example shown in FIG. 25 is an example in which the stereoscopic screen 12 is provided as a monitor display of the stereoscopic topographic map display information processing terminal 117.

For example, if data relating to a three-dimensional topographic map is stored in the database 17 and the data of a portion corresponding to a disaster area is extracted when a disaster bulletin is issued by a broadcasting station when a disaster occurs, the data of the disaster area is three-dimensional. Since the topographic map can be quickly displayed on the three-dimensional screen 12, it is possible to accurately explain the situation at the disaster site.

[0137]

As described above, according to the human interface device of the present invention, when a human conveys his / her intention to a person who is not directly faced, a human face image is stereoscopically projected to express emotions by facial expressions. You can make a non-verbal communication that represents.

Therefore, if the human interface device of the present invention is applied to a conventional non-verbal transmission device, smoother communication can be performed.

[Brief description of drawings]

FIG. 1 is a configuration diagram of a human interface device according to a first embodiment.

FIG. 2 is an explanatory diagram showing an image processing step of image processing means.

FIG. 3 is a configuration diagram of an image processing unit.

FIG. 4 is an explanatory diagram showing a specific configuration example of a facial shape screen and a projection unit.

FIG. 5 is an explanatory diagram showing a specific configuration example of a facial shape screen and a projection unit.

FIG. 6 is an explanatory diagram showing a specific configuration example of a facial shape screen and a projection unit.

FIG. 7 is an explanatory diagram showing a specific configuration example of a facial shape screen and a projection unit.

FIG. 8 is an explanatory diagram showing an example in which the first embodiment is applied to an intercom.

FIG. 9 is an explanatory diagram showing an example in which the first embodiment is applied to a clothing mannequin.

FIG. 10 is a configuration diagram of a human interface device according to a second embodiment.

FIG. 11 is an explanatory diagram showing an example in which the second embodiment is applied to a videophone.

FIG. 12 is an explanatory diagram showing an example in which the second embodiment is applied to a video conference call.

FIG. 13 is an explanatory diagram showing a method of obtaining a front image when the user does not face the front.

FIG. 14 is an explanatory diagram showing a method of obtaining a front image when the user does not face the front.

FIG. 15 is an explanatory diagram showing a method of obtaining a front image when the user does not face the front.

FIG. 16 is a configuration diagram of a human interface device according to a third embodiment.

FIG. 17 is an explanatory diagram showing an example in which the third embodiment is applied to an electronic memo device.

FIG. 18 is a configuration diagram of a human interface device according to a fourth embodiment.

FIG. 19 is an explanatory diagram showing an example in which the fourth embodiment is applied to an information processing terminal.

FIG. 20 is a configuration diagram of a human interface device according to a fifth embodiment.

FIG. 21 is an explanatory diagram showing a specific configuration example of a stereoscopic screen.

FIG. 22 is an explanatory diagram showing a specific configuration example of a stereoscopic screen.

FIG. 23 is a configuration diagram of a human interface device according to a sixth embodiment.

FIG. 24 is a configuration diagram of a human interface device according to a seventh embodiment.

FIG. 25 is an explanatory diagram showing an example in which the seventh embodiment is applied to a stereoscopic topographic map display device.

[Brief description of reference numerals]

1 ... Facial shape screen, 2 ... Projection means, 3 ... Sender,
4 ... Imaging means, 5, 8 ... Image processing means, 6 ... Recipient, 7
... communication means, 9 ... angle sensor, 10 ... image recording means, 1
DESCRIPTION OF SYMBOLS 1 ... Computer, 12 ... 3D screen, 13 ... 3D information extraction means, 14 ... Actuator control means, 15 ...
Stereoscopic method extracting means, 16 ... Stereoscopic image processing means, 17 ... Database, 18 ... Stereoscopic forming means, 30 ... Background processing section, 3
DESCRIPTION OF SYMBOLS 1 ... Memory, 32 ... Operation part, 33 ... Memory control part, 34
... control unit, 35 ... extraction unit, 36 ... input terminal, 37 ... output terminal.

Continuation of front page (51) Int.Cl. ⁶ Identification number Reference number within the agency FI Technical indication location H04N 5/66 Z (72) Inventor Norio Yatsuda No. 292 Yoshida-cho, Totsuka-ku, Yokohama-shi, Kanagawa Hitachi, Ltd. Video Inside the Media Research Institute (72) Takashi Tsunoda, 292 Yoshida-cho, Totsuka-ku, Yokohama-shi, Kanagawa Hitachi, Ltd.Video Media Research Institute (72) Hiroshi Akai, 292 Yoshida-cho, Totsuka-ku, Yokohama, Kanagawa Factory AV Equipment Division

Claims (18)

[Claims] 1. A face shape screen having a shape of a face, an image pickup means for picking up an image of a person, a face image corresponding to a face is extracted from an image of the person imaged by the image pickup means, and the face image is extracted. The inclination, position, and size of
A human interface device comprising: image processing means for normalizing the shape of the facial shape screen; and projection means for projecting the facial image normalized by the image processing means onto the facial shape screen. 2. The human interface device according to claim 1, further comprising a mannequin provided below the face-shaped screen. 3. The human interface device according to claim 1, wherein the imaging unit constitutes the same unit, and a microphone for taking in sound from the outside while the image of the person is captured by the imaging unit, and the projection unit. And a speaker that outputs the sound captured by the microphone to the outside. 4. A face shape screen having a shape of a face, an image pickup means for picking up an image of a person, a face image corresponding to a face is extracted from an image of the person imaged by the image pickup means, and the face image is extracted. The inclination, position, and size of
Image processing means for normalizing the shape of the face shape screen, transmitting means for transmitting the face image normalized by the image processing means via a communication network, and face transmitted via the communication network A receiving means for receiving the image,
A human interface device, comprising: a projection means for projecting the facial image received by the receiving means onto the facial shape screen. 5. The human interface device according to claim 4, comprising the same unit as the image pickup means, and having a microphone for taking in sound from the outside while the image of the person is picked up by the image pickup means, The transmitting means transmits the voice captured by the microphone through the communication network together with the face image normalized by the image processing means, and the receiving means together with the face image transmitted through the communication network. A human interface device comprising: a speaker that receives a voice transmitted through a communication network, forms a unit same as the projecting unit, and outputs the voice received by the receiving unit to the outside. 6. A face shape screen having a shape of a face, an image pickup means for picking up an image of a person, a face image corresponding to a face is extracted from the image of the person imaged by the image pickup means, and the face image is extracted. The inclination, position, and size of
Image processing means for normalizing the shape of the facial shape screen, recording means for recording the facial image normalized by the image processing means in a storage medium, and face recorded in the storage medium by the recording means. A human interface device, comprising: a projection means for projecting an image on the facial shape screen. 7. The human interface device according to claim 6, comprising the same unit as said image pickup means, and having a microphone for taking in sound from the outside while the image of the person is picked up by said image pickup means, The recording means records the sound captured by the microphone in a storage medium together with the face image normalized by the image processing means, constitutes the same unit as the projecting means, and the face image is reproduced by the projecting means. A human interface device comprising a speaker for outputting the sound recorded in the storage medium by the recording means to the outside while being projected on the facial shape screen. 8. The human interface device according to any one of claims 1 to 7, wherein the image processing means, based on color information of the image of the person imaged by the image capturing means, determines the skin color of the image. The part is extracted as a face image, and positions corresponding to images of two or more predetermined face parts are extracted from the extracted face image, and the face image is straightened based on these positions. As described above, the inclination of the facial image is normalized, and the position of the facial image is adjusted so that any one position of the extracted two or more positions matches the position corresponding to the position on the facial shape screen. And the ratio of the facial image to the size of the facial shape screen is calculated from the distance between the extracted two or more positions,
A human interface device for normalizing the size of the facial image so that the size of the facial image matches the size of the facial shape screen based on the obtained ratio. 9. A facial shape screen having a facial shape, and projection means for projecting a facial image, which is a portion corresponding to a human face, supplied from the outside onto the facial shape screen. Human interface device. 10. The human interface device according to claim 1, wherein the face-shaped screen is an image of a human head made of a light-colored opaque material, and the projection means is a face of the image. A human interface device, which is a projector that projects the facial image from the front side of the portion. 11. The human interface device according to claim 1, wherein the face-shaped screen is an image of a front half of a head made of a dark-colored semitransparent material, and the projecting means includes A human interface device, which is a projector that projects the facial image from the rear side. 12. The human interface device according to claim 1, wherein the face-shaped screen is a plate made of a transparent material,
Further, the rear surface is a plate that is hollowed out in the shape of a face to be translucent, and the projection means is a projector that projects the facial image from the rear surface side of the plate. Human interface device. 13. The human interface device according to claim 1, wherein the face-shaped screen is an image of a head having a hollow interior made of a transparent material, and the projection means is provided from the inside of the image. A human interface device, which is a projector for projecting the facial image. 14. The human interface device according to claim 1, wherein the projecting means is a display for displaying the facial image, and the facial shape screen is a plurality of screens arranged vertically to the display. A human interface device comprising a group of optical fibers, each of which is composed of optical fibers and has a set of ends which are not in contact with the display, forming a facial shape. 15. The human interface device according to claim 1, wherein the facial shape screen is capable of deforming its shape, and represents a three-dimensional shape of a facial image normalized by the image processing means. A human interface device comprising: stereoscopic information extracting means for calculating stereoscopic information; and screen deforming means for deforming the shape of the stereoscopic screen based on the stereoscopic information calculated by the stereoscopic information extracting means. 16. The human interface device according to claim 15, wherein the face-shaped screen includes a screen film, a plurality of pins, and a solenoid that deforms the shape of the screen film by moving the pins back and forth. The human interface device, wherein the pin actuator is composed of a group of pin actuators assembled in a grid pattern, and the screen deforming means controls the solenoid to move the pins back and forth. 17. A three-dimensional screen having a deformable shape,
Based on the database storing the three-dimensional object data including the image of the three-dimensional object and the three-dimensional information representing the three-dimensional shape of the three-dimensional object, and the three-dimensional information of the three-dimensional object data stored in the database, And a projection means for projecting an image of the three-dimensional object data stored in the database onto the three-dimensional screen whose shape is transformed by the screen transformation means. Characteristic stereoscopic image projection device. 18. The three-dimensional image projection device according to claim 17, wherein the three-dimensional screen comprises a screen film, a plurality of pins, and a solenoid that moves the pins back and forth to deform the shape of the screen film. A stereoscopic image projection apparatus, wherein the pin actuator is composed of a group of pin actuators assembled in a lattice pattern, and the screen deforming means controls the solenoid to move the pin back and forth.