JP2012079330A - Guide apparatus and guide method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a guide apparatus and guide method in which, even when a language that a partner to be communicated speaks is not known, the language can be determined.SOLUTION: Guide information is displayed (S23), an expression of a target person to observe the guide information is monitored (S22), and the monitored expression of the target person is determined (S25). During display of the guide information, in accordance with the determined expression of the target person, contents of a message to the target person are determined, and the message is displayed for the target person (S28-S32).

Description

The present invention relates to a guide device, and more particularly, to a guide device and a guide method that can perform guidance by selecting an appropriate language even for a partner who does not know the language to be used.

In recent years, the Internet has become widespread and it has come to be easily and instantly connected with people all over the world. In this way, international communication is difficult because of the language barrier, despite the fact that it is an environment where communication is possible through the Internet. However, if it is an image rather than a sentence, it is easy to interact across language barriers. Moreover, since the spread of digital technology has made it easier to upload images on the Internet, there is a possibility of deepening exchanges by using these images.

In this sense, the possibility of using images by photographs is high, but it is still a language barrier to communicate and take pictures when foreigners visit Japan or when foreigners visit Japan. It is not easy. Conventionally, various proposals have been made for translations or the like for solving this language problem. However, even if a high-performance translator appears, the number of languages is large, and a corresponding language must be selected from a plurality of languages at the place of communication.

Therefore, various proposals have also been made regarding language selection. For example, Patent Document 1 discloses an image recording apparatus capable of recognizing a language at the time of recording or reproducing a moving image and adding a caption. In this image recording apparatus, when language recognition is automatically set, voice recognition is performed in the UI language (language used for setting on the menu screen and various setting screens).

JP 2006-30874 A

In the image recording apparatus disclosed in Patent Document 1, automatic speech recognition is limited to one language set as the UI language. If the language of the person you want to communicate with is unknown, language recognition cannot be performed. In addition, in the image recording apparatus disclosed in Patent Document 1, it is possible to manually select a language. However, this forces the user to operate the mode setting, which requires extra operations and uses a voice function or the like. Communication was difficult.

An object of the present invention is to provide a guide device and a guide method capable of changing a message in accordance with the facial expression of a partner with whom communication is performed.

  In order to achieve the above object, a guide device according to a first invention includes a guide information storage unit that stores a plurality of guide information, a display unit that displays the guide information read from the guide information storage unit, and An imaging unit that monitors the facial expression of the target person observing the guide information, a facial expression determination unit that determines the facial expression of the target person monitored by the imaging unit, and the facial expression determination unit that is determined while the guide information is displayed. A display control unit that determines a message for the target person according to the facial expression of the target person and displays the message on the target person.

In the guide device according to a second aspect of the present invention, in the first aspect, the message is determined by the color of the image obtained by the imaging unit and the feature of the face information determined by the facial expression determination unit.
In the guide device according to a third aspect of the present invention, in the first aspect, the reason why the display control unit determines the message according to the determination result by the facial expression determination unit will be described.

  In the first invention, the guide device according to the fourth invention displays different guide information prior to displaying the message content, and based on the determination result of the facial expression determination unit, the psychology of the target person is displayed. An estimation unit for estimation is provided, and communication is performed based on the estimation result.

  In the guide device according to a fifth aspect based on the fourth aspect, the estimation unit determines the language used by the target person based on the determination result by the facial expression determination unit when displaying guide information in a different language. Then, the display control unit displays the message in the language used estimated by the estimation unit.

The guide device according to a sixth aspect of the present invention is the above-described fourth aspect of the present invention, further comprising a microphone that captures voice, and the estimation unit estimates a language based on the tone or timing of the captured voice.
In the guide device according to a seventh aspect of the present invention, in the first aspect, the display unit displays visually or audibly.

  A guide device according to an eighth aspect of the invention estimates a language for the target person according to the detection result while displaying the language guide information and a detection unit that detects the facial expression and voice of the target person with respect to the language guide information. A language estimation unit; and a determination unit that determines a communication phrase in a language used estimated by the language estimation unit.

  A guide device according to a ninth aspect of the present invention displays guide information, monitors the facial expression of the target person observing the guide information, determines the monitored facial expression of the target person, and displays the guide information while displaying the guide information. A message for the target person is determined according to the determined facial expression of the target person.

  ADVANTAGE OF THE INVENTION According to this invention, the guide apparatus and guide method which can change a message according to the other party's facial expression which communicates can be provided.

1 is a block diagram of a camera and its surroundings according to an embodiment of the present invention. It is a figure which shows the usage method of the camera concerning one Embodiment of this invention, (a) is a scene which determines the use language of a to-be-photographed object (the other party), (b) is a call for photography after determination of a use language. It is a figure of the scene which has generate | occur | produced. It is a figure which shows the imaging | photography method of the camera which concerns on one Embodiment of this invention, (a) shows together imaging | photography mode and (b) shows model imaging | photography mode. It is a figure which shows the external appearance of the camera which concerns on one Embodiment of this invention, (a) is an external view which shows the display at the time of the through image at the time of imaging | photography, or a reproduction image, (b) is also related phrases. FIG. FIG. 4 is a diagram illustrating a relationship between a subject image that passes through a photographing lens 2 of a camera according to an embodiment of the present invention and an imaging surface of an image sensor 3. In one Embodiment of this invention, it is a figure which compares the smile of a to-be-photographed object and a troubled face, (a) shows a smile, (b) shows a troubled face. It is a flowchart for measuring the smile degree in the camera which concerns on one Embodiment of this invention. In one Embodiment of this invention, it is a figure which compares the face when a subject is depressing and when it denies, (a) is a front view of a face, (b) is a face when whispering, (c) Is the face when denying. It is a flowchart for measuring the affirmation degree in the camera which concerns on one Embodiment of this invention. In one Embodiment of this invention, it is a graph which shows an example of the smile degree and affirmation degree with respect to each language when estimating a language. It is a flowchart for showing the control operation of the camera in the camera concerning one embodiment of the present invention. It is a flowchart for showing the control operation of the camera in the camera concerning one embodiment of the present invention. 6 is a flowchart illustrating an image management operation in the server according to the embodiment of the present invention. It is sectional drawing which shows the modification of the imaging lens of the camera which concerns on one Embodiment of this invention, (a) is the combination of a compound eye lens and a prism, (b) is the combination of a compound eye lens and a prism, (c ) Is a modification using a convex reflecting mirror.

Hereinafter, a preferred embodiment will be described using a camera to which a guide device of the present invention is applied according to the drawings. In this embodiment, prior to shooting, the language is determined in accordance with the partner's facial expression when the partner selects a language such as “Do you want to use Japanese?” Or “Do you want to use English?”. Camera guidance can be performed using guidance in a predetermined language. This makes it possible to communicate with foreigners who do not understand the language through photos. Also, you can shoot together without compromising the other party, or add comments to the captured image in the language of the camera owner and the partner's language so that you can search for images taken by both I have to.

FIG. 1 is a block diagram of a digital camera and its surroundings according to an embodiment of the present invention. The camera 100 is a flat plate type (see FIG. 2), and a photographing lens 2 composed of a wide-angle lens such as a fisheye lens is provided on the flat plate to enable photographing over a wide range. This is because the photographer himself can photograph. A camera 100 used by a user includes a photographic lens 2, an AF (autofocus) control unit 2a, an aperture 2b, an aperture control unit 2c, an image sensor 3, and an analog front end (hereinafter referred to as AFE) unit 4. Yes. The taking lens 2 has a focus lens inside, and forms an image of the incident subject 20 on the image sensor 3.

The AF control unit 2a performs image processing in an image processing unit 5 to be described later, detects an in-focus position of the photographic lens 2, detects it by a so-called hill-climbing method, drives the focus lens, and moves it to the in-focus position. Note that the hill-climbing method is a method for detecting the in-focus position (focus position) from the contrast signal peak of the photographed image. In addition to the hill-climbing method, for example, a known focusing method such as a phase difference method or a triangulation method can be used. It may be replaced. When the in-focus position is reached by AF, the distance can be determined by detecting the photographing lens 2 at this time. In this case, when the photographing lens 2 is a zoom lens, the distance is determined in consideration of the zoom position and the like.

A diaphragm 2b that provides the effects of a shutter and a diaphragm is provided in or near the photographing lens 2. The diaphragm 2b is opened to a predetermined aperture at the time of photographing, and is closed when the exposure is finished to finish the exposure. The aperture control unit 2c drives the aperture 2b to set the aperture diameter. By changing the aperture, the depth of field of the lens changes, so you can change the expression by adjusting the blurring of the background, etc. Can do. The image pickup device 3 is a CCD or CMOS sensor having a large number of light receiving surfaces (pixels), and receives an image from the subject 20 via the photographing lens 2 and converts it into an image signal.

The analog front end (AFE) unit 4 includes analog-digital (AD) conversion means, and converts a signal from the image sensor 3 into a digital signal. The AFE unit 4 performs various processes on the image signal output from the image sensor 3. The AFE unit 4 is also provided with a function of reading several pixels of the image pickup device 3 together and reading them together. For example, when the signal level of each pixel is small, such as 4 pixels (2 × 2) and 9 pixels (3 × 3), several pixel signals can be added to improve the S / N. Sensitivity can also be increased.

Sensitivity can be increased by such an operation, and it is possible to devise photography such as increasing the shutter speed. When the shutter speed is high, it is possible to shoot as if the moving subject is stopped.

The AFE unit 4 has a function of selecting signals output from the image sensor 3 and can extract image data in a limited range from the light receiving range. In general, when pixel signals thinned out from the pixels of the image sensor 3 are extracted, high-speed reading is possible. Thus, the image signal for composition confirmation is processed at high speed by the image processing unit 5 and displayed on the display panel 8 via the display control unit 8a, thereby enabling framing.

The output of the AFE unit 4 is connected to the image processing unit 5. The image processing unit 5 corrects the color, gradation, and sharpness of the input signal. Further, it has an intensifying unit that amplifies the image signal obtained from the image sensor 3 to a predetermined level and sets the correct gray level and the correct gray level. This is a digital operation so that the digitized signal level becomes a predetermined level. In addition, the image processing unit 5 includes a resizing unit that processes a signal from the image sensor 3 to a size that can be displayed on the display panel 8 so that a through image (also referred to as a live image) can be displayed on the display unit in real time. . By this function, an image incident on the image sensor can be confirmed prior to shooting, and the timing and photo opportunity at the time of shooting can be determined while viewing this.

In addition, the image processing unit 5 has an image determination function for determining characteristics of an image input from the image sensor 3 by using a signal from the image processing unit 5. For example, a shape determination unit 5a that detects the shape of what is being imaged from the contour information obtained by processing the image information, and a face detection unit 5d that checks the characteristics of the image and detects the position of the main subject. Included in this system. This is for optimizing the shooting control by determining what kind of photo the user is going to take at the time of shooting.

The image processing unit 5 has a function of determining the contrast of the image described above, and controls autofocus in conjunction with the focusing unit of the photographic lens. The distance of the subject, the distance of the background, and the like can be determined based on the information on the lens position when the photographing lens 2 is focused.

Since the photographing lens 2 is a wide-angle lens such as a fish-eye lens, as shown in FIG. 5, a subject image is formed in a wide image range 51 indicated by a circle and can be photographed. However, the image in the peripheral recording / recording range 52 causes a large fan-shaped distortion as shown in the figure. Therefore, the distortion correction unit 5b in the image processing unit 5 corrects the image to a rectangular image 53 having no distortion as illustrated. In the present embodiment, not all the images that can be photographed by the photographing lens 2 are recorded, but a part of the images (images in the photographing recording range 52) is taken out and recorded in this way, and the other image parts are It is used during another operation. Also, using this imaging result, the facial expression and the like are read as will be described later. Note that a wide range may be photographed by using a plurality of lenses and prisms as shown in FIG.

In addition, the image processing unit 5 is connected to a facial expression determination unit 5c that determines facial expression from a facial image obtained by the image sensor 3. The facial expression determination unit 5c can determine the situation determination other than the shooting and the intention of the subject. That is, as will be described later, the camera 100 has a database unit 12 of a multilingual dictionary, from which a predetermined conversation phrase is read out and output as a voice by the voice generation unit 14 while changing the language, or displayed. It is possible to determine which language is the most common word by changing the facial expression when it is displayed sequentially on the part 8 and when it is viewed. This language selection is performed using the output of the facial expression determination unit 5c. In addition, language information can be used to appropriately organize images. That is, when the photographed person is a foreign person, it is difficult to search for images in Japanese, so that the images can be organized in the language used by the photographed person.

The camera 100 includes a compression unit 6, a recording unit 9a, a recording medium 9, a data transmission / reception unit 9c, a display panel 8, a display control unit 8a, a dictionary / camera display switching unit 8b, a backlight (BL) 8c, and a backlight (BL ) A control unit 8d and a multilingual dictionary unit 12 are provided. The compression unit 6 compresses the signal output from the image processing unit 5 at the time of shooting. The compression unit 6 includes MPEG (Moving Picture Experts Group) 4 and H.264. A compression unit for moving images composed of a compression core unit such as H.264 and JPEG (Joint
A still image compression unit such as a Photographic Experts Group) core unit is provided. The compression unit 6 also performs image expansion when reproducing an image recorded on the recording medium 9 on the display panel 8.

The recording unit 9 a records the compressed image signal on the recording medium 9. The recording medium 9 is a storage recording medium that can be attached to and detached from the camera. In recording the image signal, in addition to the compressed image data described above, accompanying data such as the shooting time is also recorded. The shooting time information is measured by a clock unit 1t in the MPU 1 described later, and the MPU 1 associates with the image.

The captured image data recorded on the recording medium 9 or the recording unit 9a is transmitted to another device such as an image via the Internet 32 by the wireless data transmitting / receiving unit 9c using a technology such as an IC card such as RFID or a wireless LAN. Can be transferred to another device such as the server 34 that manages the server. The image transmitted to the server 34 can be immediately shown to the third party 31, and communication can be promoted while viewing the image. The Internet 32 is connected to a translation service server 33 that performs a translation service.

The display panel 8 is composed of, for example, a liquid crystal or an organic EL, and a backlight 8c such as a white LED is disposed on the back surface of the display panel 8. The backlight 8c is connected to a backlight (BL) control unit 8d. Connected and controlled by this. By irradiating the display panel 8 with the light from the backlight 8c, the visibility of the display panel 8 can be improved. When the appearance is poor, the BL control unit 8d controls the brightness by the user's operation. Control to increase.

Also, a multilingual dictionary database unit 12 is connected to the MPU 1 to be described later, and this multilingual dictionary database 12 stores predetermined conversation phrases and the like that are often used during travel in correspondence with multilinguals. Yes. These multilingual conversation phrases and the like are displayed on the display panel 8. Therefore, a dictionary / camera display switching unit 8b is provided in the display control unit 8a. The dictionary / camera display switching unit 8b switches between dictionary display and camera display on the liquid crystal panel 8 in accordance with various setting states, as will be described later.

Therefore, the display panel 8 displays a subject image for observing the subject at the time of photographing, displays a stretched photographed image at the time of reproduction, and displays a conversation phrase or the like when the dictionary is displayed. The display panel 8 displays a list of images that have been taken and transmitted, and from there, an image to be uploaded to a blog service or the like operated by the server 34, or a layout for uploading and displaying the image on a blog can be selected. You may do it.

The camera 100 is provided with an auxiliary light emitting unit 11, an MPU (Micro Processing Unit) 1, a release switch 1a (1a may also mean a release button), a mode switch 1b, and another switch 1c. Yes.

The auxiliary light emitting unit 11 includes a white LED and a Xe discharge light emitting tube, and the amount of light can be controlled by the amount of current. Depending on the situation, the subject 20 is irradiated with light to prevent lack of brightness and uneven brightness. Further, as will be described later, this auxiliary light also illuminates other than the subject and is used for determination at the time of mode switching. Further, the MPU 1 is provided with a sound generation unit 14 and connected to a speaker for generating sound according to the conversation phrase read from the multilingual dictionary database 12.

The MPU 1 is a control unit that controls the entire camera. A release switch 1a, a mode switch 1b, another switch 1c, and a ROM (not shown) storing a program and the like are connected to the MPU 1. As described above, the MPU 1 is provided with the clock unit 1t for detecting the date and time, and detects the shooting date and time of the photograph and associates it with the captured image. Note that the switches 1b and 1c are generically displayed switches, and are actually composed of a large number of switch groups.

Each switch 1a-1c detects a user's operation, and notifies the result to MPU1 which is a calculation control means consisting of a microcontroller. The MPU 1 switches the operation according to the operation of these switches. According to the operation of these switches, shooting / playback mode switching, shooting mode switching, image transmission setting, keyword selection, conversation phrase selection, and the like can be performed. Further, the MPU 1 functions as a shooting control unit that controls the AF control unit 2a, the aperture control unit 2c, and the like during shooting. Also, various functions such as an image reproduction function and a dictionary function are switched.

Next, the external structure and usage method of the camera 100 will be described with reference to FIGS. First, as shown in FIGS. 3 and 4, the appearance of the camera 100 is provided with a photographing lens 2 composed of a fisheye lens, a display panel 8 for display, and a release button 1a on the upper surface of the main body 10. ing. Further, the main body unit 10 is provided with a sound generation unit 14 including a speaker or the like for reproducing a language name, a conversation phrase, or the like by voice synthesis. An auxiliary light emitting unit 11 is provided on the side of the main body 10.

As shown in FIGS. 3A and 3B, the photographing lens 2 is attached to the main body 10 of the camera 100 so that the optical axis is substantially perpendicular to the surface of the display panel 8. With respect to the photographing lens 2, an arrow 16 indicates a photographing direction when the user 21 and the subject 20 (the other party) who communicates with each other by the guide device according to the present embodiment are taken together. An arrow 17 indicates a shooting direction when shooting using the subject (partner) 20 as a model. Arrows 23 and 24 indicate directions of light irradiated to the user 21 and the counterpart 20 by the backlight 8c of the display panel 8. An arrow 25 (FIG. 3B) indicates an irradiation direction by the auxiliary light emitting unit 11.

Since the photographing lens 2 is composed of a wide-angle lens such as a fish-eye lens, an image corresponding to the video range 51 shown in FIG. The image sensor 3 can photoelectrically convert the image in the imaging range 54 with respect to this video range. The shooting / recording range 52 is optically distorted as described above, but is corrected to the rectangle 53 by the distortion correction unit in the image processing unit 5. Further, when the shooting direction of the subject 20 as a model is the arrow 25 with respect to the shooting lens 2, an image is formed in the model shooting range 56. An auxiliary light range 55 by the auxiliary light emitting unit 11 and the backlight 8 c of the display panel 8 is an area that covers the photographing recording range 52 and the model photographing range 56.

In the present embodiment, an image in the video range 51 is obtained by the photographing lens 2, but only a part of the image is used for photographing. That is, when the user 21 and the subject (the other party) 20 photograph together, the image in the photographing recording range 52 is used. Further, when photographing the subject (the other party) 20, an image in the model photographing range 56 is used. Accordingly, the irradiation range of the auxiliary light only needs to be able to irradiate the shooting recording range 52 + α or the model shooting range 56 + α. As will be described later, when the user 21 and the subject (the other party) 20 exist near the camera 100 as shown in FIG. 2, the light from the backlight 8c of the display panel 8 may be used.

A method of switching between the dictionary mode using the conversation phrase and the camera mode in the camera 100 having such a guide device will be described. FIG. 2 shows that the user 21 is communicating with the subject 20 using the camera 100 prior to shooting. By the camera operation of the user 21, the sound generation unit 14 sequentially reproduces the sound of the content “Is this word OK” in the representative languages of the world. As described above, the photographing lens 2 is a wide-angle lens such as a fisheye lens, and can capture an image in the monitor range 22. Then, the facial expression of the subject person at this time is monitored based on the image signal from the imaging recording range 52 of the imaging element 3 through the photographing lens 2. The same words may be sequentially displayed on the display panel 8 instead of the voice from the voice generation unit 14 or together with the voice.

When the facial expression is monitored in this way, the facial expression of the subject 20 changes during playback of a language in which the subject (partner) 20 can be understood. This change in facial expression is recognized, and the language at the time of change is determined to be the language used by the subject 20. When the language used is known, it is possible to make a question as to whether or not photographing can be performed using the language used, as shown in FIG. As a result, permission for photographing can be obtained.

In communication, not only a change in facial expression but also a change in gesture may be detected. In addition, a language may be provided to estimate the language used by the subject 20 by supplementarily using the response tone and timing of the subject 20.

FIG. 3A shows a case where the image is taken together from the state of FIG. The camera 100 is displayed as a cross-sectional view seen from the side direction, and the arrangement of each function is shown in an easy-to-understand manner. An imaging result is displayed on the display panel 8 (see FIG. 4A), and while observing the display panel 8, a facial expression is determined and shooting is performed (referred to as “shooting together”). If the composition and expression are good, the release button 1a is operated to take a picture.

When the surroundings are dark, the subject 20 may be illuminated using the backlight 8c of the display panel 8. When illumination is performed, the liquid crystal of the display panel 8 may be brought into a transmissive state at the timing of photographing. Further, in the state of FIG. 2, when the subject 20 is photographed alone without photographing together at the same position, a wide-angle lens of the photographing lens 2 is used as shown in FIG. Images from different shooting directions 17 may be displayed on the display panel 8 and shot. This is referred to as “model photography”.

In the image obtained from the photographing lens 2 having such a wide-angle lens, the recording range and the model photographing range are the photographing recording range 52 and the model photographing range 56, respectively, as described above. Since the wide-angle lens is used, the acquired image is considerably distorted. Therefore, the distortion correction unit 5b of the image processing unit 5 corrects the upper portion of the image to be reduced so that the rectangular image 53 is obtained, and the image is displayed after being converted into a square image like a normal photograph. In addition, recording is performed on the recording medium 9 or the like.

In the case of recording together, since only the fan-shaped portion (shooting and recording range 52) is cut out and used, the imaging device 3 only needs to cover only the dotted quadrilateral imaging range 54. Further, the auxiliary light range 55 only needs to cover the area indicated by the dotted ellipse, and does not need to cover the entire wide-angle range.

Images taken in this way are arranged with keywords. When assigning keywords, as shown in FIG. 4B, the display panel 8 is translated into the keywords determined by the language of the photographer (user) 21 and the language of the subject 20 as indicated by reference numeral 23. (When using translation, the determination result of the word used by the subject is used). As a result, it becomes a keyword that is easy to search for the subject, and after taking the picture, even after parting, it is possible to search each person's daily words on the Internet, and this image can be easily searched.

In FIG. 2, it is determined which language is understood by the change in the expression of the subject 20. A method for determining the expression change will be described with reference to FIG. FIG. 6 is a comparison of a smile (a) that a person represents when the language is understood, and a troubled face (b) when the language cannot be understood. I understand that.

In other words, when the eyes are compared, it can be seen that if there are many white eyes above the line 61 connecting the centers of the eyes of both eyes, it is close to a smile. The mouth can also be determined based on the same concept. That is, if the area of the lower lip is larger than the line 62 connecting both ends of the mouth, the degree of smile increases.

A method of obtaining the smile level R using the determination method shown in FIG. 6 will be described based on the flowchart shown in FIG. First, the eyes and mouth are detected (S151). In this method, a face is detected by the face detection unit 5d based on the output of the image sensor 3, and the eye and mouth are extracted from the detected face by pattern recognition.

Subsequently, in steps S152 to S154, the smile level is detected from the area of the white of the eye. First, the area of the white eye above the line 61 connecting the eyes of both eyes is measured, and this value is set as EA (S152). Similarly, the area of the white eye below the line 61 is measured, and this value is set to EB (S153). Next, the difference between EA and EB obtained in this way is normalized by the sum of these, that is, (EA−EB) / (EA + EB) is obtained to obtain a numerical value RE (S154). It is determined that the larger the numerical value RE, the higher the smile degree.

Next, in steps S155 to S157, the smile level is detected from the area of the lips. First, the area of the lips above the line 62 connecting both ends of the mouth is measured, and this value is set as LA (S155). Similarly, the area of the lips below the line 62 is measured, and this value is set as LB (S156). The difference between LA and LB thus obtained is normalized by the sum of these, that is, (LB−LA) / (LA + LB) is obtained to obtain a numerical value RL (S157). It is determined that the larger the numerical value RL, the higher the smile degree.

Subsequently, the smile level R is obtained by adding the values RE and RL obtained in steps S154 and S157 (S158). The larger the value R, the closer to smile. Furthermore, in order to improve the accuracy of determination of the smile level, the state near the arrow 64 in FIG. 6A is determined (S159). That is, if there is a shadow at the edge of the mouth and teeth are visible, the probability of smile is high, and this is determined. As a result of the determination, if teeth are visible and there is a shadow at the mouth end, one point is added to smile level R (S160).

However, if there is a wrinkle between the eyebrows, it cannot be said that it is a smile, so determination is made in step S161. That is, it is determined whether or not there is a wrinkle between the eyes near the arrow 65 in FIG. 6B (S161). If the eyelid is detected between the eyebrows as a result of the determination, one point is deducted from the smile level R (S162). In this way, the numerical value (R) can be obtained such that the closer to a smile, the higher the score.
In the present embodiment, the degree of smile is determined based on whether there are many white eyes above the line connecting the centers of the eyes of both eyes. However, the present invention is not limited to this. Good. That is, in FIG. 6, the position ET at the highest point of the heel with respect to the width EH of the heel is obtained, and the smile degree can also be obtained by the ratio ET / EH. When ET / EH increases, it becomes close to a smile, and when this ratio decreases and the inclination of the upper eye increases, it tends to move away from the smile.

In this embodiment, since the facial expression is digitized, the face is constantly monitored, the numerical value of the facial expression is recorded, and if the facial expression changes from an incomprehensible word to an understandable word, the smile level R Since the value of or increases greatly, this is determined to determine the language for daily use.

In the present embodiment, smile level R is measured for detection of a change in facial expression. Affirmation level A, which is a modified example thereof, will be described with reference to FIGS. 8 and 9. The affirmation degree A can be used instead of the smile degree R or together with the smile degree R. FIG. 8 shows the situation when the face is facing the front (FIG. 8A), when facing downward (FIG. 8B), and when facing side (FIG. 8C). . That is, when he / she hears an understandable word, he / she speaks (affirms), and the face moves downward by ΔY from the center of the eye as shown in FIG. 8B. On the other hand, when he / she hears a word that cannot be understood, his / her face is shaken (No), and as shown in FIG. 8C, the face turns sideways by ΔX from the center of the eye. In this way, it is possible to determine whether it is positive or negative by looking at the movement of the center of the eye.

FIG. 9 is a flowchart for determining whether the subject (the other party) 20 has responded by detecting whether the center of the face has moved sideways (negative direction) or vertically (positive direction) during a predetermined period. When the positive movement is more conspicuous in terms of the number of times the face has moved sideways or the frequency and the number of vertical movements, or frequency, the value of the positive degree A is set to be large.

A method of obtaining the affirmation degree A using the determination method shown in FIG. 8 will be described based on the flowchart shown in FIG. First, a face image is detected from the output of the image sensor 3, and the process is started at a predetermined timing (S251). Subsequently, the center coordinates 66 of the eye are measured (S252). Thereafter, it waits for a predetermined time to elapse (S253). This predetermined time is a sampling time for detecting a whispering face or a lateral change.

When the predetermined time has elapsed, the center coordinate 66 of the eye is measured again (S254). Since the center coordinate 66 of the eye is measured at two timings with a predetermined time in between, a change in the Y-direction of the eye center coordinates between the two timings is obtained and set as ΔY (S255). Next, a change in the X direction of the eye center coordinates is obtained and set as ΔX (S256).

Next, it is determined whether ΔY is smaller than a predetermined value and ΔX is larger than a predetermined value (S257). As a result of the determination, if this determination condition is satisfied, the face is shaken sideways, so that one point is deducted from the affirmation A to reduce the value of the affirmation A (S258). Subsequently, it is determined whether ΔY is larger than a predetermined value and ΔX is smaller than a predetermined value (S259). As a result of the determination, if this determination condition is satisfied, it is in a state of scolding, so that one point is added to the affirmation A to increase the value of the affirmation A (S260).

When the above determination is finished, it is next determined whether the detection timing is over (S260). That is, since the measurement of the affirmation degree A is repeated a plurality of times at a predetermined time interval, it is determined whether or not this number has been reached in step S260. If not, the process returns to step S253 and is repeated.

As shown in FIG. 10, the smile level R and the positive level A are measured by monitoring the movement of the face while switching the language. In the example shown in FIG. 10, since the smile level R and affirmation level A are the highest when the language is C, it can be determined that the language used by the subject 20 is the language C.

Next, control of the camera 100 will be described with reference to FIGS. 11 and 12. Prior to shooting, switching between language selection mode (electronic dictionary mode) and camera mode can be performed, and the setting mode is determined (S1). If the language selection mode is set as a result of the determination, the language selection is performed while communicating as described in FIG.

When the language selection mode is entered, first, it is determined whether or not the subject 20 is permitted to shoot (S21). This photographing permission is permission for the above-described model photographing or simultaneous photographing (see FIG. 3). Since the switch to the setting mode for obtaining the photographing permission is performed by operating the switches of the camera 100, in this step, it is determined whether or not the setting is for photographing permission.

If the result of determination in step S21 is that permission to shoot is obtained, facial expression monitoring is then started (S22), and multilingual questions are sequentially reproduced while changing the language (S23). The question is asked by voice reproduction by the speaker of the voice generator 14 or by displaying characters on the display panel 8. At this time, the flow for obtaining the smile degree R shown in FIG. 7 is performed. Instead of this or together, a flow for obtaining the affirmation degree A shown in FIG. 9 may be performed.

Next, it is determined whether or not playback has been completed for all languages (S24). If the result of determination is that it has not ended, processing returns to step S23, and playback is performed in the next language. On the other hand, if the result of the determination is end, the language is determined from the facial expression (S25). As shown in FIG. 13, the smile level R and the affirmative level A are compared for each language, and the language with the highest smile level R and the positive level A is estimated as the language used by the subject 20.

Next, the estimated corresponding language is confirmed (S26). The confirmation is performed by voice-displaying the confirmation in the corresponding language from the voice generation unit 14 and displaying a confirmation message of the corresponding language on the display panel 8. Subsequently, it is determined whether or not the voice display or visual display of the corresponding language is OK (S27). If the result of determination is NG, processing returns to step S23, and the above steps are repeated until an appropriate corresponding language is found.

If the determination in step S27 is OK, next, whether or not to shoot is confirmed or requested, this is changed depending on the facial expression of the subject 20 at that time. First, it is determined whether or not the facial expression of the person who is the subject 20 at that time is a smile (S28). If the smile level R is high as a result of the determination, a model photographing request as shown in FIG. 3B is made in the corresponding language confirmed in step S26 (S31). On the other hand, if the smile level R is low as a result of the determination, a request is made for photographing together as shown in FIG. 3A in the corresponding language (S32). Note that the request in steps S31 and S32 may be either voice or text display.

The reason why the correspondence is changed by the smile degree R is that it is easily rejected psychologically in the case of model shooting because it shoots only one subject, whereas the shooting together is as shown in FIG. This is because the user 21 and the subject 20 are likely to be accepted because they are in the same posture.

In addition to the switching in steps S28 to S32, switching may be performed by explaining the reason why the user wants to shoot if there is a harsh expression, although there is no explanation if the expression is a smile. The reason may be a general one such as “as a memorial to come here” or “because it will be sent later”. Alternatively, the blueness of the sky may be determined, and the display of the request may be changed in consideration of the image, such as “because the weather is good”.

When the photographing request in step S31 or S32 is completed, association phrases are then candidateized (S45). This is because, when a photographed image is retrieved at a later date, if a phrase is attached in association with the image, it is easy to retrieve, so a phrase candidate for this purpose is extracted and stored. When this step ends, the process returns to the above-described step S1.

If the result of determination in step S <b> 21 is that it is not the case that photographing permission is obtained, next, a phrase is selected (S <b> 41). The camera 100 can also be used as a communication means, and phrases necessary for the user 21 to communicate with the subject 20 are stored in the multilingual dictionary database 12 in advance. Therefore, the user 21 selects a phrase stored in the database 12 in the language used by the user 21.

Next, it is determined whether or not the corresponding language information exists (S42). Before executing this step, if step S21 to step S27 are executed and the language used by the subject (the other party) 20 is specified, there is information on the corresponding language, and on the other hand, such language determination is performed. If not, there is no information on the corresponding language.

If the result of determination in step S42 is that the corresponding language information is present, the phrase selected in step S41 is translated into the corresponding language (S43). The translation result may be displayed by voice from the speaker of the voice generation unit 14 or may be visually displayed as character information on the display panel 8. On the other hand, if there is no corresponding language information as a result of the determination, a list of national languages is displayed (S44). That is, a list of languages stored in the multilingual dictionary database 12 is displayed, and a corresponding language is selected (S44). If the corresponding language is selected, the phrase selected in step S41 is displayed by voice or vision.

In this way, communication between the user 21 and the subject (partner) 20 is possible. If the other person's answer is a question that can be answered with Yes or No, a certain degree of communication is possible. Here, determination of Yes and No may also be performed with an image. Further, a microphone may be connected to enable translation by voice recognition.

When the processes of steps S43 and S44 are finished, association phrases are then candidateized as described above (S45). Since phrases and language information at the time of communication can be used effectively when searching for a photographed image, they are recorded as related phrase candidates. Based on this result, phrases may be associated in step S17 described later.

If the result of determination in step S1 is not language selection mode, shooting mode is selected (S2). In the selection of the shooting mode, either the model shooting described above or the simultaneous shooting is selected. In the model shooting, an image of the fan-shaped portion (model shooting range 56) in the lower range in FIG. 5 is used, and in the simultaneous shooting, the upper fan-shaped portion (both shooting recording range 52) is used. Subsequently, a voice call at the time of shooting is performed from the sound generator 14 (S3). This is performed when the language used by the subject 20 is known. Depending on the shout at the time of shooting, the subject smiles, such as smiling, so the shooting may be performed using this timing.

Next, it is determined whether or not shooting is performed (S4). The shooting operation is started in response to the operation of the release button 1a. If the result of determination is that the image has been shot, a shooting operation is performed (S5). In the photographing operation, the image sensor 3 photoelectrically converts a subject image based on subject light transmitted through the photographing lens 2 and performs image processing such as distortion correction on the image data in the image range selected by the AFE 4. Subsequently, the related information is associated with the image (S6). At the time of image recording, related information such as clock information and set keyword information is associated with the image. Further, a candidate for association is selected in step S45 based on an expression or the like, and language information such as a phrase selected in step S17 described later is associated.

Next, it is determined whether or not the information to be associated is language information (S7). As a result of the determination, if it is not language information, it skips and proceeds to the next step S9, and if it is language information, the language is translated (S8). That is, the translation is performed so that the association information is attached even in the language used by the subject 20. Subsequently, image data is recorded (S9). Here, the image data subjected to the image processing is compressed by the compression unit 6 and the related information is attached, and after recording on the recording medium 9, the process returns.

If the result of determination in step S4 is not shooting, it is next determined whether or not playback mode is in effect (S11). As a result of the determination, if the playback mode is selected, the image data recorded on the recording medium 9 or the like is displayed on the display panel 8. Subsequently, the process proceeds to whether or not the reproduced image is to be uploaded to the external server 34 via the Internet. First, it is determined whether or not to perform communication (S13). This is a determination as to whether or not an operation for uploading the selected reproduced image to the external server 34 has been performed.

If communication is performed as a result of the determination in step S13, a recording area such as the server 34 to upload the selected image is designated (S14). Then, the selected image is transmitted to the address of the designated recording area (S15). If the result of determination in step S13 is not communication, it is determined whether or not to perform association (S16). As a result of determination, when the association is performed, the selected phrase is associated (S17). Reproduction display is performed in step S12. In this step, the selected phrase is associated with the reproduction image.

This image-related information may determine the phrase 23 (FIG. 4B) while watching the phrase list as a keyword with the user 21 and the subject 20 as shown in FIG. . The candidates can be translated and displayed sequentially, so you can select them while watching this. If it is difficult to see on the display panel 8, a phrase obtained by translating the phrase or keyword may be uttered by the speaker of the sound generator 14. When the sound is generated sequentially, the facial expression of the subject is monitored, and the smile level R shown in FIG. 7 is measured to determine the facial expression “this is good” and associate it with the image. Of course, it may be selected by operating a switch. When the association of the selected phrase in step S17 ends, the process returns.

Next, the operation of the server 34 that records and manages the image of the camera 100 according to the present embodiment will be described using the flowchart shown in FIG. As described above, the camera 100 transmits an image in step S15. First, the server 34 determines whether content such as an image is being received via the Internet, that is, whether or not the content is to be recorded (S101). If the content is recorded as a result of the determination, recording is performed in step S102. At the time of recording, related information (articles) such as keywords and phrases attached to the image is also recorded.

Subsequently, it is determined whether or not there is translation information in the association information or the like (S103). This is because not only the language of the photographer (user) 21 country but also the language of the subject (partner) 20 country is attached to the image as a tag, and the presence / absence of translation information is used so that it can be used later in the search. Judgment. If the translation information exists as a result of the determination, the translation information is tagged (S104). By tagging the translation information in this way, the subject (the other party) 20 can search for his / her own photograph on the net without knowing the national language of the photographer (user) 21.

If the result of determination in step S101 is not content recording, it is determined whether or not to search (S111). As a result of the determination, in the case of a search, an image having a tag corresponding to the search word typed by the searcher is searched (S112), and the searched image is displayed (S113). When the selected image is displayed, next, related articles related to this image are displayed (S114). Note that the related articles are recorded together with the content recording in step S101 as described above. Subsequently, it is determined whether or not there is a translation request for this related article (S115). If there is no translation request, the process returns as it is. On the other hand, if there is a translation request, the article is translated and displayed in cooperation with the translation service server 33 on the Internet 32 that provides the translation service (S116).

If the result of determination in step S111 is not retrieval, it is next determined whether or not there is image access (S121). This is a determination as to whether a personal computer or the like connected to the Internet 32 has accessed the server 34 and has accessed an image recorded on the server 34. As a result of the determination, if there is an access, an image corresponding to the address is displayed (S113), and then the processing proceeds to the above-described step S114 and thereafter, related articles are displayed, and the process returns.

Next, a modified example of the photographing lens 2 configured by an optical system such as a wide-angle lens such as a fisheye lens in the present embodiment will be described with reference to FIG. In the present embodiment, as described above, the entire image formed by the fisheye lens is not used. Therefore, as shown in FIGS. 14A and 14B, using the compound eye lens 66 and the prism 67, the two parts shown in FIG. The range may be covered. Further, a convex reflecting mirror 69 may be arranged as shown in FIG.

As described above, in the camera according to the present embodiment, the language used for understanding and using the subject is determined by monitoring the facial expression of the subject. Therefore, even if the language that the subject can understand is not known, the language used can be estimated, and the language can be specified and communicated between the user and the subject.

In the present embodiment, if the language used can be specified, communication can be achieved using a phrase or the like in that language, and the phrase or the like is recorded in association with an image. Useful when you want. In particular, when both the photographer's language and the subject's language are recorded and uploaded to a server or the like, both can easily search and search for images.

Further, in the present embodiment, the shooting mode (model shooting mode and shooting mode together) is changed according to the facial expression of the subject. If the language used is different, it may be difficult to communicate with words alone, but in this embodiment, the shooting mode can be changed according to the state of the subject.

Furthermore, in the present embodiment, a wide-angle lens such as a fisheye lens is used as a photographing lens, and photographing is performed in the same state as model photographing, but images are output according to each video range from the output of the image sensor. Is cut out. For this reason, the camera can be photographed as it is, and even in a state where it is difficult to determine the camera angle due to language barriers, photographing according to the situation can be performed.

In the description of the present invention, the example applied to the camera has been described. However, the technique for specifying the language by monitoring the facial expression is not limited to the camera, and may be applied to various guide devices such as an electronic dictionary and an electronic interpreting device. be able to. In addition, techniques such as recording related images with recorded information can be applied to various cameras such as compact cameras and single lens reflex cameras, as well as to built-in cameras such as mobile phones and PDAs. Can do.

DESCRIPTION OF SYMBOLS 1 ... MPU, 1a ... Release switch, 1b ... Mode changeover switch, 1c ... Other switch, 1t ... Clock part, 2 ... Shooting lens, 2a ... AF (automatic Focus) control unit, 2b ... aperture, 2c ... aperture control unit, 3 ... imaging device, 4 ... analog front end (AFE) unit, 5 ... image processing unit, 5a ... Shape determination unit, 5b ... distortion correction unit, 5c ... facial expression determination unit, 5d ... face detection unit, 6 ... compression unit, 8 ... display panel, 8a ... display control unit, 8b: dictionary / camera display switching unit, 8c: backlight (BL), 8d: backlight (BL) control unit, 9: recording medium, 9a: recording unit, 9c,. Data transmission / reception unit, 10 ... main body, 11 ... auxiliary light emitting unit, 12 ... multilingual Dictionary database, 14 ... sound generation unit, 16 ... arrow (shooting direction), 17 ... arrow (shooting direction), 20 ... subject (partner), 21 ... user (photographer), 22 ... monitor range, 23 ... phrase, 31 ... other users, 32 ... internet, 33 ... translation service server, 34 ... server, 51 ... video range (from fisheye lens) Entering video range), 52... Shooting recording range, 53... Rectangular image (distortion corrected), 54... Imaging range (imaging device), 55. Range: 61 ... Line (line connecting pupils), 64 ... Arrow (mouth) 65 ... Arrow (between eyebrows), 66 ... Eye center, 67 ... Compocular lens, 68 ... Prism, 69 ... convex reflector, 100 ... camera

Claims (9)

A guide information storage unit for storing a plurality of guide information;
A display unit for displaying the guide information read from the guide information storage unit;
An imaging unit for monitoring the facial expression of the target person observing the guide information;
A facial expression determination unit that determines the facial expression of the target person monitored by the imaging unit;
A display control unit for determining a message for the target person according to the facial expression of the target person determined by the facial expression determination unit while displaying the guide information, and displaying the message on the target person;
A guide device comprising:   The guide device according to claim 1, wherein the message is determined by a color of an image obtained by the imaging unit or a feature of face information determined by the facial expression determination unit. The display control unit explains the reason for determining the message according to the determination result by the facial expression determination unit.
The guide device according to claim 1.   Prior to displaying the message, different guide information is displayed and an estimation unit for estimating the psychology of the target person based on the determination result of the facial expression determination unit is provided, and communication is performed based on the estimated result. The guide device according to claim 1. The estimation unit estimates a language used by the target person based on a determination result by the facial expression determination unit when displaying guide information in a different language,
The display control unit displays the message in the language used estimated by the estimation unit.
The guide device according to claim 4. In addition, it has a microphone to capture audio,
The estimation unit estimates a language based on the tone or timing of the captured voice,
The guide device according to claim 4.   The guide device according to claim 1, wherein the display unit performs display visually or audibly. A detection unit for detecting the facial expression and voice of the target person with respect to the language guide information;
A language estimation unit that estimates a language for the target person according to the detection result during the display of the language guide information;
A determination unit for determining a communication phrase in the language used estimated by the language estimation unit;
A guide device comprising: Display guide information,
Monitor the facial expression of the target person observing the guide information,
Determine the monitored facial expression of the target person,
Determining a message for the target person according to the determined facial expression of the target person during the display of the guide information;
A guide method characterized by that.