JP6662411B2 - Mobile information terminals and programs - Google Patents

Description

  The present invention relates to a portable information terminal and a program.

  Mobile terminals include not only communication means used for communication with base stations, but also wireless LANs (Local Area Network), Bluetooth (registered trademark), infrared rays, NFC (Near Field Communication), etc., which can directly communicate with short-range terminals. Many communication means are built-in. Exchange of contacts between portable terminals using such short-range communication is widely performed.

  For example, according to the invention disclosed in Patent Document 1, it is possible to wirelessly exchange contact information with a portable terminal within a communication range. However, when there are a plurality of mobile terminals within the communication range, there is a problem that it is not clear which mobile terminal to exchange contacts with without checking the user of the mobile terminal.

Table 2009-536484

  When there are a plurality of terminals that can communicate with each other, it is impossible to know even if there is a person who intends to communicate with the terminal.

A portable information terminal according to a first aspect of the present invention includes a photographing unit, a line-of-sight detecting unit that detects a line of sight in a specific direction based on an image photographed by the photographing unit, and the line-of-sight detecting unit detects the line of sight. Then, an informing unit for informing, a feature amount calculating unit for calculating a feature of the face photographed by the photographing unit, and a storage unit for storing the feature of the face calculated by the feature amount calculating unit, wherein the informing unit is The gaze detection unit detects a gaze in the specific direction based on an image captured by the imaging unit, and when the person having the gaze is similar to the feature stored in the storage unit, the use of the portable information terminal. Notify others.
According to a second aspect of the present invention, there is provided a program that includes: a shooting procedure for shooting; a gaze detection procedure for detecting a gaze in a specific direction based on an image captured in the shooting procedure; and a gaze detection procedure for detecting the gaze in the gaze detection procedure. Then, there is provided an informing procedure for informing, a feature amount calculating procedure for calculating a feature of the face photographed in the photographing procedure, and a storage procedure for storing the face feature calculated in the feature amount calculating procedure, wherein the informing procedure is The gaze detection procedure detects a gaze in the specific direction based on an image captured in the imaging procedure, and when the person having the gaze is similar to the feature stored in the storage procedure, the use of the portable information terminal Including a procedure for notifying the user, causes the computer to execute each procedure .

  ADVANTAGE OF THE INVENTION According to this invention, the person who turns his / her eyes can be easily determined.

Block diagram of a mobile terminal according to the first embodiment Diagram showing the usage status of the mobile terminal and an operation example of the video display unit (a: usage status, b: normal mode, c: marker mode) Flowchart showing main operations in the first embodiment Flow chart showing details of the process shown in FIG. FIG. 7 is a diagram illustrating an operation until two mobile terminals start communication in the first embodiment. The figure which shows the operation example of the video display part in the modification 5. The figure which shows the example of the interface display in the modification 6. Block diagram of mobile terminal in modification 6 Flowchart showing main operations in the second embodiment 11 is a flowchart illustrating a learning operation according to the third embodiment. Flow chart showing main operations in the third embodiment Block diagram of mobile terminal in modification 9 Block diagram of a system according to a fourth embodiment Image of search for mobile terminal in fourth embodiment 11 is a flowchart illustrating the operation of the mobile terminal according to the fourth embodiment. 17 is a flowchart illustrating the operation of the matching server according to the fourth embodiment.

(First Embodiment)
Hereinafter, a first embodiment of the present invention will be described with reference to FIGS.
First, the configuration of the first exemplary embodiment of the present invention will be described.
FIG. 1 is a block diagram illustrating a configuration of the mobile terminal 100. The mobile terminal 100 is a mobile terminal that can be worn on the body, and is fixed to the head like glasses, for example.

  The image capturing unit 101 includes a camera (not shown), and the control unit 114 captures an image of a subject around a person (hereinafter, referred to as a user) wearing the portable terminal 100. Since the camera is near the eyes, the line of sight to the center of the image captured by the camera can be regarded as the line of sight to the eyes of the person wearing the portable terminal 100. It is desirable that the camera be equipped with a lens capable of shooting a wide angle, or be equipped with a plurality of cameras.

  The video display unit 102 includes a small display (not shown) and the like, and has two operation modes. In the normal mode, the real-time video captured by the video capturing unit 101 is displayed as it is. In the marker mode, the real-time video captured by the video capturing unit 101 is displayed with a marker. The display position of this marker is designated by the control unit 114. Information input by the user from the input unit 106 is also displayed on the video display unit 102. When the control unit 114 specifies the display position and outputs the data to the video display unit 102, the data is displayed at the specified position on the video display unit 102.

The mobile terminal 100 according to the embodiment of the present invention is mainly used when an unspecified large number is in a short distance (about several meters) for a short time, such as when waiting for a signal or in a train. When the eyes are aligned, the users can exchange information with each other wirelessly without directly speaking or making physical contact.
To achieve this, directional communication is used. The reason is as follows. It is easy to simultaneously transmit information to peripheral terminals, but there is a risk that unintended persons may receive the information, and it is unknown who transmitted the information. Then, it is conceivable to perform 1: 1 communication using the identifier on the network next time, but since the information is not exchanged in advance, the identifier on the partner network is unknown. Further, communication may be encrypted. However, since information is not shared in advance, an encryption key (password) is unknown. Therefore, directional communication is performed.

  2A to 2C show an example of a change in display due to a use state of the mobile terminal 100 and a difference in an operation mode of the video display unit 102. As shown in FIG. 2A, four persons, who are also equipped with the portable terminal 100, namely, B, H, D, and Bo, are standing in front of the instep equipped with the portable terminal 100, and only the third cave from the left is looking into the instep. Has been sent. 2B and 2C are images displayed on the image display unit of the portable terminal 100 worn by the instep. In the normal mode shown in FIG. 2B, the captured image is displayed as it is. In the marker mode shown in FIG. 2C, the position at which the control unit 114 detects the line of sight is specified to the image display unit 102, and the image with the marker M is displayed. Is done.

The other person's line of sight detection unit 103 outputs to the control unit 114 the presence or absence of a person having a line of sight to the center of the image and the position on the screen from the image captured by the image capturing unit 101 using a known line of sight detection method. For example, the following method is used.
First, a still image is cut out from a video shot by the video shooting unit 101, and a human face is detected from the still image by a technique such as edge detection or pattern detection. Next, the gaze direction can be determined based on the positions of the iris and the outer corner of the eye with respect to the detected human face image (Japanese Patent Application Laid-Open No. 2003-25682). If the direction of the line of sight matches the center of the image, the position of the person's face is output to the control unit 114. If the direction of the line of sight does not match or the face cannot be detected, the control unit determines that there is no line of sight. Output to 114.

  The self-line-of-sight detection unit 104 includes an infrared camera (not shown) or the like, and outputs the direction of the user's line of sight to the control unit 114. The user's eyeball is photographed by an infrared camera, and the direction of the user's line of sight is detected by a known line-of-sight detection method. For example, a method of detecting a line of sight from the center of the pupil and the center of the corneal curvature is known. By combining the detected line of sight with a video image captured by the video image capturing unit 101, an object viewed by the user is specified.

  The communication unit 105 includes a communication circuit (not shown), a non-directional antenna, a directional antenna, an antenna movable device, and the like. The control unit 114 controls not only the start and stop of communication but also the control of the antenna movable device and the electrical connection of the antenna. That is, in the mobile terminal 100, the control unit 114 performs communication in a specific direction by changing the direction of the antenna, and switches between omnidirectional communication and directional communication by electric circuit switching. The operation of the antenna by the control unit 114 accepts not only the operation according to the flowchart described below but also a manual operation by the user of the mobile terminal 100. Therefore, the user of the mobile terminal 100 uses the directional antenna via the control unit 114. You can freely control the direction of communication.

  The input unit 106 includes a touch sensor or the like, and transmits the timing, length, interval, and the like of the user touching the touch sensor to the control unit 114. In addition, the movement of a specific finger of the user photographed by the video photographing unit 101 or the movement of a specific black eye of the user detected by the self-gaze detecting unit 104 can be transmitted to the control unit 114 as an input. .

  The face feature amount calculation unit 107 calculates the end points of the eyes, the nose, and the lips, the mutual positional relationship, the contour of the face, the distribution of the color and brightness of the face, and the like by a known method as the feature amount of the face.

  The face feature database 108 inputs an image of a person's face classified by age or gender to the face feature amount calculation unit 107 and stores the obtained feature amounts, and is used by the gender / age estimation unit 110. You. In addition, the image captured by the image capturing unit 101 can be input to the facial feature amount calculation unit 107, and the obtained feature amount can be newly stored.

  The face similarity determination unit 109 learns using a plurality of feature amounts divided into two groups, which will be described later, determines which group the newly input feature amount is closer to, and outputs the group to the control unit 114. I do. For this determination, a known method can be used. For example, a method using a support vector machine disclosed in JP-A-2004-222118 can be used.

  The gender / age estimation unit 110 determines the gender and age of the input face by a known method, and outputs the determined gender and age to the control unit 114. Since the face feature database 108 has registered therein feature values of a plurality of faces whose genders and ages are known, the feature amount of the face to be determined can be arbitrarily determined using, for example, the face similarity determination unit 109. It is possible to repeatedly determine which of the two groups to form, and determine the gender and age.

  The data storage unit 111 includes a flash memory (not shown) or the like, and stores a program for executing the flowcharts illustrated in FIG. These programs are executed by the control unit 114.

  The notification unit 112 includes a speaker, a lamp, a vibrator, and the like (not shown), and operates under the control of the control unit 114. The position detection unit 113 includes a GPS (Global Positioning System) receiver (not shown) and the like, and is used by the control unit 114 to calculate the position of the mobile terminal 100.

  The control unit 114 includes a CPU, a ROM, a RAM, and the like (not shown). The CPU loads the program stored in the ROM or the data storage unit 111 into the RAM and executes the program.

Hereinafter, the operation of the first embodiment will be described.
The processing executed by the control unit 114 will be described with reference to FIG. The control unit 114 executes the program shown in the flowchart of FIG. 3 at regular intervals (for example, every minute). However, if the previous execution has not been completed, the execution waits until it is completed. Unless otherwise specified in the following description, the subject of execution of each step is the CPU of the control unit 114.

  In step S201, the control unit 114 receives information on the line of sight looking at the user from the other person's line of sight detecting unit 103 (there is no line of sight or the position of the face having the line of sight), and the process proceeds to step S202.

  In step S202, the control unit 114 determines whether or not a line of sight has been detected, and proceeds to step S203 if there is a line of sight to the user, and proceeds to step S208 if there is no line of sight to the user.

  In step S203, the control unit 114 sets the video display unit 102 to the marker mode, instructs the video display unit 102 to use the position of the face output by the other person's gaze detection unit 103 as the marker position, and proceeds to step S204. Therefore, until the image display unit 102 is changed to the normal mode, the image display unit 102 is a real-time image captured by the image capturing unit 101 with a fixed marker attached to a position designated by the control unit 114. Is displayed.

  In step S204, the control unit 114 physically and electrically operates a directional antenna (not shown) of the communication unit 105 to communicate with the person who has detected the line of sight in step S201, and proceeds to step S205. Physically, the direction in which the directional antenna emits strong radio waves is directed to the direction of the person who has detected the line of sight in step S201. Electrically, the directional antenna is connected to the communication semiconductor.

  In step S205, the control unit 114 attempts communication using the communication unit 105, and then proceeds to step 206. At this time, the communication unit 105 randomly switches between transmission and reception at regular time intervals, and attempts communication until a certain period elapses. This is because, in the first embodiment, the communication method and various parameters can be prepared in advance with the communication partner, but the communication timing and the communication direction (transmission side / reception side) cannot be determined. Details of step S205 will be described later with reference to FIG.

  In step S206, it is determined whether or not the communication in FIG. 4 is possible. If the communication is established, the process proceeds to step S207. If the communication is unsuccessful, the flowchart in FIG. 3 ends.

  In step S207, the control unit 114 gives permission to exchange contact information and pictures with the other party, or to permit photographing. The contact information is a telephone number, a mail address, an SNS (Social Networking Service) account, or the like. All of these pieces of information may be exchanged, or the user of the portable terminal 100 may select only a part in advance, or may select from the input unit when exchanging contact information.

  In step S208, since the line of sight facing the user is not detected, the control unit 114 sets the video display unit 102 to the normal mode and ends the flowchart of FIG. Therefore, until the video display unit 102 is changed to the marker mode, the real-time video captured by the video capturing unit 101 is displayed on the video display unit 102 as it is.

  FIG. 4 is a flowchart illustrating a trial of communication executed in step S205 of FIG. According to the program shown in this flowchart, the control unit 114 checks whether communication is established.

  In step S301, the control unit 114 reads the communication parameters, the response waiting time T, and the maximum number of trials N determined in advance from the data storage unit 111, and proceeds to step S302. The communication parameters include, for example, a communication speed and a parity bit in the case of infrared communication (IrDA-InfraredData Association).

  In step S302, the control unit 114 randomly determines the communication mode, and proceeds to step S303. For example, it is determined at random whether to set the communication mode to transmission or reception.

  In step S303, the control unit 114 attempts communication for a period of time T using the communication parameters read in step S301, and proceeds to step S304. Since the purpose is to confirm that communication is established, the content of the electronic message is not particularly limited.

  In step S304, the control unit 114 determines whether the communication (transmission or reception) in step S303 has been successful. If the communication has been established, it is determined that “communication has been established”, the flow chart of FIG. 4 ends, and the process returns to step S206 of FIG. 3, and if the communication has failed, the process proceeds to step S305.

  In step S305, the control unit 114 determines whether the number of communication attempts exceeds the maximum number of attempts N. If it has exceeded, it is determined that “communication has failed”, the flow chart of FIG. 4 is terminated, and the process returns to step S206 of FIG. 3, and if it does not exceed, the process returns to step S302 to keep trying the communication.

  Note that the response waiting time T and the maximum number of trials N are adjusted based on the connection establishment time of the communication standard to be used and the level of mobility of a person in the usage environment of the mobile terminal 100. The response waiting time T is desirably about twice as long as the connection establishment time of the communication standard to be used, in consideration of a cycle shift between the two portable terminals 100 communicating with each other.

Next, an example of communication between persons having the portable terminal 100 will be described with reference to FIGS. 3 and 5.
FIG. 5 shows a process until communication between the portable terminal 100a possessed by Party A and the portable terminal 100b possessed by Ding starts. The left side of FIG. 5 shows the operation of the portable terminal 100a or the instep, and the right side of FIG. In addition, time flows from the top to the bottom of the figure, and the electric (lightning) mark in the figure indicates that transmission of radio waves for communication has started, and the eye mark indicates that Party A or Party B is looking at the other party's eyes. Indicates that

  At time t1, the control unit 114 of the portable terminal 100a on the instep attempts to detect another person's line of sight to the instep (step S201 in FIG. 3), but fails to detect the line of sight. Step S208). Therefore, the video (FIG. 2B) as captured by the video capturing unit 101 is displayed on the video display unit 102 of the mobile terminal 100a.

  At time t2, the Ding begins to look at the instep, but since it is not the timing of detecting the other person's line of sight of the mobile terminal 100a, the image display unit 102 of the mobile terminal 100a does not change.

  At time t3, the control unit 114 of the mobile terminal 100a performs another person's line-of-sight detection again (step S201 in Fig. 3), and detects the line of sight of the Ding. Therefore, the control unit 114 of the mobile terminal 100a specifies the position of the second party in the video, changes the video display unit 102 to the marker mode, and specifies the position where the marker is to be displayed (see FIG. 3). Step S203).

  At time t4, following the previous process, the control unit 114 of the mobile terminal 100a directs the antenna toward the dent (step S204 in FIG. 3) and attempts communication toward the dent (step S205 in FIG. 3). ). However, at this time, communication is not established because the portable terminal 100b has not started communication yet (step S206: NO in FIG. 3).

At time t4, the video display unit 102 of the portable terminal 100a is already in the marker mode, and the marker M is attached to the din looking at the instep as shown in FIG. 2C. However, the former could not notice at the moment when it was displayed.
At time t5, the instep is displayed on the video display unit 102, and notices and looks at the claw with the marker.

  At time t6, since the control unit 114 of the portable terminal 100b detects the line of sight of the instep by detecting the line of sight of another party (step S201 in FIG. 3) (step S202 in FIG. 3: YES), the control unit 114 specifies the position of the instep. The mode of the video display unit 102 was changed. Further, the directional antenna is pointed at the instep A (step S204 in FIG. 3), and a communication trial with the instep portable terminal 100a (step S205 in FIG. 3) is started at time t7.

  At this time, since the control unit 114 of the mobile terminal 100a continues to detect the line of sight of the toe, it continues to try to communicate with the mobile terminal 100b, and communication is established at time t7 (step in FIG. 3). 206: YES). Therefore, after that, data exchange (step 207 in FIG. 3) is performed between Party A and Party D.

  According to the first embodiment, the portable terminal 100 includes a video photographing unit 101 for photographing a surrounding subject, a video display unit 102 for displaying a video photographed by the video photographing unit 101, and a video photographing unit 101. The camera includes a third-party gaze detection unit 103 that detects a line of sight from a captured image, and a communication unit 105 that can perform directivity communication. The third-party gaze detection unit 103 moves in a specific direction from the image captured by the image capturing unit 101. When the line of sight is detected, the communication unit 105 starts communication, so that it is possible to perform communication between persons wearing the portable terminal 100 and matching the line of sight.

  In addition, when the other person's line of sight detecting unit 103 detects a line of sight in a specific direction from the image captured by the image capturing unit 101, the mobile terminal 100 displays a marker indicating the starting point of the line of sight in the specific direction on the image. Since the information is displayed on the unit 102, the owner of the mobile terminal 100 can quickly know the presence of the person who is looking at the user. In addition, since the marker is superimposed on the captured video and displayed, the holder of the mobile terminal 100 can easily determine who is looking at the user.

(Modification 1)
In the first embodiment, in the trial of communication (step S303 in FIG. 4), the content of the message is not particularly limited. However, information necessary for encrypting the communication is transmitted and received, and the subsequent communication (FIG. 3 in FIG. 3) is performed. In step S207), encrypted communication may be performed. For example, a wireless LAN SSID (Service Set Identifier) and an encryption key (password) are transmitted and received. Then, in the subsequent communication, an omnidirectional antenna may be used, and communication may be performed using the SSID and the encryption key.
According to the first modification, stable high-speed communication can be performed while limiting the communication destination device. Although the directional antenna is used to limit the communication destination device, the communication speed is reduced or the communication is disconnected due to a change in the position or orientation of the mobile terminal 100 or a change in the position or the orientation of the communication destination terminal. There is a risk that. Therefore, by transmitting and receiving information necessary for encrypted communication while using the omni-directional antenna, stable communication can be performed while limiting the communication destination.

(Modification 2)
The video photographing unit 101 may be a camera that can photograph the entire circumference. For example, the portable terminal 100 may be of a glasses type, and a small camera may be embedded in the temples on the front and left and right sides of the glasses to capture 360 degrees horizontally.
According to the second modification, even if the user receives a line of sight from outside the field of view of the user, the user can notice the line of sight. This is because, when the other person's line of sight detecting unit 103 detects the line of sight, the person sending the line of sight to the user is marked with a marker and displayed on the video display unit 102.

(Modification 3)
After the user succeeds in the data exchange shown in step S207 in FIG. 3, a new person database may be constructed and the obtained information may be stored. Further, the age and gender may be determined by the gender / age estimation unit 110, and the output may be stored together. Further, when the communication unit 105 starts the communication shown in step S205 of FIG. 3, the position is detected by the position detection unit 113, and the video image capturing unit 101 captures an image of a person who is looking at the user, or an image of the video. May be stored together with the image cut out from the image.
According to the third modification, since the photograph and the data of the data exchange partner are stored together, there is an advantage that the person can be easily remembered. Furthermore, by photographing the other party from a slightly distant position, the whole body can be photographed, so that clothes and belongings are also recorded at the same time, and it is possible to obtain many opportunities to remember the person.

(Modification 4)
The shape of the portable terminal 100 is not limited to a terminal that can be fixed to the head like glasses, but may be a terminal that is fixed to the shoulder or the chest, or a terminal that is held by a plate-shaped or folded hand. In this case, since the position of the camera of the video photographing unit 101 and the position of the eyes of the holder of the portable terminal 100 do not match, the user inputs in advance the positional relationship between the two to the portable terminal 100, and the other party's gaze detecting unit 103 The direction of the line of sight to be detected is corrected.
In the case of a terminal held by hand, the same function as the portable terminal 100 in the first embodiment can be performed by holding the terminal in front of the user's face during use.
According to the fourth modification, various terminals can be used.

(Modification 5)
In the first embodiment, when the video display unit 102 is set to the marker mode, the video display unit 102 attaches the marker M to a position designated by the control unit 114, that is, a position at which the control unit 114 detects a line of sight. Although the video is displayed as a display, the display of the video display unit 102 in the marker mode is not limited to this. As shown below, as shown in FIG. 6, the control unit 114 may also describe, on the marker M, a schematic representation of the heart rate of the person whose gaze is detected.
This is based on the fact that the light absorption on the face surface increases when the blood on the face increases due to the heartbeat, and decreases when the blood on the face decreases. The surroundings of the position where the other person's line of sight has detected the line of sight, that is, the face of the person having the line of sight to the user, are analyzed by the image photographing unit 101 and the image is analyzed. By counting the number of increasing and decreasing cycles, the heart rate of a person having a line of sight to the user can be measured. Then, the control unit 114 evaluates the number of heart rates on a five-point scale, and outputs to the video display unit 102 such that as the heart rate increases, more heart marks are written on the marker M.

(Modification 6)
In the first embodiment, when the communication is established (step S206 in FIG. 3: YES), the user has selected the contact information to be transmitted by inputting to the input unit 106. Not limited to choice. When the communication is established, the control unit 11 specifies a position at which the interface 900 shown in FIG. 7 is to be displayed on the video display unit 102 and causes the video display unit 102 to display the interface 900. You may.

  The interface 900 is provided with a field for selecting whether or not to take a picture, a field for selecting whether or not to transmit information about the user, and a decision button. Information about the user includes, for example, a nickname, a mail address, a telephone number, an SNS account, a real name, and the like. When taking a photograph, an image displayed on the video display unit 102 including the marker M or an image of only a person with a marker is stored in the data storage unit 111. In the interface 900 shown in FIG. 7, the brackets indicate the selected state, the current setting indicates that a photograph is taken, the nickname is transmitted, but the e-mail address, telephone number, SNS account, and real name are not transmitted. ing. In the initial state, NO is selected in all cases, so that neither photography nor transmission of any data is performed.

  Eye movements by the user include, for example, wink and gaze. Winking is an act of intentionally closing the eyes for a short time, and the time during which the eyes are closed is longer than blinking, which is a physiological phenomenon. The gaze is a gaze of the user on the interface 900 shown in FIG. In the sixth modification, the mobile terminal 100 further includes a self wink detection unit 121 and a display fixation detection unit 122, as shown in FIG.

  The self wink detecting unit 121 for detecting wink includes two reflective photointerrupters corresponding to the left and right eyes. When detecting wink, the self wink detecting unit 121 outputs the wink detection to the control unit 114 together with information on the left or right eye. The reflection type photo interrupter emits infrared rays and receives the infrared rays reflected by the object. Since the reflectance of infrared rays differs between the eyeball and the skin, the time during which the eyes are closed can be measured by constantly irradiating the infrared rays emitted by the reflective photointerrupter to the eyes and monitoring the amount of received infrared rays. A case where the eyes are closed for a predetermined time, for example, 0.2 seconds to 1.0 seconds can be recognized as a wink.

  For example, the control unit 114 moves the cursor C one step down when the left eye wink is output from the self wink detection unit 121, and performs a toggle operation of the selector when the right eye wink is output. When the right-eye wink is output in a state where the cursor C is moved to “determination” by the left-eye wink, the control unit 114 performs transmission of the selected item and photography in accordance with the selection. That is, as shown in FIG. 7, when the wink detection is output from the self wink detection unit 121 to the control unit 114 when the cursor C is in the nickname column, the cursor C moves to the mail address column. When the wink detection of the right eye is output from the self wink detection unit 121 to the control unit 114 in the state shown in FIG. 7, NO is selected in the nickname column. In the state shown in FIG. 7, when the wink of the right eye is output in a state where the cursor C is set to "OK", photographing and transmission of a nickname are performed. Alternatively, only the wink of one eye may be detected, and two types of winks may be determined based on the time interval at which the wink was detected.

  The display gaze detection unit 122 that detects gaze acquires the direction of the user's gaze output from the self-gaze detection unit 104 and the position where the interface 900 is displayed on the video display unit 102 from the control unit 114. If the user's line of sight does not move for a predetermined time, for example, two seconds, it is determined that the user is gazing at something, and the user's gazing direction is determined by the direction of the user's line of sight and the image display unit 102. Is determined from the position where the interface 900 is displayed, and output to the control unit 114. When the target gazed by the user is output from the display gazing detection unit 122, the control unit 114 sets the gazing target to a selected state if the gazed target is a selection item, and sets the selected item if the gazing target is “determined”. Send and take a picture according to your selection. That is, in the state shown in FIG. 7, when the display gaze detecting unit 122 detects that the user gazes at YES in the column of the telephone number and thereafter gazes at “decision”, the control unit 114 sets the display gaze detecting unit 122 In response to the output, a photograph is taken and a nickname and a telephone number are transmitted.

According to the sixth modification described above, the following operation and effect can be obtained.
(1) The data storage unit 111 that records the captured video, the display gaze detection unit 122 that detects the position of the video display unit 102 that the user of the mobile terminal 100 is watching, and the other person's gaze detection unit 103 capture the video. A control unit 114 for outputting to the video display unit 102 an interface 900 for selecting whether or not to transmit information about the user and / or whether or not to take a photo when detecting a gaze to the user based on the video captured by the unit 101; , Is provided. The control unit 114 detects the user's selection on the interface 900 by the display gaze detection unit 122. Then, based on the user's selection, the video photographing unit 101 photographs a person having a line of sight in a specific direction and records the photographed person in the data storage unit 111, and / or transmits information about the user selected by the user through the communication unit 105. I do.

  Therefore, the user can select whether or not to take a picture and select a transmission item only by watching, and the operation is simple. In addition, the operation can be performed even when the hand cannot be used, such as when holding a baggage in both hands or holding on a hanging strap on a train.

(2) A data storage unit 111 that records a captured image, a self wink detection unit 121 that detects a wink operation of the user, and a user's line of sight detection unit 103 that transmits to the user based on the image captured by the image capturing unit 101. The control unit 114 further outputs, to the video display unit 102, an interface 900 for selecting whether or not to transmit information on the user and / or whether or not to take a picture when the gaze is detected. The control unit 114 detects the user's selection on the interface 900 by the self wink detection unit 121. Then, based on the user's selection, the video photographing unit 101 photographs a person having a line of sight in a specific direction and records the photographed person in the data storage unit 111, and / or transmits information about the user selected by the user through the communication unit 105. I do.

  Therefore, the user can select whether or not to take a picture or to select a transmission item simply by winking, and the operation is simple. In addition, since the wink operation can be performed quickly, there is an advantage that the operation can be performed in a short time.

It should be noted that the wink is detected by using an infrared camera provided in the self-line-of-sight detection unit 104, and when the user cannot recognize the black eye of the user from the image obtained by the infrared camera, it is determined that the user has closed his eyes, and the wink is detected. Is also good.
Note that the selection of the transmission item and the availability of photographing may be performed using both wink and gaze, or only one of them may be used. The items displayed on the interface 900 include at least one of a transmission item for selecting whether transmission is possible and an item for selecting whether photography is permitted.

(Modification 7)
In the first embodiment, information about the user is transmitted to the person sending the line of sight to the user (step S207 in FIG. 3), but the information to be transmitted is not limited to this. For example, a message may be sent to a person who is looking at the user. Further, when a message is transmitted in addition to information on the person sending the line of sight to the user, both of them may be received.

In step S207 in FIG. 3, when information to be transmitted is selected from the input unit, message creation and transmission are provided as one of the options. When creation and transmission of a message are selected by the user and character input from the input unit or voice input by a microphone (not shown) is performed, the control unit 114 transmits this as a message.
According to the seventh modification, the following operation and effect can be obtained.

(1) The communication started by the communication unit 105 when the other person's line of sight detecting unit 103 detects the line of sight to the user based on the image captured by the image capturing unit 101 includes transmission of a message by voice or text or / And receiving.
For this reason, the user can communicate not only by transmitting and receiving information about the user but also by immediately transmitting and receiving a message, which is useful in communication requiring immediacy. Although the message can be transmitted using the obtained user information, for example, the mail address of the mobile phone, the other party may not be able to receive the message immediately due to the congestion of the communication line. For example, there is an advantage that a message can be transmitted without being affected by a failure of another communication means. Further, there is an advantage that the message can be transmitted without informing the user of information such as an e-mail address at all.

(Second embodiment)
Hereinafter, a second embodiment of the present invention will be described with reference to FIG.

  The configuration in the second embodiment is almost the same as the configuration in the first embodiment, and the block diagram of the portable terminal 100 is the same as that in FIG. 1 described in the first embodiment. However, the programs stored in the data storage unit 111 are different. The data storage unit 111 also stores the preferred gender and age entered by the user.

Hereinafter, the operation of the second embodiment will be described.
The program shown in the flowchart of FIG. 9 is executed by the control unit 114 in the second embodiment, instead of the program shown in the flowchart of FIG. 3 in the first embodiment. Steps having the same contents as the processing in FIG. 3 of the first embodiment are denoted by the same step numbers, and description thereof is omitted. That is, the second embodiment is different from the first embodiment only in that steps S401 and S402 are added after step S202 of the first embodiment.

  In step S401, the control unit 114 inputs the face image of the person whose gaze is detected by the other person's gaze detection unit 103 in step S201 to the gender / age estimation unit 110 to estimate gender and age, and proceeds to step S402. move on.

  In step S402, the control unit 114 determines whether the gender and age estimated in step S401 match the preferred gender and age stored in the data storage unit 111. If both the gender and the age match, the process proceeds to step S203. If any one does not match, the process proceeds to step S208.

  According to the second embodiment, the portable terminal 100 includes an input unit 106 for the user to input his / her favorite gender and age, and a gender / age for estimating the gender and age from the video captured by the video capturing unit 101. When the gaze / age estimation unit 110 detects a gaze in a specific direction from the video captured by the video imaging unit 101, the gender / age estimation unit 110 further includes an estimation unit 110. The communication unit 105 starts communication when the estimated gender and age are matched with the input gender and age.

  Therefore, even if the user detects a line of sight from a person other than the gender and age preferred by the user, the portable terminal 100 behaves as if the line of sight has not been detected. That is, not only does the communication not start, but also the marker is not displayed on the video display unit 102, so that the user does not see unnecessary information.

(Third embodiment)
Hereinafter, a third embodiment of the present invention will be described with reference to FIGS.

  The configuration in the third embodiment is almost the same as the configuration in the first embodiment, and the block diagram of the portable terminal 100 is the same as that in FIG. 1 described in the first embodiment. However, the programs stored in the data storage unit 111 are different. Further, the face feature database 108 includes a face feature database 108P for storing the feature amount of the face watched by the user, and a face feature database 108N for storing the feature value of the face not watched by the user.

Hereinafter, the operation of the third embodiment will be described.
FIG. 10 is a flowchart illustrating processing of a program executed by the control unit 114. Unless otherwise specified in the following description, the subject of execution of each step is the CPU of the control unit 114.

  The control unit 114 executes the flowchart of FIG. 10 at regular time intervals (for example, every one minute) to determine whether the user is looking at the face type based on the presence or absence of the user's gaze. The face feature database 108P and the other type of face feature database 108N are constructed.

  In step S501, the control unit 114 repeatedly detects the user's line of sight by the self-line-of-sight detection unit 104 for a certain length of time, and proceeds to step S502. For example, when the flowchart of FIG. 10 is executed every minute, the line of sight is detected for 50 seconds. At this time, only one eye may detect the line of sight, but it is desirable to use both eyes in order to accurately specify the target that the user is gazing at in step S502.

  In step S502, the control unit 114 specifies the target that the user gazes at by combining the video captured by the video capturing unit 101 and information on the position where the user's line of sight has been fixed for a certain period of time, and proceeds to step S503. .

  In step S503, the control unit 114 determines whether or not the target watched by the user is a person. For example, when it is estimated that the target watched by the user is a human face by a method such as edge detection or pattern detection, it can be determined that the target watched by the user is a human. If the target being watched is a person, the process proceeds to step S504. If the target is not a person, the process proceeds to step S506.

  In step S504, the control unit 114 calculates the feature amount of the face of the person watched by the user by the face feature amount calculation unit 107, and proceeds to step S505. In step S505, the feature amount calculated in step S504 is stored in the face feature database 108P, and the process proceeds to step S506.

  In step S <b> 506, it is determined whether there is a person who has turned his / her face to the image capturing unit 101 in the image captured by the image capturing unit 101, in addition to the person who watched the user. If there is another person facing the face, the process proceeds to step S507, and if there is no other person facing the face, the flowchart of FIG. 10 ends.

  In step S507, the face feature amount is calculated in the same manner as in step S504. In step S508, it is stored in the face feature database 108N, and the flowchart in FIG. 10 ends.

  The program shown in the flowchart of FIG. 11 is executed by the control unit 114 in the third embodiment instead of the program shown in the flowchart of FIG. 3 in the first embodiment. Steps having the same contents as the processing in FIG. 3 of the first embodiment are denoted by the same step numbers, and description thereof is omitted. That is, the third embodiment is different from the first embodiment only in that steps S601 to S603 are added after step S202 of the first embodiment.

  In step S601, the control unit 114 inputs the face image of the person whose gaze is detected by the other person's gaze detection unit 103 in step S201 to the face feature amount calculation unit 107 to calculate the face feature amount, and proceeds to step S602. . In step S602, the control unit 114 determines which of the feature amount calculated in step S601 is closer to the feature amount stored in the face feature database 108P or the feature amount stored in the face feature database 108N. I do. The face similarity determination unit 109 can be used for this determination. If the result of the determination is that the face is close to the face feature database 108P, the process proceeds to step S603. If the result is close to the face feature database 108N, the process proceeds to step S208. In step S603, the control unit 114 activates the notification unit 112 to notify the user of the presence of a person similar to the person who has watched in the past by sound or vibration, and proceeds to step S203.

According to the third embodiment, the portable terminal 100 includes a notifying unit 112 for notifying a person using the portable terminal 100 and a face feature amount calculating unit for calculating a face feature amount of a person photographed by the video photographing unit 101. Unit 107, a face feature database 108 for accumulating face features, and a face similarity determination unit 109 for determining the similarity of faces. Among persons photographed by the video photographing unit 101, a gaze target detection process ( The features of the person's face detected in steps S501 and S502 in FIG. 10 are stored in the face feature database 108, and the other person's eye gaze detecting unit 103 detects a gaze in a specific direction from the video captured by the video capturing unit 101. At this time, when the face similarity determination unit 109 determines that the person having the line of sight is similar to the face stored in the database (step S602 in FIG. 11: YES), the notification unit 112 Characterized in that it informs the user of the terminal.
Therefore, the user's favorite face is constructed as a database by gazing, and when the gaze is detected, the user is notified by an alarm or vibration, so that the user does not miss the favorite face and exchanges contact information. be able to.

(Modification 8)
In any of the first to third embodiments, when the other person's line of sight detecting unit 103 detects the line of sight, the communication is started without any explicit action of the user. When 103 detects a line of sight in a specific direction from an image captured by the image capturing unit 101, the self-line-of-sight detection unit 104 detects a line of sight to a person having a line of sight in the specific direction, and the line of sight is The communication may be started when it is fixed.
According to the eighth modification, since communication is not started unless the user gazes, communication with an unintended partner can be avoided.

(Modification 9)
In any of the first to third embodiments, when the other person's line of sight detecting unit 103 detects the line of sight to the user even once, communication with the person having the line of sight to the user is started. Is not limited to this. The condition may be that communication is started with a predetermined number of gaze detections, pupil enlargement, winking, or a combination thereof. Furthermore, communication may be started on the condition that the user of the mobile terminal 100 detects a line of sight to a person having a line of sight to the user, enlarges the pupil, winks, or a combination thereof. This is based on the human habit of seeing an object of interest many times, sometimes winking at a favored partner, and expanding the pupil when viewing an object of interest. It is.

  FIG. 12 is a block diagram illustrating a configuration of the mobile terminal 100 according to the ninth modification. In addition to the configuration of the mobile terminal 100 shown in FIG. 1 according to the first embodiment, a self wink detection unit 121, another person wink detection unit 123, a self pupil measurement unit 124, and another person pupil measurement unit 125 Is further provided. The other configuration is the same as that of the first embodiment, and the description is omitted.

  The self wink detecting unit 121 that detects the wink of the user of the mobile terminal 100 includes two reflective photointerrupters corresponding to the left and right eyes. Output detection. The reflection type photo interrupter emits infrared rays and receives the infrared rays reflected by the object. Since the reflectance of infrared rays differs between the eyeball and the skin, the time during which the eyes are closed can be measured by constantly irradiating the infrared rays emitted by the reflective photointerrupter to the eyes and monitoring the amount of received infrared rays. A case in which the eyes are closed for a predetermined time, for example, 0.2 seconds to 1.0 seconds can be detected as wink.

  The other person wink detecting unit 123 that detects a wink of a person having a line of sight to the user acquires an image from the image capturing unit 101, specifies a person having a line of sight to the user, and monitors the eye movement of the person. . Since the color of the eyes is different from the color of the skin, the state in which the eyes are closed and the state in which the eyes are open can be determined by the color. Then, a case where the time during which the eyes are closed is a predetermined time, for example, 0.2 to 1.0 seconds can be detected as wink.

  The self-pupil measurement unit 124 that measures the size of the pupil of the user of the mobile terminal 100 includes a camera (not shown), captures the eyes of the user, and outputs the size of the pupil to the control unit 114. The pupil is black and has a circular iris around the pupil having a radial linear pattern centered on the pupil. Therefore, from the photographed eye image, two concentric circles, the inside of the inner circle is painted black, and the outer circle is searched for a shape having a radial straight line pattern, and the inner circle is searched for. It can be determined as a pupil. Then, the size of the pupil is output to the control unit 114 by the number of pixels of the captured image. In the ninth modification, since the size of the pupil is measured in order to detect a change in the relative size, information on the physical size is not required.

  The other person's pupil measurement unit 125 that measures the size of the pupil of a person having a line of sight to the user includes a camera, a motorized zoom lens, and a motorized pan head (not shown), and photographs the eyes of the person having a line of sight to the user. Then, the size of the pupil is output to the control unit 114. The other person's pupil measurement unit 125 acquires an image from the image photographing unit 101, specifies a person having a line of sight to the user, and controls the electric zoom lens and the electric pan head so that the person's eyes are photographed as large as possible. Then, an eye of a person having a line of sight to the user is photographed. The pupil detection method is the same as that of the self-pupil measurement unit 124.

In the first to third embodiments, step S202 in FIGS. 3, 9, and 11 determines presence / absence of a line of sight to the user by the other person's line of sight detecting unit 103. It may be determined whether or not the above condition or a condition combining A to C is satisfied.
A) The other person's line of sight detecting unit 103 detects the line of sight to the user a predetermined number of times or more within a predetermined time. For example, detecting the line of sight to the user three times or more per minute. However, if the user keeps sending the line of sight to the user, the conversion is performed once for a certain period of time, for example, every 10 seconds. In addition, the predetermined time is the time after the first detection of the line of sight. If the predetermined number of times is one, there is no time limit.

  B) The other person's wink detection unit 123 detects a wink operation of a person having a line of sight to the user a predetermined number of times or more within a predetermined time. For example, detecting twice or more in one minute. However, the predetermined time is the time after the wink is first detected, and if the predetermined number of times is one, there is no time limit.

C) The other person's pupil measurement unit 125 measures the diameter of the pupil of a person having a line of sight to the user for a certain period of time, and the maximum diameter in the measurement time is equal to or more than a predetermined ratio with respect to the minimum diameter. For example, the maximum diameter of the pupil in one-minute measurement is 1.2 times or more the minimum diameter.
That is, by combining the above A and B, the other person's line of sight detecting unit 103 detects the line of sight to the user twice or more in one minute, and the other person's wink detecting unit 123 further performs the wink operation of the person having the line of sight to the user. Whether or not is detected once or more may be used as the determination condition in step S202 in FIGS. 3, 9, and 11.

  Further, immediately before or immediately after step S203 in FIGS. 3, 9, and 11, it is determined whether any of the following conditions D to F or a condition combining D to F is satisfied. The process may be continued only when it is determined that the condition is satisfied, and may be terminated when it is determined that the condition is not satisfied.

  D) The self-line-of-sight detection unit 104 detects a gaze at a person having a line of sight to the user a predetermined number of times or more within a predetermined time. For example, detecting gaze at a person having a line of sight to a user twice or more in one minute. However, when gaze is continued, conversion is performed once for a certain period of time, for example, every 10 seconds. In addition, the predetermined time is the time after the first detection of the line of sight. If the predetermined number of times is one, there is no time limit.

E) The self wink detection unit 121 detects the user's wink operation a predetermined number of times or more within a predetermined time after the self-gaze detection unit 104 detects gaze at a person having a gaze to the user. For example, detecting twice or more in one minute. However, the predetermined time is the time after the wink is first detected, and if the predetermined number of times is one, there is no time limit.
F) The self-pupil measuring unit 124 measures the diameter of the user's pupil for a certain period of time within a predetermined time after the self-gaze detecting unit 104 detects gaze at a person having a line of sight to the user. The diameter is equal to or greater than a predetermined ratio with respect to the minimum diameter. For example, the maximum diameter of the pupil in one-minute measurement is 1.2 times or more the minimum diameter.

  That is, by combining the above D and E, the self-gaze detection unit 104 detects the gaze of the person having the gaze to the user twice or more in one minute, and further, the self-wink detection unit 121 detects the user's wink operation by one. The process may be continued when it is determined that the detection has been performed more than once, and when it is determined that this condition is not satisfied, the flowcharts of FIGS. 3, 9 and 11 may be ended.

Further, the replacement of the determination conditions of A to C in step S202 and the addition of the determination of D to F immediately before or immediately after step S203 may be combined.
According to the ninth modification, the following operation and effect can be obtained.
(1) When the communication unit 105 detects a line of sight to the user at a first predetermined number of times or more within a first predetermined period based on the video captured by the video capturing unit 101 by the other party's visual line detection unit 103, When the self-line-of-sight detection unit 104 detects the gaze of the person having the line of sight to the user at least a second predetermined number of times within the second predetermined period, the communication is started.
Therefore, the communication is not started only when the line of sight is accidentally detected, but is started only when the line of sight is adjusted a plurality of times with an interest, so that unnecessary communication can be reduced.

(2) The portable terminal 100 detects the wink operation of the user of the portable terminal 100 by a self-wink detecting unit 121, and the other detects the wink operation of a person having a line of sight to the user from the video image captured by the video capturing unit 101. A wink detection unit 123. The communication unit 105 starts communication when the wink detection unit 123 detects a wink operation and the self-line-of-sight detection unit 104 detects a wink operation of the user.
Therefore, by detecting wink, which is a means of clearly indicating the favor, communication is started only when there is an interest in each other, and communication with a partner whose friendship is not clear can be avoided.

(3) The communication unit 105 causes the other person's wink detection unit 123 to detect the wink operation of the person having the line of sight to the user for a third predetermined number of times or more within the third predetermined period, and to perform the self-line-of-sight detection unit 104 When the self wink detection unit 121 detects the user's wink operation for a fourth predetermined number of times or more within a fourth predetermined period after detecting the gaze at a person having a line of sight to the user, communication is started.
Therefore, by setting the third predetermined number of times to be large, it is possible to more positively communicate only with a person who shows an action by winking.

(Fourth embodiment)
Hereinafter, a fourth embodiment of the present invention will be described with reference to FIGS.
The fourth embodiment is characterized in that a matching server searches for a communication target.

FIG. 13 shows a configuration according to the fourth embodiment.
The communication system illustrated in FIG. 13 includes a plurality of mobile terminals 100 and a matching server 200, which are connected to each other by an IP (Internet Protocol) network. In the following description, when it is necessary to distinguish a plurality of mobile terminals 100, a and b are added to the end and expressed as 100a and 100b.

The configuration of the mobile terminal 100 is the same as FIG. 1 described in the first embodiment except for the following four points. At the first point, the position detection unit 113 further includes a compass. Second, the programs stored in the data storage unit 111 are different. Third, since the communication unit 105 only needs to be able to perform omnidirectional communication, the directional antenna and the antenna movable device may not be provided. Fourth, the data storage unit 111 further stores the unique identification information of the mobile terminal 100.
The mobile terminal 100 constantly transmits to the matching server 200 two pieces of position information, that is, the position of the mobile terminal 100 and the unique identification information of the mobile terminal 100. When specifying the target that the user is gazing at, the mobile terminal 100 determines the position of the mobile terminal 100, the direction in which the mobile terminal 100 is facing, the time at which the user is gazing, the unique identification information of the mobile terminal 100, and the gazing target. The photographed image of the person is transmitted to the matching server 200 as gaze information.

Hereinafter, the configuration of the matching server 200 will be described.
The communication unit 201 is used for communication with the mobile terminal 100. In the position database 202, the above-described position information transmitted from the portable terminal 100, that is, the position of the portable terminal 100 and the unique identification information of the portable terminal 100 are recorded.

  The gaze target search unit 203 searches for another mobile terminal 100b that the user of the mobile terminal 100a is watching, based on the above-mentioned gaze information and the position database 202 transmitted from the mobile terminal 100a. For example, assuming that the watch target holds the mobile terminal 100b, the watch target holding the mobile terminal 100b can be found by the following procedure.

First, a starting point and a direction for searching for the mobile terminal 100b are determined from the position and the facing direction of the mobile terminal 100a included in the gaze information. Then, the distance L between the mobile terminal 100a and the mobile terminal 100b is estimated from the size of the captured image of the gaze target person included in the gaze information (for example, how many pixels represent the whole body). Furthermore, by referring to the time of gazing contained in the gaze information and the positions of all the mobile terminals 100 stored in the position database 202, the corresponding mobile terminal 100b can be found.
For example, when the above-mentioned gaze information and the position database 202 transmitted from the mobile terminal 100a are visualized, the result is as shown in FIG. The positions of all the mobile terminals 100 are clear from the position database 202, and the positions of the mobile terminals 100a possessed by the watcher, the direction in which the mobile terminals 100a are facing, the distance L from the mobile terminals 100a to the mobile terminals 100b. Is known, the portable terminal 100b can be found as shown in FIG.
When the mobile terminal 100b to be watched is found as a result of the search, the unique identification information (hereinafter, watched terminal information) of the mobile terminal 100a possessed by the watcher is stored in the matching database 204. The three pieces of information of the unique identification information of the portable terminal 100b possessed (hereinafter, watched terminal information) and the time of watching are written. Further, these three pieces of information are transmitted to the portable terminal 100b to be watched.

The matching determination unit 205 searches the matching database and determines, using the unique identification information, that there is a pair of mobile terminals that are gazing at each other. The terminals 100 communicate with each other.
The control unit 206 includes a CPU, ROM, and RAM (not shown). The control unit 206 constantly monitors data received by the communication unit 201. If the data is position information, the data is stored in the position database 202. The unit 203 is started.

Hereinafter, the operation of the fourth embodiment will be described.
The program shown in the flowchart of FIG. 15 is executed by the control unit 114 of the mobile terminal 100 in the fourth embodiment. Steps having the same contents as the processing in FIG. 3 of the first embodiment and the processing in FIG. 10 of the third embodiment are denoted by the same step numbers, and description thereof is omitted. Unless otherwise specified in the following description, the subject of execution of each step is the CPU of the control unit 114.

  In step S701, the position of the mobile terminal 100 detected by the position detection unit 113 and the unique identification information of the mobile terminal 100 are transmitted to the matching server 200, and the process proceeds to step S501. This information is stored in the position database 202 of the matching server 200. In steps S501 to S502 following this, the target that the user is gazing at is specified. In step S503, if the target is a person, the process proceeds to step S702; otherwise, the process proceeds to step S704.

  In step S702, the azimuth of the mobile terminal 100a while the user is gazing is detected by the azimuth meter of the position detection unit 113, and the process proceeds to step S703. In step S703, the above-mentioned gaze information, that is, the position of the mobile terminal 100a, the orientation of the mobile terminal 100a, the time of gaze, the unique identification information of the mobile terminal 100a, and the captured image of the gaze target are stored in the matching server 200. Send to

  In a succeeding step S704, it is determined whether or not the matching server 200 has notified that the line of sight has been received since the execution of the previous step S704. If the notification has been received, the process proceeds to step S203. If the notification has not been received, the process proceeds to step S208. In addition, since the notification that the line of sight is received includes information indicating the relative positional relationship between the person sending the line of sight and the person receiving the notification, the control unit 114 The position displayed on the video photographing unit 101 is estimated, and the position is instructed to the video display unit 102 as the marker position.

  In step S705 executed after step S203 or step S208, it is determined whether or not communication partner information has been received from matching server 200 since the previous execution of step S705. If the information has been received, the data exchange of step S207 is performed using the received communication information. If not received, the flow chart of FIG. 15 ends.

  Next, the operation of the gaze target search unit 203 of the matching server 200 will be described. FIG. 16 is a flowchart illustrating the operation of the program executed by the fixation target search unit 203. Since the communication unit 201 is always waiting for communication from the mobile terminal 100, when the gaze information is received from the mobile terminal 100, the program illustrated in the flowchart of FIG.

  In step S801, the gaze target search unit 203 searches for another mobile terminal 100b that the user of the mobile terminal 100a is watching, based on the above-described gaze information and the position database 202 transmitted from the mobile terminal 100a. .

In a succeeding step S802, it is determined whether or not the corresponding portable terminal 100b has been found in the step S801. When the portable terminal 100b can be found, the process proceeds to the step S803.
In step S803, the gaze target search unit 203 notifies the found portable terminal 100b that the user has been gazing and the position of the portable terminal 100a used by the gazing person, and proceeds to step S804.

  In step S804, the gaze target search unit 203 stores 100a as the gaze terminal information, 100b as the gaze target terminal information, and the time of gaze in the matching database 204, and terminates the flowchart illustrated in FIG.

  Finally, the operation of the matching determination unit 205 of the matching server 200 will be described. The matching determination unit 205 monitors the writing of the matching database 204, and reads the entire matching database 204 when there is any writing. Then, a combination in which the unique identification information described in the watched terminal information and the watched terminal information is switched, that is, a combination in which the user is gazing in both directions is searched. When such a combination is found, the identification information (for example, IP address) of those portable terminals 100 on the network is transmitted to both portable terminals 100.

  According to the fourth embodiment, since the matching server 200 searches for the mobile terminal 100 of the watching target, the mobile terminal 100 only needs to have a device capable of performing omnidirectional communication. No moving devices are required. Therefore, the manufacturing cost and the rate of failure of the portable terminal 100 are reduced. In addition, since the gazed information can be left in the matching database 204, the two can not exchange information at the same time, and can exchange contacts even when gazed at each other for a few seconds or gazed at a different day. .

(Modification 10)
In the fourth embodiment, when the matching determination unit 205 finds a combination that is being watched in both directions, the matching determination unit 205 transmits the identification information of the other mobile terminal 100 on the network to those mobile terminals 100, and thereafter, the mobile terminal 100 Although the communication is performed between the terminals 100, the communication may be performed via the matching server 200.
According to the tenth modification, since it is possible to leave the record of communication in the matching server 200 without knowing the identification information on each other's network, there is an advantage that security is excellent.

Reference Signs List 100: portable terminal, 101: video photographing unit, 102: video display unit, 103: other person's line of sight detection unit, 104: self-line of sight detection unit, 105: communication unit, 106: input unit, 107: face feature amount calculation unit, 108: face feature database, 109: face similarity determination unit, 110: gender / age estimation unit, 111: data storage unit, 112: notification unit, 113: position detection unit, 114: control unit, 121: self wink detection unit , 122: display gaze detection unit, 123: wink detection unit of another person, 124: self-pupil measurement unit, 125: pupil measurement unit of another person, 200: matching server, 201: communication unit, 202: position database, 203: gaze target Searching unit, 204: matching database, 205: matching determining unit, 206: control unit

Claims (10)

A shooting unit,
A gaze detection unit that detects a gaze in a specific direction based on an image captured by the imaging unit,
A notification unit that notifies that the line-of-sight detection unit detects the line-of-sight,
A feature amount calculation unit that calculates a feature of the face imaged by the imaging unit,
A storage unit that stores the features of the face calculated by the feature amount calculation unit,
The notifying unit detects a line of sight in the specific direction based on an image captured by the line of sight detecting unit by the image capturing unit, and when the person having the line of sight is similar to the feature stored in the storage unit, the mobile unit may perform mobile communication. A portable information terminal that notifies the user of the information terminal. The mobile information terminal according to claim 1,
The portable information terminal is a direction toward the eyes of a user of the portable information terminal. The mobile information terminal according to claim 1 or 2,
A portable information terminal comprising: a display unit that, when detecting a line of sight in the specific direction, displays an index indicating a position of a viewpoint of the line of sight in the specific direction. The portable information terminal according to any one of claims 1 to 3 , wherein
The storage unit stores gender and age ,
When the line of sight in the specific direction is detected by the line of sight detection unit, based on an image captured by the image capturing unit, a gender age estimation unit that estimates the gender and age of the person having the line of sight,
A determination unit that determines whether the gender and age estimated by the gender age estimation unit match the gender and age stored in the storage unit ,
The portable information terminal that notifies , when the determination unit determines that the gender and age based on the video imaged by the imaging unit match the gender and age stored in the storage unit . The mobile information terminal according to claim 4, wherein
The determination unit further comprises a portable information terminal a communication unit that performs communication when sex and age based on the image captured is determined to be consistent with gender and age stored in the storage unit by the imaging unit. The portable information terminal according to any one of claims 1 to 4 ,
A gaze target detection unit that detects a target gazed by the user of the portable information terminal,
When the sight line detection unit, said detecting a line of sight to a specific direction based on the image taken by the imaging unit, the gaze target detection unit detects a gaze of the person having a line of sight of the specific direction, A portable information terminal further comprising a communication unit for performing communication. The portable information terminal according to claim 6, wherein
The communication unit detects the line of sight in the specific direction at least a first number of times within a first period based on an image captured by the line of sight detection unit by the imaging unit, and the gaze target detection unit detects the line of sight. A portable information terminal that performs communication when the number of times that a gaze target and a person having a line of sight in the specific direction detected by the line of sight detection unit are the same exceeds a second number within a second period. The portable information terminal according to claim 6 or 7 ,
A first wink detection unit that detects a wink operation of a user of the portable information terminal;
A second wink detection unit that detects a wink operation of a person having a line of sight in the specific direction from the image captured by the imaging unit,
The portable information terminal, wherein the communication unit communicates when the second wink detection unit detects a wink operation and the first wink detection unit detects a wink operation. The portable information terminal according to claim 8 , wherein:
In the communication unit, the second wink detection unit detects a wink operation of a person having a line of sight in the specific direction at least a third number of times within a third period, and the gaze target detection unit detects the wink operation. When the first wink detection unit detects a wink operation of the user of the portable information terminal by a fourth number of times or more within a fourth period after detecting gaze at a person having a line of sight in the direction of Personal digital assistant to do. Shooting procedure to shoot,
A gaze detection procedure for detecting a gaze in a specific direction based on the video taken in the shooting procedure,
A notification procedure for notifying that the visual axis is detected in the visual axis detection procedure,
A feature amount calculating procedure for calculating features of the face photographed in the photographing procedure,
Storing a feature of the face calculated in the feature amount calculation procedure,
The notifying step detects a line of sight in the specific direction based on the image captured in the image capturing step by the line of sight detection step, and when the person having the line of sight is similar to the feature stored in the storage step, the mobile terminal performs A program that includes a procedure for notifying a user of an information terminal and causes a computer to execute each procedure .

Images