JP2011205475A - Portable electronic device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a portable electronic device which uses a simple configuration to more preferably respond to an incoming call, depending on the party on the other end.SOLUTION: A cellular phone 1 includes: an imaging controller 462 which, when receiving an incoming call from a communication part 42, activates and causes a main camera 33 or a sub-camera 23 to capture an image of a subject; a face recognition part 464 which recognizes a human face from the captured image data of the subject; a facial expression recognition part 465 which recognizes a facial expression of the recognized face; and a conversation controller 466 which responds in a different manner to an incoming call received from the communication part 42 based on the recognized facial expression.

Description

  The present invention relates to a portable electronic device having an imaging unit.

  In recent years, portable electronic devices such as mobile phones have become widespread and can make calls in various situations. On the other hand, the mobile electronic device may accept an incoming call even in a situation where the user does not want to talk. In view of this, a portable electronic device that can determine whether or not a user makes a call in response to an incoming call has been proposed (see, for example, Patent Document 1).

  In Patent Literature 1, information indicating the importance of the other party is displayed on the display unit together with information for notifying an incoming call, and whether or not a call is made with respect to the incoming call when the user visually recognizes the displayed information. A portable electronic device that can be determined by a user is disclosed.

Japanese Patent Laid-Open No. 2003-218995

  However, in the portable electronic device described in Patent Literature 1, information indicating the importance level of the other party is created, and a configuration for transmitting information indicating the importance level of the other party is added together with information for notifying incoming calls. There was a need to do.

  In view of the above, an object of the present invention is to provide a portable electronic device that can more suitably respond to an incoming call according to a destination using a simple configuration.

  In order to solve the above problems, a portable electronic device according to the present invention includes an imaging unit, a communication unit that receives a signal related to an incoming call, a notification unit that notifies information about a destination that has received an incoming call by the communication unit, When an incoming call is received by the communication unit, an imaging control unit that captures an image of the subject by the imaging unit, and an expression that recognizes the facial expression from the face of the person in the image data of the subject imaged according to the control of the imaging control unit A recognition unit; and a call control unit that responds in a different manner to the incoming call received by the communication unit based on the facial expression recognized by the facial expression recognition unit.

  The portable electronic device further includes a facial expression determination unit that determines whether the facial expression recognized by the facial expression recognition unit is a first facial expression, and the call control unit includes the facial expression determination unit. When it is determined that the facial expression is the first facial expression, it responds to the incoming call received by the communication unit with a preset first voice, and the facial expression determining unit responds with the facial expression. When it is determined that is not the first facial expression, it is preferable to make a call with respect to the incoming call received by the communication unit.

  The call is preferably a hands-free call.

  The facial expression determination unit determines whether the facial expression recognized by the facial expression recognition unit is a second facial expression, and the call control unit determines whether the facial expression is the facial expression by the facial expression determination unit. When it is determined that the expression is the second facial expression, it is preferable to respond to the incoming call received by the communication unit with a preset second voice.

  In addition, when the portable electronic device receives an incoming call by the communication unit, the portable electronic device causes the notification unit to notify the information on the partner in the first mode, and recognizes the facial expression by the facial expression recognition unit. It is preferable to provide a notification control unit that causes the notification unit to notify the information regarding the second aspect.

  The facial expression recognition unit recognizes the movement of the person along with the facial expression from the image data captured by the imaging unit, and the facial expression determination unit recognizes the facial expression recognized by the facial expression recognition unit. Is a third facial expression, and whether or not the action of the person is a predetermined action, the call control unit determines that the facial expression is a third facial expression and the action of the person is the If it is determined by the facial expression determination unit that the movement is a predetermined action, it responds with the second voice to the incoming call received by the communication unit, and the facial expression is not the third facial expression, or the person When the facial expression determination unit determines that the operation is not a predetermined operation, it is preferable to make a call with respect to the incoming call received by the communication unit.

  The call is preferably a hands-free call.

  The portable electronic device includes a count unit that counts the number of times the facial expression is recognized by the facial expression recognition unit, and a storage unit that stores the number of times counted by the count unit for each facial expression. And a first threshold value determination unit that determines whether or not the number of times the first facial expression is recognized exceeds a first threshold value, and a first threshold value that determines whether or not the number of times the second facial expression is recognized exceeds a second threshold value. A second threshold value determination unit, wherein the call control unit is configured to set a preset value for the incoming call received by the communication unit when the number of times the first facial expression is recognized exceeds the first threshold value. It is preferable to respond with two voices.

  The call control unit preferably performs a hands-free call for the incoming call received by the communication unit when the number of times the second facial expression is recognized exceeds the second threshold.

  In addition, the call control unit, when the number of times the second facial expression is recognized does not exceed the second threshold and the number of times the first facial expression is recognized does not exceed the first threshold, It is preferable to respond with the first voice set in advance to the incoming call received by.

  In order to solve the above problems, a portable electronic device according to the present invention receives an image pickup unit, a communication unit that receives an incoming call, a notification unit that notifies information about a destination that has received an incoming call by the communication unit, and an incoming call And picking up the subject, recognizing the face of the person from the image data of the picked up subject, recognizing the recognized facial expression, and differing from the received incoming call based on the recognized facial expression And a control unit that responds in a manner.

  ADVANTAGE OF THE INVENTION According to this invention, the portable electronic device which can perform the response with respect to an incoming call more suitably according to the other party can be provided using a simple structure.

It is an external appearance perspective view in the state where a cellular phone was opened. It is a perspective view of the state which folded the mobile phone. It is a block diagram which shows the function structure of the mobile telephone which concerns on 1st Embodiment. It is a flowchart shown about the process performed by the control part which concerns on 1st Embodiment. It is a flowchart shown about the process performed by the control part which concerns on 2nd Embodiment. It is a flowchart shown about the process performed by the control part which concerns on 3rd Embodiment. It is a figure shown about the operation | movement of the person recognized by the facial expression recognition part. It is a flowchart shown about the process performed by the control part which concerns on 4th Embodiment. It is a block diagram which shows the function structure of the mobile telephone which concerns on 5th Embodiment. It is a figure shown about the structure of the address book which concerns on 5th Embodiment. It is a flowchart shown about the process performed by the control part which concerns on 5th Embodiment. It is a flowchart shown about a facial expression count process. It is a flowchart shown about the other form of an expression count process.

<First Embodiment>
The best mode for carrying out the present invention will be described below with reference to the drawings.
With reference to FIG. 1, a basic structure of a mobile phone 1 as a mobile electronic device will be described. FIG. 1 shows an external perspective view of the cellular phone 1 in an opened state.

  As shown in FIG. 1, the mobile phone 1 includes an operation unit side body 2 and a display unit side body 3 as a housing. The operation unit side body 2 and the display unit side body 3 are connected so as to be openable and closable via a connecting part 4 having a hinge mechanism. Specifically, the upper end portion of the operation unit side body 2 and the lower end portion of the display unit side body 3 are connected via a connecting portion 4. Thereby, the mobile phone 1 is configured to be able to relatively move the operation unit side body 2 and the display unit side body 3 connected via the hinge mechanism. That is, in the mobile phone 1, the operation unit side body 2 and the display unit side body 3 are opened (open state), and the operation unit side body 2 and the display unit side body 3 are folded. (Closed state). Here, the closed state is a state in which both housings are arranged so as to overlap each other, and the open state is a state in which both housings are arranged so as not to overlap each other.

  The outer surface of the operation unit side body 2 includes a front case 2a and a rear case 2b. The operation unit side body 2 is configured such that an operation key group 11 and a microphone 12 as a microphone to which a voice uttered by a user of the mobile phone 1 is input are exposed on the front case 2a side. The

  The operation key group 11 includes a function setting operation key 13 for operating various functions such as various settings, a telephone book function, and a mail function, and an input operation key 14 for inputting a telephone number and characters such as mail. And a determination operation key 15 as an operation member for performing determination in various operations, scrolling in the vertical and horizontal directions, and the like. Each key constituting the operation key group 11 has a predetermined function according to the open / close state of the operation unit side body 2 and the display unit side body 3, various modes, or the type of the activated application. Assigned (key assignment). Then, when the user presses each key, an operation corresponding to the function assigned to each key is executed.

  The microphone 12 is disposed on the outer end side opposite to the connecting portion 4 side in the longitudinal direction of the operation unit side body 2. That is, the microphone 12 is disposed on one outer end side when the mobile phone 1 is in the open state.

  An interface (not shown) for communicating with an external device (for example, a host device) is disposed on one side surface of the operation unit side body 2. A side key to which a predetermined function is assigned and an interface (not shown) through which an external memory is inserted and removed are arranged on the other side surface of the operation unit side body 2. The interface is covered with a cap. Each interface is covered with a cap when not in use.

The outer surface of the display unit side body 3 includes a front panel 3a, a front case 3b, a rear case 3c, and a rear panel 3d. In the display unit side body 3, a main display 21 for displaying various information, a speaker 22 as a receiver for outputting the voice of the other party of the call, and a sub camera 23 for imaging a subject are exposed to the outside. Are arranged as follows.
The main display 21 includes a liquid crystal panel, a drive circuit that drives the liquid crystal panel, and a light source unit such as a backlight that emits light from the back side of the liquid crystal panel.

  FIG. 2 is a perspective view of the cellular phone 1 in a folded state. The operation unit side body 2 includes, on one side surface, a side key 30 to which a predetermined function is assigned and an interface cap 31 in which external memory is inserted and removed. Further, the display unit side body 3 has a main camera 33 that captures an image of a subject, a light 34 that irradiates the subject with light, and a sub that displays various information on the surface of the rear panel 3d of the display unit side body 3. A display 35 and an LED 36 for notifying an incoming call or e-mail are formed.

FIG. 3 is a block diagram showing a functional configuration of the mobile phone 1 according to the first embodiment.
As shown in FIG. 3, the mobile phone 1 includes an operation key group 11, a microphone 12, a main display 21, a speaker 22, a sub camera 23, a main camera 33, a sub display 35, an LED 36, and communication. A unit 42, a vibrator 43, a storage unit 44, a timer 45, and a control unit 46 are provided.

  The microphone 12 and the speaker 22 operate according to the control of the control unit 46. For example, when performing a hands-free call, the control unit 46 performs control so that the sensitivity of the microphone 12 is higher than in a normal call, and increases the volume of the speaker 22 than in a normal call. To do.

  The main camera 33 and the sub camera 23 include a camera module including a lens, a CCD (Charge Coupled Device), a drive mechanism, a DSP (Digital Signal Processor), and the like. The main camera 33 and the sub camera 23 capture an image of a subject and generate image data. Then, the main camera 33 and the sub camera 23 output the generated image data to the control unit 46.

  The communication unit 42 includes a main antenna 40 and an RF circuit unit 41, and communicates with a predetermined communication destination. Examples of communication destinations with which the communication unit 42 communicates include an external terminal device that transmits / receives telephones and mails to / from the mobile phone 1, and an external device such as an external Web server that connects the mobile phone 1 to the Internet.

  The RF circuit unit 41 demodulates the signal received by the main antenna 40 and supplies the processed signal to the control unit 46. In addition, the RF circuit unit 41 modulates the signal supplied from the control unit 46 and transmits it to the external apparatus (base station) via the main antenna 40.

  The main display 21 and the sub display 35 are configured by an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence), and display images, moving images, and the like based on various types of information under the control of the control unit 46.

  The vibrator 43 is attached to the inside of the operation unit side body 2 and generates vibrations under the control of the control unit 46. When the vibration generated by the vibrator 43 is transmitted to the operation unit side body 2, the mobile phone 1 vibrates.

  The storage unit 44 is composed of, for example, a nonvolatile memory. The storage unit 44 includes a working memory and is used for arithmetic processing by the control unit 46. In addition, the storage unit 44 stores an address book 441 that stores an operating system (OS), a telephone number, a mail address, and the like, a plurality of application programs, various data files, and the like. The storage unit 44 may also serve as a removable external memory.

The timer 45 measures time based on an instruction from the control unit 46 and notifies the control unit 46 of the measured time.
The control unit 46 controls the entire mobile phone 1 and includes a central processing unit (CPU) and the like.
The control unit 46 also includes a notification control unit 461, an imaging control unit 462, a time measurement unit 463, a face recognition unit 464, a facial expression recognition unit 465, a call control unit 466, and a facial expression determination unit 467. .

Next, processing by the control unit 46 at the time of incoming call will be described.
When the communication control unit 461 receives a signal related to an incoming call by the communication unit 42, the notification control unit 461 refers to the address book 441 in the storage unit 44 and displays (notifies) information on the destination of the incoming call on the main display 21 or the sub display 35. . Further, the notification control unit 461 notifies an incoming call by turning on the LED 36, vibrating the vibrator 43, or outputting a sound from the speaker 22.
Here, the information regarding the destination of the incoming call is, for example, a name or an image associated with the incoming telephone number in the address book 441.

The imaging control unit 462 activates the main camera 33 or the sub camera 23 when receiving an incoming call by the communication unit 42. The imaging control unit 462 causes the main camera 33 or the sub camera 23 to image a subject (for example, a user's face).
Here, the imaging control unit 462 causes the main camera 33 to image the subject when the mobile phone 1 is in the closed state. On the other hand, when the mobile phone 1 is in the open state, the imaging control unit 462 causes the sub camera 23 to image the subject.

  The time measuring unit 463 causes the timer 45 to start measuring time at a predetermined timing. In addition, the time measuring unit 463 ends the time measurement after a predetermined time has elapsed after starting the time measurement. The operation of the time measuring unit 463 will be described in detail with reference to FIG.

  The face recognition unit 464 recognizes the face of a person from the image data of the subject imaged by the main camera 33 or the sub camera 23 according to the control of the imaging control unit 462. For example, the face recognition unit 464 recognizes a face region from image data by detecting feature points such as eyebrows, eyes, and mouth in the image data. Note that the face recognition method by the face recognition unit 464 is not limited to the above-described method, and other known methods can be used.

  The facial expression recognition unit 465 recognizes the facial expression of the person recognized by the face recognition unit 464. The facial expression recognition unit 465 detects, for example, feature points such as eyebrows, eyes, and mouth in the face area of the image data. Then, the facial expression recognition unit 465 recognizes the facial expression by obtaining a change in the relative position of the detected feature points. Note that the facial expression recognition method by the facial expression recognition unit 465 is not limited to the above-described method, and other known methods can be used.

  The call control unit 466 responds to the incoming call received by the communication unit 42 in a different manner for each facial expression based on the facial expression recognized by the facial expression recognition unit 465.

  As described above, according to the first embodiment, the mobile phone 1 can vary the incoming call response to the other party based on the facial expression imaged by the main camera 33 or the sub camera 23. In addition, the mobile phone 1 performs the above-described control using existing configurations such as the main camera 33, the sub camera 23, and the control unit 46. That is, the mobile phone 1 can more suitably perform a response according to the other party using a simple configuration.

Here, a specific example of a response by the call control unit 466 will be described.
The facial expression determination unit 467 determines whether or not the facial expression recognized by the facial expression recognition unit 465 described above is a first facial expression (for example, an angry facial expression).

  When the facial expression determination unit 467 determines that the facial expression is the first facial expression, the call control unit 466 connects to the external server (not shown) by the communication unit 42 and receives the incoming call received by the communication unit 42 In response to the first voice preset in the external server. Here, as the first voice, for example, a voice (message) for notifying the other party that the answering machine function has been established can be used.

  On the other hand, when the facial expression determination unit 467 determines that the facial expression is not the first facial expression, the call control unit 466 performs a hands-free call for the incoming call received by the communication unit 42.

  Thus, according to the first embodiment, when the recognized facial expression is the first facial expression, the mobile phone 1 responds with the first voice. For this reason, for example, when the other party displayed on the main display 21 or the sub display 35 is the other party who does not want to talk, the user captures the first facial expression with the main camera 33 or the sub camera 23. , You can not make a call with the other party you do not want to call. That is, the mobile phone 1 can more suitably perform a response according to the other party.

  When the mobile phone 1 determines that the facial expression is not the first facial expression, the mobile phone 1 performs a hands-free call in response to the incoming call received by the communication unit 42, so that the user is away from the mobile phone 1. Even when the user is in the telephone, the telephone call can be suitably performed.

  Next, processing executed by the control unit 46 according to the first embodiment will be described with reference to FIG. FIG. 4 is a flowchart illustrating processing executed by the control unit 46 according to the first embodiment.

  In step S <b> 1, when the notification control unit 461 receives a signal related to an incoming call by the communication unit 42, the notification control unit 461 refers to the address book 441 of the storage unit 44 and displays information on the destination of the incoming call on the main display 21 or the sub display 35. (Notify). Further, the notification control unit 461 notifies an incoming call by turning on the LED 36, vibrating the vibrator 43, or outputting a sound from the speaker 22.

  In step S <b> 2, the imaging control unit 462 activates the main camera 33 or the sub camera 23 when receiving an incoming call by the communication unit 42. The imaging control unit 462 causes the main camera 33 or the sub camera 23 to image a subject (for example, a user's face).

In step S3, the time measuring unit 463 determines whether or not the operation key group 11 has been operated. When the operation key group 11 is operated (YES), the process proceeds to step S12. If the operation key group 11 is not operated (NO), the process proceeds to step S4.
In step S <b> 4, the time measuring unit 463 causes the timer 45 to start measuring time.

In step S <b> 5, the face recognition unit 464 determines whether or not a person's face has been recognized in the subject image data captured by the main camera 33 or the sub camera 23. If a human face is recognized (YES), the process proceeds to step S7. If the face of the person is not recognized (NO), the process proceeds to step S6.
In step S6, the time measuring unit 463 determines whether or not the set time T has elapsed since the start of time measurement. If the set time T has elapsed (YES), the process proceeds to step S8. If the set time T has not elapsed (NO), the process returns to step S3.

  In step S7, the facial expression determination unit 467 determines whether or not the facial expression recognized by the facial expression recognition unit 465 is the first facial expression (for example, an angry facial expression). If it is the first facial expression (YES), the process proceeds to step S8. If it is not the first facial expression (NO), the process proceeds to step S10.

In step S8, the notification control unit 461 cancels the operation for notifying the incoming call. Further, the imaging control unit 462 stops the main camera 33 or the sub camera 23 activated in step S2.
In step S <b> 9, the call control unit 466 responds to the incoming call received by the communication unit 42 with a preset first voice.

In step S10, the notification control unit 461 cancels the operation for notifying the incoming call. Further, the imaging control unit 462 stops the main camera 33 or the sub camera 23 activated in step S2.
In step S <b> 11, the call control unit 466 performs a hands-free call for the incoming call received by the communication unit 42.

In step S12, the notification control unit 461 cancels the operation of notifying the incoming call. Further, the imaging control unit 462 stops the main camera 33 or the sub camera 23 activated in step S2.
In step S <b> 13, the call control unit 466 performs a normal call for the incoming call received by the communication unit 42.
In this way, the mobile phone 1 can vary the incoming call response to the other party based on the facial expression imaged by the main camera 33 or the sub camera 23.

  Hereinafter, another embodiment of the portable electronic device of the present invention will be described. In the other embodiments, points different from the first embodiment will be mainly described, the same reference numerals are given to the same configurations as those in the first embodiment, and the description will be omitted. The description of the first embodiment is appropriately applied to points that are not particularly described in the other embodiments.

Second Embodiment
Next, a second embodiment according to the portable electronic device of the present invention will be described.
The cellular phone 1 according to the second embodiment is different from the first embodiment in that the facial expression determination unit 467 determines whether or not it is the second facial expression as well as the first facial expression.
That is, the facial expression determination unit 467 determines whether or not the facial expression recognized by the facial expression recognition unit 465 is a second facial expression (for example, a smiling facial expression).

  Then, when the facial expression determination unit 467 determines that the facial expression is the second facial expression, the call control unit 466 connects to an external server (not shown) by the communication unit 42 and receives by the communication unit 42. The incoming call is answered with the second voice preset in the external server. Here, as the second voice, for example, a voice (message) that is different from the first voice and that informs the other party that the user of the mobile phone 1 cannot answer the call is used. be able to. Further, as the second voice, a voice (message) that tells that the user does not want to talk to the other party may be used.

  As described above, according to the second embodiment, when the recognized facial expression is the second facial expression, the mobile phone 1 responds with the second voice different from the first voice. For this reason, for example, when the other party displayed on the main display 21 or the sub display 35 is the other party who does not want to talk, the user captures the second facial expression with the main camera 33 or the sub camera 23. It is possible to respond to the other party with a second voice different from the first voice. And the mobile telephone 1 can differentiate the response with respect to an other party by making the 2nd audio | voice the voice (message) which tells that it does not want to call with the other party, for example.

  Next, processing executed by the control unit 46 according to the second embodiment will be described with reference to FIG. FIG. 5 is a flowchart illustrating processing executed by the control unit 46 according to the second embodiment. In addition, the process of step S21-step S26 is the same as the process of step S1-step S6 in 1st Embodiment, and the process of step S35 and step S36 is the same as the process of step S12 and step S13 in 1st Embodiment. It is the same.

  In step S <b> 21, when the notification control unit 461 receives a signal related to an incoming call by the communication unit 42, the notification control unit 461 refers to the address book 441 of the storage unit 44 and displays information about the destination of the incoming call on the main display 21 or the sub display 35. (Notify). Further, the notification control unit 461 notifies an incoming call by turning on the LED 36, vibrating the vibrator 43, or outputting a sound from the speaker 22.

  In step S <b> 22, the imaging control unit 462 activates the main camera 33 or the sub camera 23 when receiving an incoming call by the communication unit 42. The imaging control unit 462 causes the main camera 33 or the sub camera 23 to image a subject (for example, a user's face).

In step S23, the time measuring unit 463 determines whether or not the operation key group 11 has been operated. If the operation key group 11 is operated (YES), the process proceeds to step S35. If the operation key group 11 is not operated (NO), the process proceeds to step S24.
In step S <b> 24, the time measuring unit 463 causes the timer 45 to start measuring time.

In step S <b> 25, the face recognition unit 464 determines whether or not a person's face has been recognized in the subject image data captured by the main camera 33 or the sub camera 23. If a person's face is recognized (YES), the process proceeds to step S27. If the face of the person is not recognized (NO), the process proceeds to step S26.
In step S26, the time measurement unit 463 determines whether or not the set time T has elapsed since the start of time measurement. If the set time T has elapsed (YES), the process proceeds to step S29. If the set time T has not elapsed (NO), the process returns to step S23.

  In step S27, the facial expression determination unit 467 determines whether or not the facial expression recognized by the facial expression recognition unit 465 is a second facial expression (for example, a smiling facial expression). If it is the second facial expression (YES), the process proceeds to step S33. If it is not the second facial expression (NO), the process proceeds to step S28.

  In step S28, the facial expression determination unit 467 determines whether or not the facial expression recognized by the facial expression recognition unit 465 is the first facial expression (for example, an angry face). If it is the first facial expression (YES), the process proceeds to step S31. If it is not the first facial expression (NO), the process proceeds to step S29.

In step S29, the notification control unit 461 cancels the operation of notifying the incoming call. In addition, the imaging control unit 462 stops the main camera 33 or the sub camera 23 activated in step S22.
In step S <b> 30, the call control unit 466 responds to the incoming call received by the communication unit 42 with a preset first voice.

In step S31, the notification control unit 461 cancels the operation of notifying the incoming call. In addition, the imaging control unit 462 stops the main camera 33 or the sub camera 23 activated in step S22.
In step S <b> 32, the call control unit 466 responds to the incoming call received by the communication unit 42 with the second voice set in advance.

In step S33, the notification control unit 461 cancels the operation for notifying the incoming call. In addition, the imaging control unit 462 stops the main camera 33 or the sub camera 23 activated in step S22.
In step S <b> 34, the call control unit 466 performs a hands-free call for the incoming call received by the communication unit 42.
As described above, the mobile phone 1 can differentiate the response to the other party by using the second voice as a message notifying that the user does not want to talk to the other party rather than the first voice.

<Third Embodiment>
Next, a third embodiment of the portable electronic device of the invention will be described.
The mobile phone 1 according to the third embodiment is different from the second embodiment in that the notification control unit 461 performs notification of an incoming call twice in a different manner.

  In other words, when the communication control unit 461 accepts an incoming call, the notification control unit 461 causes the main display 21 as the notification unit and the LED 36 to notify the information on the other party in the first mode. Here, the information control part 461 displays the information regarding an other party on the main display 21, for example as a 1st aspect, and makes LED36 light.

  In addition, when the facial expression recognition unit 465 recognizes the facial expression, the notification control unit 461 uses the main display 21, the speaker 22, the LED 36, and the vibrator 43 as the notification unit to display information about the other party in a second mode different from the first mode. Let me know. Here, as a 2nd mode, information control part 461 displays information about an other party on main display 21, for example, makes LED36 light, and also outputs a sound from speaker 22, and vibrates vibrator 43.

  Thus, according to 3rd Embodiment, since the mobile telephone 1 alert | reports an incoming call twice in a different aspect and timing, it can call a user's attention to an incoming call suitably.

  Next, processing executed by the control unit 46 according to the third embodiment will be described with reference to FIG. FIG. 6 is a flowchart illustrating processing executed by the control unit 46 according to the third embodiment.

In step S <b> 41, the imaging control unit 462 activates the main camera 33 or the sub camera 23 when receiving an incoming call by the communication unit 42. The imaging control unit 462 causes the main camera 33 or the sub camera 23 to image a subject (for example, a user's face).
In step S <b> 42, the notification control unit 461 causes the main display 21 and the LED 36 to notify the information related to the other party that has received the incoming call in the first mode.

In step S43, the time measurement unit 463 determines whether or not the operation key group 11 has been operated. If the operation key group 11 is operated (YES), the process proceeds to step S59. If the operation key group 11 is not operated (NO), the process proceeds to step S44.
In step S <b> 44, the time measuring unit 463 causes the timer 45 to start measuring time.

  In step S45, the face recognition unit 464 determines whether or not a person's face has been recognized in the image data of the subject imaged by the main camera 33 or the sub camera 23. If a person's face is recognized (YES), the process proceeds to step S54. If the face of the person is not recognized (NO), the process proceeds to step S46.

  In step S46, the time measurement unit 463 determines whether or not the set time T has elapsed since the start of time measurement. If the set time T has elapsed (YES), the time measurement by the timer 45 is stopped and the process proceeds to step S47. If the set time T has not elapsed (NO), the process returns to step S43.

  In step S <b> 47, the notification control unit 461 causes the main display 21, the speaker 22, the LED 36, and the vibrator 43 to notify the second party of information related to the other party that has received the incoming call.

In step S48, the time measuring unit 463 determines whether or not the operation key group 11 has been operated. If the operation key group 11 is operated (YES), the process proceeds to step S60. If the operation key group 11 is not operated (NO), the process proceeds to step S49.
In step S49, the time measuring unit 463 resets the timer 45 and restarts the time measurement by the timer 45.

  In step S <b> 50, the face recognition unit 464 determines whether or not a person's face has been recognized in the subject image data captured by the main camera 33 or the sub camera 23. If a person's face is recognized (YES), the process proceeds to step S54. If the face of the person is not recognized (NO), the process proceeds to step S51.

  In step S51, the time measurement unit 463 determines whether or not the set time T has elapsed since the start of time measurement in step S49. If the set time T has elapsed (YES), the time measurement by the timer 45 is stopped and the process proceeds to step S52. If the set time T has not elapsed (NO), the process returns to step S48.

In step S52, the notification control unit 461 cancels the operation of notifying the incoming call. Further, the imaging control unit 462 stops the main camera 33 or the sub camera 23 activated in step S41.
In step S53, the call control unit 466 responds to the incoming call received by the communication unit 42 with a preset first voice.

  In step S54, the facial expression determination unit 467 determines whether or not the facial expression recognized by the facial expression recognition unit 465 is a second facial expression (for example, a smiling facial expression). If it is the second facial expression (YES), the process proceeds to step S57. If it is not the second facial expression (NO), the process proceeds to step S55.

In step S55, the notification control unit 461 cancels the operation of notifying the incoming call. Further, the imaging control unit 462 stops the main camera 33 or the sub camera 23 activated in step S41.
In step S <b> 56, the call control unit 466 responds to the incoming call received by the communication unit 42 with the second voice set in advance.

In step S57, the notification control unit 461 cancels the operation of notifying the incoming call. Further, the imaging control unit 462 stops the main camera 33 or the sub camera 23 activated in step S41.
In step S58, the call control unit 466 performs a hands-free call in response to the incoming call received by the communication unit 42.

In step S59, the notification control unit 461 cancels the operation of notifying the incoming call. Further, the imaging control unit 462 stops the main camera 33 or the sub camera 23 activated in step S41.
In step S <b> 60, the call control unit 466 performs a normal call for the incoming call received by the communication unit 42.
As described above, since the mobile phone 1 performs the notification of the incoming call twice with different modes and timings, it is possible to suitably alert the user to the incoming call.

<Fourth embodiment>
Next, a fourth embodiment according to the portable electronic device of the present invention will be described.
The cellular phone 1 according to the fourth embodiment is different from the third embodiment in that the facial expression determination method in the facial expression determination unit 467 is different.

That is, the facial expression recognition unit 465 recognizes the movement of a person along with facial expressions from image data captured by the main camera 33 or the sub camera 23.
FIG. 7 is a diagram illustrating an operation of a person recognized by the facial expression recognition unit 465. It should be noted that the facial expression recognition unit 465 may detect feature points from image data in the same manner as face recognition for human actions, or may use other known methods.

  The facial expression determination unit 467 is such that the facial expression recognized by the facial expression recognition unit 465 is a third facial expression (for example, an ordinary facial expression (see FIG. 7A)), and a human motion is a predetermined motion (for example, It is determined whether or not the operation is to raise one finger (see FIG. 7B). The predetermined operation may be, for example, an operation with the palm open as shown in FIG. 7C or another operation.

  If the facial expression determination unit 467 determines that the facial expression is the third facial expression and the person's movement is a predetermined movement, the call control unit 466 uses the communication unit 42 to perform an external server (not shown). To the incoming call received by the communication unit 42 and responds with a second voice preset in the external server.

  On the other hand, when the facial expression determination unit 467 determines that the facial expression is not the third facial expression and / or the person's action is not the predetermined action, the call control unit 466 receives the incoming call received by the communication unit 42. Make a hands-free call.

  As described above, according to the fourth embodiment, the mobile phone 1 performs not only facial expressions but also facial expressions using facial movements (gestures) together with facial expressions. Recognition can be performed appropriately.

  Next, processing executed by the control unit 46 according to the fourth embodiment will be described with reference to FIG. FIG. 8 is a flowchart illustrating processing executed by the control unit 46 according to the fourth embodiment. In addition, since the process of step S71-step S83, step S89, and step S90 is the same as the process of step S41-step S53, step S59, and step S60 in 3rd Embodiment, description is abbreviate | omitted.

  In step S84, the facial expression determination unit 467 determines whether or not the facial expression recognized by the facial expression recognition unit 465 is the third facial expression, and the action of the person is a predetermined action. If it is the third facial expression and it is a predetermined motion (YES), the process proceeds to step S85. If it is not the third facial expression and / or if it is not the predetermined action (NO), the process proceeds to step S87.

In step S85, the notification control unit 461 cancels the operation of notifying the incoming call. Further, the imaging control unit 462 stops the main camera 33 or the sub camera 23 activated in step S71.
In step S <b> 86, the call control unit 466 responds to the incoming call received by the communication unit 42 with the second voice set in advance.

In step S87, the notification control unit 461 cancels the operation of notifying the incoming call. Further, the imaging control unit 462 stops the main camera 33 or the sub camera 23 activated in step S71.
In step S88, the call control unit 466 performs a hands-free call for the incoming call received by the communication unit 42.
As described above, since the mobile phone 1 performs not only facial expressions but also facial expressions using facial movements (gestures), facial expressions (motions) can be appropriately recognized. .

<Fifth Embodiment>
Next, a fifth embodiment of the portable electronic device of the invention will be described.
The cellular phone 1 according to the fifth embodiment is mainly different from the third and fourth embodiments in that a facial expression is counted and a threshold is set for each facial expression counted.

  FIG. 9 is a block diagram showing a functional configuration of the mobile phone 1 according to the fifth embodiment. As shown in FIG. 9, the mobile phone 1 according to the fifth embodiment includes an expression count unit 468 and a threshold determination unit 469. In the mobile phone 1, the configuration of the address book 441 is different. Since other configurations are the same as those of the first embodiment, description thereof is omitted.

  Expression count unit 468 counts the number of times facial expression is recognized by expression recognition unit 465. The facial expression counting unit 468 stores the counted number in the address book 441 of the storage unit 44 for each facial expression.

  FIG. 10 is a diagram showing the configuration of the address book 441 according to the fifth embodiment. As shown in FIG. 10, the address book 441 stores the count number of the first facial expression and the count number of the second facial expression in association with the name, the telephone number, the mail address, and the like. In addition, the counts of the first facial expression and the second facial expression are counted and stored for each telephone number (name) by the facial expression counting unit 468. Further, the address book 441 stores a threshold value T1 (first threshold value) and a threshold value T2 (second threshold value) for the first facial expression count and the second facial expression count for each of the first facial expression and the second facial expression. Yes. The threshold values T1 and T2 may be the same value or different values. Further, the threshold values T1 and T2 may be different values for each telephone number.

  The threshold determination unit 469 determines whether or not the number of times the first facial expression (for example, an angry facial expression) has been recognized exceeds the threshold T1. Further, the threshold determination unit 469 determines whether or not the number of times the second facial expression (for example, a smiling facial expression) has been recognized exceeds the threshold T2.

  When the number of times the first facial expression is recognized exceeds the threshold T1, the call control unit 466 is connected to the external server by the communication unit 42, and is set in advance in the external server for an incoming call received by the communication unit 42. Answer with the second voice.

  As described above, according to the fifth embodiment, when the number of times the first facial expression is recognized exceeds the threshold value T1, the mobile phone 1 responds with the second voice. Can be determined appropriately.

  In addition, when the number of times the second facial expression is recognized exceeds the threshold T2, the call control unit 466 performs a hands-free call for the incoming call received by the communication unit 42.

  As described above, according to the fifth embodiment, when the number of times the second facial expression is recognized exceeds the threshold T2, the mobile phone 1 performs a hands-free call. It can be judged appropriately.

  Further, when the number of times the second facial expression is recognized does not exceed the threshold value T2 and the number of times the first facial expression is recognized does not exceed the threshold value T1, the call control unit 466 connects to the external server through the communication unit 42. In response to the incoming call received by the communication unit 42, the response is made with the first voice preset in the external server.

  Thus, according to the fifth embodiment, the mobile phone 1 determines that the number of times the second facial expression has been recognized does not exceed the threshold T2, and the number of times the first facial expression has been recognized does not exceed the threshold T1. Since the response is made with one voice, it is possible to appropriately determine the other party who should respond with the first voice.

  Next, processing executed by the control unit 46 of the fifth embodiment will be described with reference to FIGS. 11 and 12. FIG. 11 is a flowchart illustrating processing executed by the control unit 46 according to the fifth embodiment.

  In step S101, when a signal related to an incoming call is received by the communication unit 42, the threshold determination unit 469 refers to the address book 441 and determines whether or not the number of counts of the first facial expression exceeds the threshold T1. If the number of counts of the first facial expression exceeds the threshold value T1 (YES), the process proceeds to step S119. If the number of counts of the first facial expression exceeds the threshold value T1 (YES), the process proceeds to step S102.

In step S <b> 102, the imaging control unit 462 activates the main camera 33 or the sub camera 23 when receiving an incoming call by the communication unit 42. The imaging control unit 462 causes the main camera 33 or the sub camera 23 to image a subject (for example, a user's face).
In step S <b> 103, the notification control unit 461 causes the main display 21 and the LED 36 to notify the information related to the other party that received the incoming call in the first mode.

In step S104, the time measurement unit 463 determines whether or not the operation key group 11 has been operated. If the operation key group 11 is operated (YES), the process proceeds to step S122. If the operation key group 11 is not operated (NO), the process proceeds to step S105.
In step S <b> 105, the time measuring unit 463 causes the timer 45 to start measuring time.

  In step S <b> 106, the face recognition unit 464 determines whether or not a person's face has been recognized in the subject image data captured by the main camera 33 or the sub camera 23. If a person's face is recognized (YES), the process proceeds to step S115. If the person's face is not recognized (NO), the process proceeds to step S107.

  In step S107, the time measurement unit 463 determines whether or not the set time T has elapsed since the start of time measurement. If the set time T has elapsed (YES), the time measurement by the timer 45 is stopped and the process proceeds to step S108. If the set time T has not elapsed (NO), the process returns to step S104.

  In step S <b> 108, the notification control unit 461 causes the main display 21, the speaker 22, the LED 36, and the vibrator 43 to notify information related to the other party that has received the incoming call in the second mode.

In step S109, the time measurement unit 463 determines whether or not the operation key group 11 has been operated. When the operation key group 11 is operated (YES), the process proceeds to step S124. If the operation key group 11 is not operated (NO), the process proceeds to step S110.
In step S110, the time measuring unit 463 resets the timer 45 and restarts the time measurement by the timer 45.

  In step S <b> 111, the face recognition unit 464 determines whether or not a person's face has been recognized in the subject image data captured by the main camera 33 or the sub camera 23. If a person's face is recognized (YES), the process proceeds to step S115. If the person's face is not recognized (NO), the process proceeds to step S112.

  In step S112, the time measurement unit 463 determines whether or not the set time T has elapsed since the start of time measurement in step S110. If the set time T has elapsed (YES), the time measurement by the timer 45 is stopped and the process proceeds to step S113. If the set time T has not elapsed (NO), the process returns to step S109.

In step S113, the notification control unit 461 cancels the operation of notifying the incoming call. Further, the imaging control unit 462 stops the main camera 33 or the sub camera 23 activated in step S41.
In step S <b> 114, the call control unit 466 responds to the incoming call accepted by the communication unit 42 with a preset first voice.

  In step S115, the facial expression counting unit 468 counts the number of times facial expression is recognized by the facial expression recognition unit 465. The facial expression counting process for counting the number of times facial expressions are recognized will be described in detail with reference to FIG.

In step S116, the threshold determination unit 469 determines whether or not the number of times the first facial expression is recognized is equal to or greater than the threshold T1. If it is equal to or greater than the threshold value T1 (YES), the process proceeds to step S120. If it is less than the threshold value T1 (NO), the process moves to step S117.
In step S117, the threshold determination unit 469 determines whether or not the number of times the second facial expression is recognized is equal to or greater than the threshold T2. If it is equal to or greater than the threshold value T2 (YES), the process proceeds to step S118. If it is less than the threshold value T2 (NO), the process proceeds to step S113.

In step S118, the notification control unit 461 cancels the operation of notifying the incoming call. Further, the imaging control unit 462 stops the main camera 33 or the sub camera 23 activated in step S102.
In step S <b> 119, the call control unit 466 responds to the incoming call received by the communication unit 42 with the preset second voice.

In step S120, the notification control unit 461 cancels the operation of notifying the incoming call. Further, the imaging control unit 462 stops the main camera 33 or the sub camera 23 activated in step S102.
In step S <b> 121, the call control unit 466 performs a hands-free call for the incoming call received by the communication unit 42.

  In step S122, the facial expression counting unit 468 counts the number of times facial expression is recognized by the facial expression recognition unit 465. The facial expression counting process for counting the number of times facial expressions are recognized will be described in detail with reference to FIG.

In step S123, the notification control unit 461 cancels the operation of notifying the incoming call. Further, the imaging control unit 462 stops the main camera 33 or the sub camera 23 activated in step S102.
In step S <b> 124, the call control unit 466 performs a normal call for the incoming call received by the communication unit 42.

FIG. 12 is a flowchart showing the facial expression counting process in steps S115 and S122 of FIG.
In step S131, the facial expression counting unit 468 determines whether or not the facial expression recognized by the facial expression recognition unit 465 is the first facial expression. If it is the first facial expression (YES), the process proceeds to step S135. If it is not the first facial expression (NO), the process proceeds to step S132.

  In step S132, the facial expression counting unit 468 determines whether or not the facial expression recognized by the facial expression recognition unit 465 is the second facial expression. If it is the second facial expression (YES), the process proceeds to step S133. If it is not the second facial expression (NO), the process is terminated.

In step S <b> 133, facial expression counting section 468 reads from address book 441 the number of times the second facial expression associated with the telephone number of the other party that accepted the incoming call has been counted.
In step S134, the facial expression counting unit 468 increments the number of times the read second facial expression is counted by 1 (+1) and overwrites (stores) it in the address book 441.

In step S <b> 135, facial expression counting section 468 reads from address book 441 the number of times the first facial expression associated with the telephone number of the other party that accepted the incoming call has been counted.
In step S136, the facial expression counting unit 468 increments (+1) the number of times the read first facial expression is counted by 1 and overwrites (stores) it in the address book 441.

  The facial expression counting process for counting facial expressions described above is not limited to the process executed in steps S115 and S122 in FIG. 11, and may be executed after the call is ended, for example. FIG. 13 is a flowchart showing another form of facial expression count processing.

In step S141, the call control unit 466 executes processing for terminating the call with the other party.
In step S142, the imaging control unit 462 activates the main camera 33 or the sub camera 23. The imaging control unit 462 causes the main camera 33 or the sub camera 23 to image a subject (for example, a user's face).

  In step S143, the facial expression counting unit 468 counts the number of times facial expression is recognized by the facial expression recognition unit 465, as described in FIG.

  As mentioned above, although embodiment of this invention was described, this invention is not restrict | limited to embodiment mentioned above, It can change suitably.

  In the above-described embodiment, the mobile phone 1 as a mobile electronic device has been described. However, the present invention is not limited to this, and is not limited to this. It may be a device, a personal computer, a notebook computer, a portable game device, or the like.

1 Mobile phone (mobile electronic device)
21 Main display (notification part)
23 Sub camera (imaging part)
33 Main camera (imaging part)
34 LED (notification part)
35 Sub-display (notification part)
43 Vibrator (notification part)
462 Imaging control unit 465 Facial expression recognition unit

Claims (11)

An imaging unit;
A communication unit for receiving a signal related to an incoming call;
An informing unit for informing information on a destination that has received an incoming call by the communication unit;
When an incoming call is received by the communication unit, an imaging control unit that images the subject by the imaging unit;
A facial expression recognition unit for recognizing the facial expression of a person's face in the image data of the subject imaged according to the control of the imaging control unit;
A portable electronic device comprising: a call control unit that responds in a different manner to the incoming call received by the communication unit based on the facial expression recognized by the facial expression recognition unit. A facial expression determination unit for determining whether or not the facial expression recognized by the facial expression recognition unit is a first facial expression;
When the facial expression determination unit determines that the facial expression is the first facial expression, the call control unit responds with the first voice set in advance to the incoming call received by the communication unit The portable electronic device according to claim 1, wherein when the facial expression determination unit determines that the facial expression is not the first facial expression, the mobile electronic device makes a call to the incoming call received by the communication unit.   The portable electronic device according to claim 2, wherein the call is a hands-free call. The facial expression determination unit determines whether or not the facial expression recognized by the facial expression recognition unit is a second facial expression;
When the facial expression determination unit determines that the facial expression is the second facial expression, the call control unit responds with the preset second voice to the incoming call received by the communication unit The portable electronic device according to claim 2.   When an incoming call is received by the communication unit, information on the destination is notified in the first mode by the notification unit, and when the facial expression is recognized by the facial expression recognition unit, information on the destination is updated by the notification unit. The portable electronic device according to claim 1, further comprising a notification control unit that performs notification in two modes. The facial expression recognition unit recognizes the movement of a person along with the facial expression from the image data captured by the imaging unit,
The facial expression determination unit determines whether the facial expression recognized by the facial expression recognition unit is a third facial expression, and whether or not the action of the person is a predetermined action;
The call control unit receives the incoming call received by the communication unit when the facial expression is a third facial expression and the facial expression determination unit determines that the action of the person is the predetermined action. In response to the second voice, and when the facial expression determination unit determines that the facial expression is not the third facial expression or the person's movement is not a predetermined movement, the communication unit The portable electronic device according to claim 4, wherein a call is made with respect to the received incoming call.   The portable electronic device according to claim 6, wherein the call is a hands-free call. A count unit for counting the number of times the facial expression is recognized by the facial expression recognition unit;
A storage unit that stores the number of times counted by the counting unit for each facial expression;
A first threshold value determination unit for determining whether or not the number of times the first facial expression is recognized exceeds a first threshold value;
A second threshold determination unit that determines whether or not the number of times the second facial expression is recognized exceeds a second threshold;
The call control unit responds to the incoming call received by the communication unit with a preset second voice when the number of times the first facial expression is recognized exceeds the first threshold. The portable electronic device as described in.   The portable electronic device according to claim 8, wherein the call control unit performs a hands-free call for the incoming call received by the communication unit when the number of times the second facial expression is recognized exceeds the second threshold. machine.   When the number of times the second facial expression is recognized does not exceed the second threshold value and the number of times the first facial expression is recognized does not exceed the first threshold value, the call control unit is accepted by the communication unit. The portable electronic device according to claim 8 or 9, which responds to the incoming call with a preset first voice. An imaging unit;
A communication unit that accepts incoming calls;
An informing unit for informing information on a destination that has received an incoming call by the communication unit;
When the incoming call is accepted, the subject is imaged, the face of the person is recognized from the image data of the taken subject, the recognized facial expression is recognized, and the received incoming call is recognized based on the recognized facial expression. A portable electronic device comprising a control unit that responds in a different manner.

Images