JP2014149554A - Information processing apparatus, information processing method and information processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing apparatus capable of supporting a user's operation to register a piece of unregistered information including an image of a person related to the user in a telephone directory and the like.SOLUTION: The information processing apparatus includes: a storage 33 storing images; an address directory 18 that stores an address and an image of a person corresponding to the address; a mail analysis section 20 that extracts an address which is not registered in the address directory 18 from a mail; a search processing section 30 that estimates an image of a person corresponding to the unregistered address extracted from the images stored in the storage 33; and a display unit 13 that displays the extracted unregistered address, the image of the estimated person and a piece of registration recommendation information to recommend to register the unregistered address and the image of the person in the address directory 18.

Description

  The present invention relates to an information processing apparatus, an information processing method, and an information processing program. In particular, the present invention relates to a technique for managing a database such as a telephone directory mounted on a portable terminal or the like.

  Devices such as mobile phone terminals and portable information terminals are often equipped with a database function such as a phone book (address book). In other words, information such as the other party's telephone number and the other party's mail address that represents the destination of the e-mail are stored in the phone book database as needed to connect the line for voice calls with the other party's terminal. It is managed.

  In such a telephone directory database, information such as telephone numbers and e-mail addresses is generally managed in association with individual names, company names, etc. for each destination. In addition, in order to facilitate identification of the other party, not only character information such as a personal name and address but also an image such as the face of the corresponding person may be simultaneously registered in the telephone directory database.

  Regarding the contents of registration in such a phone book, in general, a user of a device such as a mobile phone terminal manually inputs information for each destination as necessary, and performs a predetermined registration operation. Can be stored in a database. Therefore, it is necessary for the user to repeat a troublesome input operation over time until information sufficient for the user is accumulated in the telephone directory of the terminal.

  As techniques for performing data management based on a telephone directory database, the following patent documents 1 and 2 are known.

  Patent Document 1 discloses a technique for registering an image of a person in a telephone directory database in an apparatus equipped with a videophone communication function. Specifically, during a videophone call, the telephone directory is searched based on the telephone number of the other party, and it is identified whether the image of the corresponding person has been registered. If not registered, the received person image of the other party is taken and registered in the phone book.

  Patent Document 2 discloses a technique for finely protecting a mail address in an apparatus that handles broadcast mail. Specifically, when a broadcast mail is received, it is identified whether or not the destination mail address described in the header part of the mail is registered in the address book, and the unregistered mail address is assigned to the header part. Remove automatically from.

JP 2007-329632 A JP 2010-271754 A

  However, in the technique of Patent Document 1, an image of a person of a communication partner who is not registered in the telephone book can be automatically registered only when videophone communication is actually performed. Therefore, when performing only general voice calls and e-mail exchanges without image transmission / reception, images of persons not registered in the phone book must be individually registered manually by the user, It takes time and effort. In particular, when an image is registered in the phone book, unlike text information, image information cannot be input by a simple button operation, so the registration work is often very troublesome for the user.

In the technique of Patent Document 2, it is automatically identified whether or not the mail address included in the received broadcast mail is registered in the address book. It is not possible to perform such an operation. In addition, even if an unregistered person is an acquaintance, the corresponding email address will be deleted from the received broadcast email,
The received information cannot be used to enrich the contents of the address book, and any exchanges with the acquaintance will be blocked.

  The present invention relates to an information processing apparatus, an information processing method, and an information processing program capable of supporting a user's input operation for registering unregistered information including an image of a person related to the user in a telephone directory or the like. The purpose is to provide.

  An information processing apparatus according to the present invention includes an image storage unit that stores an image, an address storage unit that stores an address and an image of a person corresponding to the address, and an unregistered address that is not stored in the address storage unit. An information extraction unit that extracts from the image stored in the image storage unit, a person image estimation unit that estimates an image of a person corresponding to an unregistered address extracted by the information extraction unit, and the information extraction unit The extracted unregistered address, the person image estimated by the person image estimation unit, and the recommended registration information for prompting the user to register the unregistered address and the person image in the address storage unit are displayed. A display unit.

  According to this configuration, it is possible to support a user operation for registering unregistered information including a person image in the address storage unit. That is, since the person image estimation unit automatically estimates the image of the person corresponding to the unregistered address, the user can register the unregistered information with a simple operation in accordance with the recommended registration information displayed.

  The information processing method of the present invention includes a step of extracting an unregistered address that is not stored in an address storage unit that stores an address and an image of a person corresponding to the address from an email, and an image storage unit that stores the image Estimating an image of a person corresponding to the extracted unregistered address from the stored image, the extracted unregistered address, the image of the estimated person, the unregistered address, and the Displaying registration recommendation information for urging the user to register an image of a person in the address storage unit.

  According to this method, it is possible to support a user operation for registering unregistered information including an image of a person in the address storage unit. That is, since the image of the person corresponding to the unregistered address is automatically estimated, the user can register the unregistered information with a simple operation according to the displayed recommended registration information.

  Moreover, the information processing program of this invention is a program for making a computer perform each step of the said information processing method.

  According to this program, it is possible to support a user operation for registering unregistered information including an image of a person in the address storage unit. That is, since the image of the person corresponding to the unregistered address is automatically estimated, the user can register the unregistered information with a simple operation according to the displayed recommended registration information.

  The present invention can support a user's input operation for registering unregistered information including an image of a person related to the user in a telephone directory or the like.

The block diagram which shows the structural example of the principal part of the information processing apparatus in embodiment of this invention. The flowchart which shows the operation example at the time of registration assistance of the unregistered person information by the information processing apparatus in the embodiment of the present invention The flowchart which shows the operation example at the time of registration assistance of the unregistered person information based on each trigger by the information processing apparatus in the embodiment of the present invention (A)-(D) The figure for demonstrating estimation of the unregistered face image based on the extracted image in embodiment of this invention (A)-(D) The figure which shows the example of a screen display of the information processing apparatus in embodiment of this invention.

  Embodiments of the present invention will be described below with reference to the drawings.

  A configuration example of a main part related to the information processing apparatus of this embodiment is shown in FIG. As the information processing apparatus shown in FIG. 1, devices such as a mobile phone terminal, a portable information terminal, and a personal computer are assumed.

  As illustrated in FIG. 1, the information processing apparatus 1 includes a communication unit 11, an operation unit 12, a display unit 13, an e-mail receiving unit 14, an image capturing processing unit 15, a person authentication processing unit 16, a key input processing unit 17, an address. A book 18, an address book management unit 19, a mail analysis unit 20, a search processing unit 30, and a storage unit 33 are configured.

  The mail analysis unit 20 includes a header information analysis unit 21 and a text analysis unit 22. The search processing unit 30 includes a still image data search processing unit 31 and a moving image data search processing unit 32. The storage unit 33 can store a moving image data file 34 and a still image data file 35. The moving image data file 34 and the still image data file 35 are not stored in the storage unit 33 in advance, but may be acquired from a server on the WEB via the communication unit 11 and stored in the storage unit 33, for example. .

  Note that each function of the e-mail receiving unit 14, the image capturing processing unit 15, the person authentication processing unit 16, the key input processing unit 17, the address book management unit 19, the mail analysis unit 20, and the search processing unit 30 illustrated in FIG. This can be realized by a computer (not shown) that controls the information processing apparatus 1 executing a predetermined program. Of course, it is also possible to replace with dedicated hardware that realizes the same function. In addition, the present embodiment can be implemented in a state in which some components of each function shown in FIG. 1 are omitted.

  The communication unit 11 is connected to a network such as the Internet via a predetermined communication line, and performs data communication including exchange of e-mail information with other devices such as a predetermined server.

  The operation unit 12 includes various buttons and keys that accept user input operations. The display unit 13 has a display screen composed of a liquid crystal or the like, and displays various character strings, figures, images, etc. on the screen as visible information. Note that the operation unit 12 may be configured as a transparent touch panel arranged on the screen of the display unit 13.

  The e-mail receiving unit 14 receives an e-mail addressed to the user of the own station transmitted from another device from a predetermined communication network via the communication unit 11. Note that the e-mail handled by the e-mail receiving unit 14 is configured in accordance with existing standards. That is, it includes a header including information such as a mail address and a title representing each of the destination and the transmission source, and text information representing the body of the mail.

  The data of each e-mail received by the e-mail receiving unit 14 is stored as a received mail file 23 on a predetermined storage device. In addition, every time an e-mail receiving unit 14 receives one e-mail, the e-mail receiving unit 14 outputs a notification indicating an incoming mail to the mail analyzing unit 20.

  The image capturing processing unit 15 captures an image when an instruction is given by a user input operation. Although not shown, the information processing apparatus 1 in FIG. 1 is equipped with a camera. Using this camera, the image capturing processing unit 15 captures a necessary image as a still image or a moving image.

  The moving image data captured by the image capturing processing unit 15 is stored in the storage unit 33 as a moving image data file 34. Further, the image data of the still image captured by the image capturing processing unit 15 is stored in the storage unit 33 as a still image data file 35. That is, the storage unit 33 has a function as an image storage unit that stores images. In addition, when the shooting of the still image or moving image is completed, the image shooting processing unit 15 outputs a notification indicating the shooting completion to the mail analysis unit 20.

  The person authentication processing unit 16 performs image recognition processing on the data of the moving image data file 34 and the still image data file 35 stored in the storage unit 33 when the start of authentication is instructed by a user input operation or the like. Perform authentication processing. Specifically, a person present in each image is detected, and each individual is authenticated from features such as the face of each person. When the face authentication is completed, the person authentication processing unit 16 outputs a notification of face authentication completion to the mail analysis unit 20. Note that person authentication may be performed by a method other than face authentication.

  The address book 18 is a database that holds personal information of various persons such as friends and acquaintances of the user of the information processing apparatus 1. Specifically, information such as an image including a name, an e-mail address, a telephone number, and a face, an address, and the like are held in a state of being associated with each other. That is, the address book 18 has a function as an address storage unit that stores an address and an image of a person corresponding to the address.

  The address book management unit 19 manages the registered contents of the address book 18. The information held in the address book 18 is personal information, and nothing is registered in the address book 18 in the initial state. For example, when the user performs a predetermined input operation on the operation unit 12, necessary personal information is input, and the personal information is registered in the address book 18. The information processing apparatus 1 has a function for supporting a registration operation of personal information in the address book 18 for a person who is not registered in the address book 18. This function will be described in detail later.

  The key input processing unit 17 executes a process for accepting a user input operation on the operation unit 12, and gives an appropriate instruction to each unit according to the operation mode of the information processing apparatus 1 and the display state of the operation unit 12.

  The mail analysis unit 20 analyzes the contents of the received mail file 23 when a predetermined trigger occurs. For example, the destination determination unit 21a of the header information analysis unit 21 analyzes the data in the header area for each received e-mail, and is designated as a destination (destination other than the user) or a source (sender). The e-mail addresses that are present. That is, the mail analysis unit 20 has a function as an information extraction unit that extracts unregistered addresses that are not registered in the address book 18 from mail. In addition, the telephone book registration address determination unit 21b identifies whether or not the person information of each mail address extracted by the destination determination unit 21a has been registered in the address book 18. The conditions for generating the trigger will be described later.

  The search processing unit 30 stores the face image of the person corresponding to the unregistered address (the mail address of the person not registered in the address book 18) detected by the mail analysis unit 20 in the image data stored on the storage unit 33. Extract from inside. The still image data search processing unit 31 searches the still image data file 35 to detect a corresponding face image. In addition, the moving image data search processing unit 32 searches the moving image data file 34 to detect a corresponding face image. The search method will be described later. The person's face image is an example of a person's image, and may be an image of a part of another person or an image of the entire person.

  When a predetermined trigger occurs and the mail analysis unit 20 detects an unregistered address from the received mail file 23, the display unit 13 displays the unregistered address, the face image of the person, and recommended registration information. Display on the screen. The face image of the person is an extraction result of the search processing unit 30. The recommended registration information is information for prompting the user to register the information of the person (face image or the like) in the address book 18.

  Next, an operation example of the information processing apparatus 1 will be described.

  An example of the operation of the information processing apparatus 1 for supporting the registration of personal information of an unregistered person in the address book 18 is shown in FIGS. First, the operation shown in FIG. 2 will be described.

  In step S11, the mail analysis unit 20 extracts the mail address of the destination and the sender from the header area of the received mail file 23 to be processed, and whether or not one or more mail addresses registered in the address book 18 exist. Identify

  If there is no registered mail address in the address book 18, the process proceeds from step S11 to S12. If there is a registered mail address in the address book 18, the process proceeds to step S15 or S12 depending on the selected operation mode. That is, in the operation mode in which the face image of an unregistered person is estimated using the registered face image, the process proceeds to step S15. In the case of the operation mode in which the face image of an unregistered person is estimated using time information or location information, the process proceeds to step S12.

  In step S12, the text analysis unit 22 analyzes the content of the text of the received mail file 23 to be processed, and information indicating the time (time information) or information indicating the location (location information) is included in the text. ) Is present or not. When time or place (position) information is included, the text analysis unit 22 extracts the information. If neither time nor place information exists, the face image of an unregistered person cannot be estimated, and the processing in FIG. 2 is terminated. If the time information is included in the mail body, the process proceeds to step S13. If the place information is included, the process proceeds to step S14.

  In step S13, the search processing unit 30 refers to each moving image of the moving image data file 34 and each image data of the still image data file 35 held in the storage unit 33, and specifies a specific moving image corresponding to an unregistered person. Alternatively, a still image is extracted. Specifically, the mail analysis unit 20 refers to time information included in the mail text of the received mail file 23 in which an unregistered person is detected, and extracts an image whose shooting time is similar to this time information. To do. For example, an image captured when the time difference between the capturing time and this time information is within a predetermined time is extracted, or images having different dates are extracted at approximately the same time in order to distinguish only by the difference in date.

  In step S14, the search processing unit 30 refers to the moving image data in the moving image data file 34 and the still image data file 35 held in the storage unit 33, and performs a specific moving image or still image corresponding to an unregistered person. Extract a stroke. Specifically, the mail analysis unit 20 refers to the location information included in the email text of the received email file 23 in which an unregistered person is detected, and extracts an image whose shooting location is similar to this location information. To do. For example, an image taken at a point where the distance between the shooting location and the shooting information is within a predetermined distance is extracted, or an image having a similar shooting location name is extracted.

  In step S15, the search processing unit 30 refers to the data of each image in the moving image data file 34 and the still image data file 35 held in the storage unit 33, and a specific moving image or still image corresponding to an unregistered person. To extract. Specifically, the person authentication processing unit 16 compares the characteristics of the face and the like included in each image with the characteristics of the face of the registered person registered in the address book 18, and the registered person is included. Extract still images or movies.

  In step S16, if the search processing unit 30 can extract a moving image identifying whether or not a still image or moving image satisfying the condition has been extracted in any of steps S13, S14, and S15, step S17 is performed. If the still image can be extracted, the process proceeds to step S18. If neither the moving image nor the still image can be extracted, the process of FIG. 2 ends.

  In step S17, the moving image data search processing unit 32 estimates a face image corresponding to an unregistered person based on the moving image data extracted in any of steps S13, S14, and S15. A specific example of this process will be described later.

  In step S18, the still image data search processing unit 31 estimates a face image corresponding to an unregistered person based on the still image data extracted in any of steps S13, S14, and S15. A specific example of this process will be described later.

  As described above, the search processing unit 30 has a function as a person image estimation unit that estimates an image of a person corresponding to an unregistered address extracted from the image stored in the storage unit 33 by the mail analysis unit 20.

  In step S19, it is identified whether or not a face image corresponding to an unregistered person could be estimated in step S17 or S18. If it can be estimated, the process proceeds to step S20. If it cannot be estimated, the process of FIG. 2 is terminated.

  In step S20, the display unit 13 prompts the user to register the face image of the unregistered person estimated in step S17 or S18, the unregistered mail address included in the mail, and the information. The recommended registration information is displayed on the screen.

  In step S21, the key input processing unit 17 receives a user input operation for the screen display in step S20, and identifies whether or not to execute registration according to the content of the input operation. If an operation for executing registration is detected, the process proceeds to the next step S22, and if not registered, the process of FIG.

  In step S <b> 22, the address book management unit 19 registers information on an unregistered person including a face image extracted by estimation in the address book 18 in accordance with a user input operation. When a plurality of face images are extracted as candidates, the address book management unit 19 registers a specific face image selected by the user via the operation unit 12 or the like in the address book 18 from the plurality of candidates. May be.

  As described above, the search processing unit 30 has not yet obtained from images captured at a time having a predetermined similarity with the time information extracted by the mail analysis unit 20 among the images stored in the storage unit 33. The person image corresponding to the registered address is estimated. This makes it possible to extract an image that is highly likely to match the person to be registered.

  In addition, the search processing unit 30 extracts an unregistered address from images captured at a position having a predetermined similarity with the position information extracted by the mail analysis unit 20 among the images stored in the storage unit 33. An image of a person corresponding to is estimated. This makes it possible to extract an image that is highly likely to match the person to be registered.

  In addition, the search processing unit 30 selects an image of the person corresponding to the unregistered address from the images including the person corresponding to the registered address extracted by the mail analysis unit 20 among the images stored in the storage unit 33. presume. This makes it possible to automatically extract an image that is highly likely to match the person to be registered.

  Next, the operation shown in FIG. 3 will be described. In the present embodiment, a trigger for supporting registration of information on an unregistered person occurs in various situations, and the processing shown in FIG. 2 is executed each time. Specifically, when the e-mail receiving unit 14 notifies the mail analyzing unit 20 of an incoming mail, when the image capturing processing unit 15 notifies the mail analyzing unit 20 of completion of capturing, and when the person authentication processing unit 16 A trigger is generated when the mail analysis unit 20 is notified of the completion of authentication.

  In step S31, when the e-mail receiving unit 14 receives a new mail, the mail analyzing unit 20 is notified of the incoming mail. A trigger is generated by this notification, and the process proceeds to step S34.

  In step S <b> 32, the image capturing processing unit 15 notifies the mail analyzing unit 20 of the completion of shooting when shooting of a moving image or a still image is completed according to a user instruction via the operation unit 12 or the like. When a trigger is generated by this notification, the process proceeds to step S36.

  In step S33, when the person authentication processing unit 16 completes authentication of a face or the like for a person included in each image of the moving image data file 34 or the still image data file 35 held in the storage unit 33, face authentication is completed. Is sent to the mail analysis unit 20. When a trigger is generated by this notification, the process proceeds to step S36.

  In step S34, the mail analysis unit 20 analyzes the data of the received mail file 23, and includes an unregistered address that is not registered in the address book 18 as one of the destination and sender mail addresses. Identify whether or not. If an unregistered address is included, the process proceeds to the next step S35, and if not included, the process of FIG. Note that the received mail file 23 to be processed in step S34 may be limited to only the mail received last by the e-mail receiving unit 14, or other mail may be processed simultaneously.

  In step S35, the information processing apparatus 1 extracts an unregistered address from the content of the corresponding received mail file 23, and performs an operation for registering the mail address and the estimated face image in the address book 18. Provide support. That is, the process of FIG. 2 already described is executed in step S35.

  In step S36, the mail analysis unit 20 refers to the contents of each mail header in order from the mail inbox in which the received mail file 23 received in the past is stored, and selects either the destination or the sender. Search for e-mail addresses that contain unregistered addresses.

  In step S37, it is identified whether or not there is a mail including an unregistered address. If there is a mail including an unregistered address, the process proceeds to step S35, and if not, the process of FIG.

  As described above, when the e-mail receiving unit 14 receives a mail, the search processing unit 30 estimates a person image corresponding to the unregistered address. Accordingly, the user can perform a registration operation regarding an unregistered person at a timing triggered by reception of the mail.

  In addition, when the image capturing processing unit 15 completes image capturing, the search processing unit 30 estimates a person image corresponding to an unregistered address. Thereby, the user can perform a registration operation regarding an unregistered person at a timing triggered by the completion of image capturing.

  In addition, when the face authentication process is completed by the person authentication processing unit 16, the search processing unit 30 estimates a person image corresponding to an unregistered address. Thus, the user can perform a registration operation regarding an unregistered person at a timing triggered by the completion of the authentication process for the face in the image.

  Next, a method for estimating a person's face image corresponding to an unregistered address will be described.

  As for processing for estimating the face image of the person corresponding to the unregistered address, the search processing unit 30 can perform search by applying various types of search conditions (judgment criteria) as described below. Conceivable. In addition, it is possible to estimate a more appropriate face image by using independent search conditions for extracting a corresponding face image from a still image and for extracting a corresponding face image from a moving image.

<Specific examples of criteria for estimation from video>
(Judgment criteria A: Range is specified by the number of frames or time)
Among a number of frames arranged in chronological order constituting a moving image, a frame in which a face image of a person registered in the address book 18 appears as a reference frame, and a frame group within a predetermined range before and after the reference frame is set as a search target. To do. The search target range is defined by the number of frames or time. Then, when a face image of an unregistered person is detected in the frame group within the predetermined range, the unregistered face image is estimated as a face image of a person corresponding to the unregistered address.

  For example, when the frame rate at the time of moving image shooting is 30 frames / second, it is within the range of 3 seconds before and after the reference frame, that is, within the range from the position of 90 frames ahead of the reference frame to the position of 90 frames behind the reference frame. An unregistered face image that appears is estimated as a face image of an unregistered address.

  As described above, the search processing unit 30 adds the address book to frames within a predetermined number of frames before and after the frame including the person corresponding to the registered address stored in the address book 18 in the moving image stored in the storage unit 33. When an unregistered person image that is an image of another person not stored in 18 is included, the unregistered person image is estimated to be an image of a person corresponding to an unregistered address. As a result, if an unregistered person appears in a frame close to the frame in which a registered person appears in the moving image, it is highly likely that the unregistered person is the user's acquaintance. It can be estimated as a person.

(Criteria B: Number of appearances specified)
A frame in which a face image of a person registered in the address book 18 and an unregistered face image appear at the same time is searched from among a large number of frames constituting the moving image. Then, the number of frames in which the same unregistered face image appears together with the registered person is counted as the number of appearances. An unregistered face image whose appearance count is equal to or greater than a predetermined value is estimated as a face image of a person corresponding to an unregistered address.

  For example, when the number of appearances is 3, the unregistered face image is detected when three or more frames in which a registered person's face image and the same unregistered face image appear simultaneously are detected. Is estimated as a face image of an unregistered address.

  As described above, the search processing unit 30 includes a person who corresponds to the registered address stored in the address book 18 in the moving image stored in the storage unit 33 and is not stored in the address book 18. When there are a predetermined number or more of frames including unregistered person images that are images of the person, it is estimated that the unregistered person images are images of persons corresponding to unregistered addresses. As a result, if there are a large number of moving image frames in which a registered person and an unregistered person appear, it is highly likely that the person is an acquaintance of the user, so the unregistered person is estimated as a person to be registered. Can do.

  For example, if a registered person's face image and an unregistered face image appear simultaneously in the first frame to the fourth frame, the seventh frame to the tenth frame, and the fifteenth frame to the seventeenth frame, they are consecutive. A frame (frame group) may be the number of appearances described above. In this case, the number of appearances is three. Each frame may be the number of appearances described above. In this case, the number of appearances is 10.

(Criteria C: Specify the number of files)
When there are a plurality of moving image files that record moving image data stored in the storage unit 33, a moving image file that includes both face images of persons registered in the address book 18 and unregistered face images. Search for. Then, when the number of moving image files in which a registered person and the same unregistered face image appear is greater than or equal to a specified value, the detected unregistered face image is estimated as the face image of the person corresponding to the unregistered address To do.

  For example, when the prescribed value of the number of files is 3, an unregistered face image is detected when three or more moving image files in which a registered person's face image and an unregistered same face image appear are detected. Is estimated as a face image of an unregistered address.

  As described above, the search processing unit 30 includes a person corresponding to the registered address stored in the address book 18 among the moving images stored in the storage unit 33 and is not stored in the address book 18. When there are a predetermined number or more of moving image files including unregistered person images that are images of other persons, the unregistered person images are estimated to be images of persons corresponding to unregistered addresses. As a result, if there are a large number of moving image files in which registered persons and unregistered persons appear, there is a high possibility that the person is an acquaintance of the user. Can be estimated.

(Judgment criterion D: Appearance time specified)
A frame in which a face image of a person registered in the address book 18 and an unregistered face image appear at the same time is searched from among a large number of frames constituting the moving image. Then, for a time-series frame group in which the same unregistered face image appears together with the registered person, when the length of time that the unregistered same face image continuously appears is equal to or greater than a predetermined value, An unregistered face image is estimated as a face image of a person corresponding to an unregistered address.

  For example, when the specified value of the appearance time is 3 seconds, the same unregistered face image is continuously displayed in the frame group in which the registered person face image and the unregistered same face image appear simultaneously. When the length of time that has appeared in is 3 seconds or more, an unregistered face image is estimated as a face image of an unregistered address.

  As described above, the search processing unit 30 is stored in the address book 18 in a frame group including a person corresponding to the registered address stored in the address book 18 in the moving image stored in the storage unit 33. If there is a frame including an unregistered person image that is an image of another person who is not present for a predetermined time or more, the unregistered person image is estimated to be an image of a person corresponding to an unregistered address. As a result, in the case where a registered person and an unregistered person appear in a moving image where the unregistered person appears for a relatively long time, it is highly possible that the person is an acquaintance of the user. A person can be estimated as a person to be registered.

<Specific examples of criteria for estimation from still images>
(Judgment criteria E: Specify the distance between people)
When a face image of a person registered in the address book 18 and an unregistered face image appear at the same time in one still image, the space between the registered face image and the unregistered face image Detect the distance. If this distance is less than or equal to the specified value, an unregistered face image is estimated as the face image of the person corresponding to the unregistered address.

  For example, as shown in FIG. 4A, when the distance between the person P1 at the registered address and the unregistered face image Q1 is short, the two persons are likely to be acquaintances with each other, and are not registered. Since there is a high possibility that the person of the face image Q1 is an acquaintance of the user, the unregistered face image Q1 can be estimated as a face image of an unregistered address. On the other hand, as shown in FIG. 4D, when the distance between the person P1 at the registered address and the unregistered face image Q1 is far away, there is a possibility that both persons are not acquaintances. Therefore, it is estimated that the person is not a person with an unregistered address.

  As described above, the search processing unit 30 is configured to store other images not stored in the address book within a predetermined distance from the person corresponding to the registered address stored in the address book 18 in the still image stored in the storage unit 33. If there is a person, the other person's image is estimated to be the person's image corresponding to the unregistered address. As a result, in a still image including registered persons and unregistered persons, if the distance between the two is relatively close, there is a high possibility that the person is an acquaintance of the user. It can be estimated that the person should be registered.

(Judgment criteria F: Specify the distance from the photographer)
If a face image of a person registered in the address book 18 and an unregistered face image appear simultaneously in one still image, the distance between the registered person's face image and the photographer The difference between the unregistered face image and the photographer is detected. In this case, for example, detection is performed based on a difference in face size or a difference in person's height. If the ratio between the distance between the registered person's face image and the photographer and the distance between the unregistered face image and the photographer is within a specified value, the unregistered face image Q1 is set to the unregistered address. Is estimated as the face image of the person corresponding to.

  For example, as shown in FIG. 4B, even when a person P1 with a registered address and an unregistered face image Q1 appear in the same image, the image is taken compared to the person P1 with a registered address. Since the person of the unregistered face image Q1 that is clearly far from the person is highly likely not to be an acquaintance, it is estimated that the person is not a person with an unregistered address.

  As described above, the search processing unit 30 uses the first distance that is the distance between the person corresponding to the registered address stored in the address book 18 and the photographer of the still image in the still image stored in the storage unit 33. If the ratio between the other person who is not stored in the address book 18 included in the still image and the second distance, which is the distance between the photographer of the still image, is equal to or less than a predetermined ratio, the image of the other person Is an image of a person corresponding to an unregistered address. As a result, if the distance from the photographer is the same among the still images that include the registered person and the unregistered person, it is highly possible that the user is an acquaintance of the user. Can be estimated as a person to be registered.

(Criteria G: defines the number of faces between people)
If a face image of a person registered in the address book 18, an unregistered face image, and another face image appear at the same time in one still image, the registered face image The number of other face images existing between the unregistered face images is detected, and if this number is less than the specified value, the unregistered face image is estimated as a person face image candidate corresponding to the unregistered address. To do.

  For example, as shown in FIG. 4D, when there are three persons corresponding to the other face images R1 to R3 between the person P1 at the registered address and the unregistered face image Q1. There is a high possibility that the relationship between the person of the registered address and the person of the unregistered face image is sparse. That is, in the case of FIG. 4D, it is highly probable that the person in the unregistered face image is not the user's acquaintance, so it is estimated that the person is not a person at the unregistered address.

  As described above, the search processing unit 30 includes, in the still image stored in the storage unit 33, a person corresponding to the registered address stored in the address book 18, and other persons not stored in the address book 18. If the number of persons existing in between is less than or equal to the predetermined number, it is estimated that the images of other persons are images of persons corresponding to unregistered addresses. As a result, if the number of third parties existing between a registered person and an unregistered person in a still image containing a large number of persons is less than or equal to a predetermined number, the user may be an acquaintance of the user Therefore, the unregistered person can be estimated as a person to be registered.

  Next, a screen display example of the information processing apparatus 1 will be described.

  Specific examples of the appearance and various screen display states of the information processing apparatus 1 are shown in FIGS. The display examples shown in FIGS. 5A to 5D will be described below.

  When the information processing apparatus 1 receives an e-mail, information as shown in FIG. 5A is displayed on the display unit 13 as the contents of the received mail. The received mail displayed on the screen of FIG. 5A includes information of date 51a, sender 51b, destination 51c, destination 51d, and text 51e. In the example of the mail in FIG. 5A, the mail address of “Kiro-kun” is included as the information of the sender 51b, and the user's mail address is included as the information of the destination 51c.

  In the example shown in FIG. 5A, it is assumed that the mail address “aabbcc@xxx.ne.jp” of the destination 51d is not yet registered in the address book 18. Therefore, for example, by the trigger generated when this e-mail is received, the process proceeds from step S31 to S34 shown in FIG. 3, and since the destination “aabbcc@xxx.ne.jp” is an unregistered address, step S35 ( The content of FIG. 2 is executed.

  Then, in step S20 shown in FIG. 2, information as shown in FIG. 5B or 5C is displayed on the display unit 13. That is, when only a single face image is detected as a candidate as a result estimated in steps S17 and S18 of FIG. 2, the screen of FIG. 5B is displayed, and a plurality of face images are detected as candidates. If this happens, the screen of FIG. 5C is displayed.

  In FIG. 5B, the character string information 52a of the unregistered address “aabbcc@xxx.ne.jp”, the face image 52b obtained by estimation, and the registration thereof are displayed in the pop-up window. Is displayed as registration recommendation information 52c prompting the user ("isn't it?"). Further, in the screen of FIG. 5B, a mail link button 52d, a still image link button 52e, an address book registration button 52f, and a registered person display section 52g are also displayed.

  The mail link button 52d is assigned a link for calling a specific mail whose unregistered address is detected and displaying the contents.

  The still image link button 52e is displayed when the estimated face image is extracted from the still image. This button is assigned a link for calling up the still image and displaying the contents. When the estimated face image is extracted from the moving image, a moving image link button (not shown) for calling the moving image is displayed instead of the still image link button 52e.

  The address book registration button 52f is assigned a function for transitioning to a display screen for registering unregistered addresses and estimated face image information displayed on the screen of FIG. 5B in the address book 18. ing. That is, when the user clicks the address book registration button 52f via the operation unit 12 while this screen is displayed, the screen proceeds to a screen for registering unregistered addresses and estimated face image information in the address book 18. be able to.

  The registered person display unit 52g displays information related to the registered person. In the example of FIG. 5B, the sender “K-ro-kun” is a registered person, and this person (K-ro-kun) and the person at the destination unregistered address may be acquainted. A message to be displayed is displayed on the registered person display section 52g.

  In FIG. 5C, the character string information 53a of “aabbcc@xxx.ne.jp”, which is an unregistered address, and a plurality of candidates as face images obtained by estimation exist in the pop-up window. A message 53b indicating that the registration is to be performed, and information (“Do you want to register?”) As recommended registration information 53h that prompts registration are displayed. Further, a face image candidate display button 53c, a mail link button 53d, a still image link button 53e, an address book registration button 53f, and a registered person display section 53g are also displayed in the screen of FIG.

  A link for transitioning to the screen shown in FIG. 5D is assigned to the face image candidate display button 53c. That is, when the user clicks the face image candidate display button 53c via the operation unit 12 or the like, as shown in FIG. 5D, a plurality of estimated face images are displayed as a candidate list 54a. The user can select one of them.

  A link for calling a specific mail whose unregistered address is detected and displaying the contents is assigned to the mail link button 53d.

  The still image link button 53e is displayed when the estimated face image is extracted from the still image. This button is assigned a link for calling up the still image and displaying the contents. When the estimated face image is extracted from the moving image, a moving image link button (not shown) for calling the moving image is displayed instead of the still image link button 53e.

  The address book registration button 53f is assigned a function for transitioning to a display screen for registering unregistered addresses and estimated face image information displayed in the screen of FIG. ing. However, when there are a plurality of face image candidates, the screen 54 is displayed, and after the user selects one of the face images as a registration target via the operation unit 12 or the like, the screen for registration is displayed. Transition.

  According to the information processing apparatus 1 of this embodiment, when a trigger is generated under a predetermined condition, if there is an unregistered address as the sender or destination mail address of the received e-mail, this information is used by the user. Can be registered in the address book 18 with a simple operation. In particular, since the face image of a person corresponding to the unregistered address is automatically estimated and displayed, it becomes easy to register personal information including the face image.

  The present invention relates to an information processing apparatus, an information processing method, an information processing program, and the like capable of supporting a user's input operation for registering unregistered information including an image of a person related to the user in a telephone directory or the like Useful for.

DESCRIPTION OF SYMBOLS 1 Information processing apparatus 11 Communication part 12 Operation part 13 Display part 14 E-mail receiving part 15 Image photographing process part (an example of an image photographing part)
16 Person authentication processing unit (an example of a face authentication processing unit)
17 Key input processing unit 18 Address book (an example of an address storage unit)
19 Address book management unit 20 Mail analysis unit (an example of information extraction unit)
21 header information analysis unit 21a destination determination unit 21b telephone directory registration address determination unit 22 text analysis unit 23 received mail file 30 search processing unit (an example of a person image estimation unit)
31 still image data search processing unit 32 moving image data search processing unit 33 storage unit (an example of an image storage unit)
34 Movie data file 35 Still image data file

Claims (16)

An image storage unit for storing images;
An address storage unit for storing an address and an image of a person corresponding to the address;
An information extraction unit that extracts unregistered addresses not stored in the address storage unit from the mail;
A person image estimation unit for estimating an image of a person corresponding to an unregistered address extracted by the information extraction unit from the image stored in the image storage unit;
Registration recommendation for urging registration of the unregistered address extracted by the information extraction unit, the person image estimated by the person image estimation unit, and the unregistered address and the person image in the address storage unit A display unit for displaying information;
An information processing apparatus comprising: The information processing apparatus according to claim 1,
The information extraction unit extracts time information included in the mail,
The person image estimation unit is configured to store the unregistered image from an image captured at a time having a predetermined similarity with the time information extracted by the information extraction unit among the images stored in the image storage unit. An information processing apparatus that estimates an image of a person corresponding to an address. The information processing apparatus according to claim 1,
The information extraction unit extracts position information included in the mail,
The person image estimation unit is configured to store the unregistered image from an image photographed at a position having a predetermined similarity with the position information extracted by the information extraction unit among the images stored in the image storage unit. An information processing apparatus that estimates an image of a person corresponding to an address. The information processing apparatus according to claim 1,
The information extraction unit extracts a registered address stored in the address storage unit from an email,
The person image estimation unit includes an image of a person corresponding to the unregistered address from images including a person corresponding to the registered address extracted by the information extraction unit among the images stored in the image storage unit. Information processing apparatus for estimating The information processing apparatus according to any one of claims 2 to 4,
In the moving image stored in the image storage unit, the person image estimating unit stores the address storage in frames within a predetermined number of frames before and after a frame including a person corresponding to a registered address stored in the address storage unit. An information processing apparatus that estimates that an unregistered person image is an image of a person corresponding to the unregistered address when an unregistered person image that is an image of another person not stored in the section is included. The information processing apparatus according to any one of claims 2 to 4,
The person image estimation unit includes a person corresponding to a registered address stored in the address storage unit in the moving image stored in the image storage unit, and is not stored in the address storage unit. An information processing apparatus that estimates that an unregistered person image is an image of a person corresponding to the unregistered address when there are a predetermined number or more of frames including unregistered person images that are images of the person. The information processing apparatus according to any one of claims 2 to 4,
The person image estimation unit includes a person corresponding to a registered address stored in the address storage unit among moving images stored in the image storage unit, and is not stored in the address storage unit An information processing apparatus that estimates that an unregistered person image is an image of a person corresponding to the unregistered address when a predetermined number or more of moving image files including unregistered person images that are images of other persons exist . The information processing apparatus according to any one of claims 2 to 4,
The person image estimation unit is stored in the address storage unit in a frame group including a person corresponding to a registered address stored in the address storage unit in the moving image stored in the image storage unit. An information processing apparatus that estimates that an unregistered person image is an image of a person corresponding to the unregistered address when a frame including an unregistered person image that is an image of another person who is not present exists for a predetermined time or more. . The information processing apparatus according to any one of claims 2 to 4,
The person image estimation unit is not stored in the address storage unit within a predetermined distance from the person corresponding to the registered address stored in the address storage unit in the still image stored in the image storage unit. An information processing apparatus that estimates that an image of another person is an image of a person corresponding to the unregistered address. The information processing apparatus according to any one of claims 2 to 4,
The person image estimation unit is a first distance that is a distance between a person corresponding to a registered address stored in the address storage unit and a photographer of the still image in the still image stored in the image storage unit. And a second distance, which is a distance between another person who is not stored in the address storage unit included in the still image and a photographer of the still image, is equal to or less than a predetermined ratio, An information processing apparatus that estimates a person image as a person image corresponding to the unregistered address. The information processing apparatus according to any one of claims 2 to 4,
The person image estimation unit includes a person corresponding to a registered address stored in the address storage unit in the still image stored in the image storage unit, and another person not stored in the address storage unit, An information processing apparatus that estimates that an image of another person is an image of a person corresponding to the unregistered address when the number of persons existing between the two is less than or equal to a predetermined number. The information processing apparatus according to claim 1, further comprising:
Equipped with a communication unit that performs mail communication,
The person image estimation unit is an information processing apparatus that estimates a person image corresponding to the unregistered address when an email is received by the communication unit. The information processing apparatus according to claim 1, further comprising:
It has an image capture unit that captures images,
The person image estimation unit is an information processing apparatus that estimates an image of a person corresponding to the unregistered address when image shooting is completed by the image shooting unit. The information processing apparatus according to claim 1, further comprising:
A face authentication processing unit that performs face authentication processing on the image stored in the image storage unit;
The person image estimation unit is an information processing apparatus that estimates a person image corresponding to the unregistered address when the face authentication processing is completed by the face authentication processing unit. Extracting from the mail an unregistered address that is not stored in an address storage unit that stores an address and an image of a person corresponding to the address;
Estimating an image of a person corresponding to the extracted unregistered address from an image stored in an image storage unit that stores the image;
Displaying the extracted unregistered address, the estimated person image, and registration recommendation information for prompting the user to register the unregistered address and the person image in the address storage unit;
An information processing method comprising:   An information processing program for causing a computer to execute each step of the information processing method according to claim 15.

Images