JP2010193207A - Mobile information terminal, image information management method, and image information management program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable effective utilization of photographs or videos which are obtained by photographing or the like and stored. <P>SOLUTION: A controller 10 generates a face index image from a photograph or video which are obtained by photographing or externally acquired and stored, and stores the face index image in a memory 15 with corresponding to registration information of the original photograph or video, or telephone directory. In addition, the controller 10 stores utilization history of browsing or transmission and reception of the photograph or video in the memory 15. Further, when the browsing or transmission and reception of the photograph or video is carried out, the controller 10 notifies user of whether the photograph or video is browsed or whether transmission and reception of the photograph or video is carried out or the like, based on the utilization history of the photograph or video, or transmission and reception history of the photograph or video, or the like. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a portable information terminal having various functions such as retention of phone book (address book) information and outgoing / incoming call history, transmission / reception of e-mail, browsing of photos / videos, and the like, such as a mobile phone terminal, The present invention relates to an image information management method and an image information management program for managing photographs and moving images in a portable information terminal.

  In recent years, mobile phone terminals, which are representative examples of portable information terminals, have improved photo and video shooting functions, and the memory capacity installed has also increased. For this reason, it is possible to easily take pictures, moving pictures, and the like with the mobile phone terminal, and the number of pictures and moving pictures stored in the terminal are increased accordingly.

  On the other hand, recent digital still cameras and digital video cameras detect face images using face recognition technology from captured images displayed on the viewfinder, for example, focus on the face images or smile. It is equipped with functions such as detection and automatic shooting. In particular, a digital video camera recognizes a face image from a recorded video by using a face recognition technology, for example, creates a face index image from the recognized face image or records it. There is also a model equipped with a function that allows a moving image including a desired face image to be selectively extracted from a plurality of moving images and viewed.

  As another technique for using a photographed photographic image or the like, for example, Japanese Patent Application Laid-Open No. 2005-267146 discloses a subject image stored in an address book memory in association with a mail address. And a subject image portion shown in a group photo image, and a technique for creating and sending an e-mail addressed to a mail address corresponding to the matched subject image is disclosed. According to the technique described in Patent Document 1, even when there are many destination mail addresses, it is possible to easily select a desired address and create and send an e-mail addressed to that address.

  Further, as a technique for using a photographed photograph image or the like, Japanese Patent Application Laid-Open No. 2008-5470 discloses a patent document (Patent Document 2) in which a face image extracted from a still image matches a registered face image in a telephone directory. A technique for linking a person face image in the still image with a telephone number or a mail address of a registered person in the telephone directory is disclosed. In the technique disclosed in Patent Document 2, when a desired person is selected from the displayed still image, the telephone number linked to the face image of the person is automatically transmitted, or the e-mail address is input. Display the created mail creation screen. According to the technique described in Patent Document 2, it is possible to make a telephone call or send an e-mail by an operation of selecting a person on a still image.

Japanese Patent Laying-Open No. 2005-267146 (FIG. 1) JP 2008-5470 A (FIG. 1)

  By the way, as described above, a large number of photographs and videos taken by a portable information terminal such as a cellular phone terminal are often simply stored and are said to be effectively used. hard. For this reason, it is desired to make effective use of these photos and videos. In particular, in mobile phone terminals, which are typical examples of portable information terminals, effective use of photos and videos by improving browsing and sharing functions and effective use of photos and videos by using face recognition technology are desired. Yes.

  The present invention has been proposed in view of such a situation, and a portable information terminal, an image information management method, and an image information management program that enable effective use of a photograph and a moving image that are captured and stored The purpose is to provide.

  The portable information terminal of the present invention includes a face information extraction unit that extracts face information from an image that has been captured or acquired and stored, and a face information storage unit that stores the extracted face image in association with an extraction source image And an address association storage unit that stores the extracted face information in association with the address information, and a usage history storage unit that stores image usage history and image usage history information through the address information. Further, when using the image, the portable information terminal uses the usage history of the image based on the usage history of the image and the usage history of the address information associated with the face information corresponding to the image, Alternatively, the user is notified of the unused status of the image through the address information, and when using the address information, based on the usage history of the image corresponding to the face information associated with the address information, An information control unit is provided for notifying the user of an unused state of an image through the address information or an unused state of the image. Thereby, this invention solves the subject mentioned above.

  That is, according to the present invention, when an image or address information is used, the usage history of the image, the usage history of the address information associated with the face information corresponding to the image, the face information associated with the address information Is used to notify the user of the unused status of the image, the unused status of the image through the address information, and the like. Thereby, according to this invention, utilization of the image which is not utilized can be accelerated | stimulated.

  In the present invention, when an image or address information is used, it corresponds to the use history of the image, the use history of the address information associated with the face information corresponding to the image, and the face information associated with the address information. Based on the usage history of the image, notify the user of the unused status of the image or the unused status of the image through the address information. Effective use is possible.

It is a block diagram which shows the schematic internal structural example of the mobile telephone terminal of embodiment of this invention. It is a figure which shows an example of a face index image. It is a figure which shows an example of a face template image. It is a figure used for description of correlation of a face index image and a telephone directory. It is a figure which shows one example of a display when the photograph and moving image linked | related with the person of the other party of communication represented by a certain outgoing / incoming call history are not browsed yet on the screen where the outgoing / incoming call history is displayed. . FIG. 10 is a diagram showing a display example when a photograph or a movie associated with a certain destination email address is not yet attached to an email or an upload destination URL is not yet notified on an email creation / transmission screen. is there. If the photo or video data file has not yet been sent as an email attachment to the email address associated with the face image in the photo or video browsing screen, or the upload destination URL has not been announced yet It is a figure which shows the example of a display of a case. 6 is a flowchart showing a flow of processing when a face index image is generated from an image of a photograph or a moving image in the mobile phone terminal of the present embodiment. 5 is a flowchart of image information management processing when a phone book, outgoing / incoming call history, and mail transmission / reception history are being browsed in the mobile phone terminal of the present embodiment. 5 is a flowchart of image information management processing at the time of creating an e-mail in the mobile phone terminal of the present embodiment. In the mobile phone terminal of this embodiment, image information management processing when a photo or video associated with the phone book registrant is being browsed by face index, and the size of the photo / video is greater than or equal to a threshold value It is a flowchart which shows the process of. In the mobile phone terminal of this embodiment, image information management processing when a photo or video associated with a phone book registrant is being browsed by face index, and the size of the photo / video is less than a threshold It is a flowchart which shows the process of.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  As an example to which the present invention is applied, in the present embodiment, a mobile phone terminal with a camera function capable of capturing a still image (photograph) or a moving image is taken as an example, but of course, the contents described here are It is an example to the last, and it goes without saying that the present invention is not limited to this example.

[Schematic configuration of mobile phone terminal]
FIG. 1 shows a schematic internal configuration example of the mobile phone terminal of the present embodiment.

  In FIG. 1, a communication antenna 12 is a built-in antenna, for example, and transmits / receives signal radio waves for packet communication such as telephone calls and e-mails. The communication circuit 11 performs frequency conversion, modulation, demodulation, and the like of transmission / reception signals.

  The speaker 20 includes a speaker for receiving provided on the mobile phone terminal, and a speaker for outputting ringer (ringing tone), alarm sound, warning sound, reproduced music, digital sound, reproduced moving image sound, and the like. The digital audio data supplied from the unit 10 is amplified after digital / analog conversion, and the amplified audio signal is converted into an acoustic wave and output to the air.

  The microphone 21 is a microphone for transmitting and collecting external audio, converts an acoustic wave into an audio signal, amplifies and analog / digital converts the input audio signal, and converts the audio data after the analog / digital conversion. Send to control unit 10.

  The operation unit 13 includes keys such as a numeric keypad, an utterance key, and an end / power key provided on a casing (not shown) of the mobile phone terminal according to the present embodiment, and operation elements such as a cross key and a jog dial. An operation signal generator that generates an operation signal when the operation element is operated.

  The display unit 14 includes, for example, a display device such as a liquid crystal display or an organic EL (ElectroLuminescent) display, and a display drive circuit for the display. An image signal supplied from the control unit 10 causes an e-mail or the like on the display. Various characters and messages are displayed, and still images and moving images are displayed.

  The camera 23 includes an image sensor, an optical system, and the like for taking photographic images and moving images. The imaging control unit 22 performs control of the imaging element of the camera 23, autofocus control, exposure control, and the like based on instructions from the control unit 10. Then, photographic image data and moving image data taken by the camera 23 are sent to the control unit 10 via the data line, subjected to image processing as needed, or sent to the memory unit 15 for storage. .

  The charging control unit 24 supplies various power such as power supplied from the battery 26 or power supplied from the charging terminal 25 to each unit, and charging the battery 26 with power supplied from the charging terminal 25. Perform management.

  The memory unit 15 includes a ROM (Read Only Memory) and a RAM (Random Access Memory). The ROM includes a rewritable storage medium such as a NAND-type flash memory. For example, an OS (Operating System) program, a control program for the control unit 10 to control each unit, In addition to the application program, for example, compression-coded music data content and moving image data content, various initial setting values, font data, dictionary data, model name information, terminal identification information, and the like are also stored. The RAM stores data as needed as a work area when the control unit 10 performs various data processing.

  In the example of FIG. 1, among the storage areas of the memory unit 15, the program storage area 31, the telephone directory storage area 32, the history storage area 33, the photo data storage area 34, the moving picture data storage area 35, the index A template image storage area 36 and an e-mail data storage area 37 are shown.

  The control program and application program are stored in the program storage area 31. In the present embodiment, an image information management program for the control unit 10 to execute management of photographs / moving images as described later is stored. .

  The telephone book storage area 32 stores data of a telephone book (address book) registered in the mobile phone terminal of the present embodiment.

  In the history storage area 33, call history and e-mail transmission / reception history, so-called web (Web) connection and web data upload, download web connection history, photo and video browsing history, various processes in the mobile phone terminal Various kinds of history data such as the execution history are stored. Note that the call history includes telephone number information, which is address information of a call partner, information on the call date, and the like. The e-mail transmission / reception history includes e-mail transmission / reception address information, e-mail transmission / reception date / time information, a photo or video in the e-mail, a face index image to be described later, and a face template image to be described later. If the data file is attached, history information indicating the data file is included. In the communication history of the web connection, in addition to the URL that is the address information of the web connection destination and the date of the connection, uploading or downloading of a data file such as a photo or a video is performed for the desired connection destination URL. In some cases, it includes upload / download history information such as photos and moving images. The browsing history of the photos and videos includes historical information such as browsing date, browsing time, browsing count, etc. when the user browses the photos and videos previously displayed on the screen. As an execution history of various processes in the mobile phone terminal, a user's browsing history by displaying pictures and videos on the display unit 14, a user's browsing history by displaying the contents of the phone book on the screen, The browsing history of the user when the outgoing / incoming history is displayed on the screen, the browsing history of the user when the transmission / reception history of electronic mail is displayed on the screen, and the like are included. In this embodiment, the history of photos / videos, such as the browsing history of the above photos / videos, email transmission history of photos / video files, web connection history when uploading / downloading photo / video files, etc. This is called the usage history of images such as photos and videos.

  The photo data storage area 34 stores still image data captured by the camera 23, still image data acquired through e-mail or the web, and still image data acquired through an external connector or an external memory. Similarly, the moving image data storage area 35 stores still image data captured by the camera 23, still image data acquired through e-mail or the web, and still image data acquired through an external connector or an external memory.

  The e-mail data storage area 37 stores e-mail data created by the mobile phone terminal of the present embodiment, and transmitted / received. In addition, when files such as photographs and moving images and face index images and face template data described later are attached to the transmitted and received e-mails, these files are also stored in the e-mail image storage area 37.

  As will be described later, the index / template image storage area 36 stores face index image data and face template image data generated from a face image extracted by face recognition or the like from a photo or video. In particular, in the case of the present embodiment, when the photo or moving image data that is the basis of the face index image or the face template image exists in the photo data storage area 34 or the moving image data storage area 35, the face index image, the face Information indicating the association between the template image, the photo, and the moving image data, that is, information such as the face index image, the photo that is the basis of the face template image, and the storage address of the moving image data is also stored in the index / template image storage area 36 and the like Is done. In the present embodiment, the face index image and the face template image can be associated with each registration information of the telephone directory. When the face index image and the face template image are associated with the registration information of the phone book, the association information is, for example, in the phone book storage area 32 as storage address information of the face index image and face template image data. Are stored in association with each registration information.

  The control unit 10 includes a CPU (central processing unit), and performs communication control, audio signal processing, image signal processing, other various signal processing, control of each unit, and the like in the communication circuit 11. The control unit 10 executes various control programs and application programs stored in the memory unit 15 and performs various data processing associated therewith.

  In particular, in the case of the present embodiment, the control unit 10 executes an image information management program stored in the program storage area 31 of the memory unit 15 to thereby execute an index template generation function, an index template-phone described below. Each function such as a book association function, an image communication history management function, an image browsing history management function, a history browsing assistance function, an image browsing assistance function, and an image communication assistance function can be realized.

  The index / template generation function extracts a face image from a photograph or video by face recognition processing, etc., and generates a face index image as shown in FIG. 2 or a face template image as shown in FIG. 3 from the face image. And a function for storing them in the index / template image storage area 36 of the memory unit 15. 2 shows an example of a plurality of face index images, and FIG. 3 shows an example of a face template image generated from a photograph. In particular, the face template image is, for example, a grayscale image of about 16 × 16 pixels used for matching a human face image, and can be created from a human face photo. In addition, human face image matching is performed by calculating a correlation between a face template image and a photograph or movie from which a person's face image is to be extracted, and an image region where a high correlation value is obtained from the photograph or movie. Is determined to be the face image area of the person. Thereby, it becomes possible to detect the image area of a specific person's face from a photograph or a moving image using a face template image. Further, for example, a plurality of face template images of the user himself / herself can be generated and stored from, for example, face images of different hairstyles, and in this way, the detection accuracy of the face image can be improved.

  The index template-phone book association function is a function for registering the face index image and face template image generated by the index template generation function in association with each registration information of the phone book in the phone book storage unit 32. It is. FIG. 4 shows an example of association registration between a plurality of face index images and each registration information in the phone book. That is, in the example of FIG. 4, when the friend f, friend a, and friend e are registered in the phone book, the phone book registration information of the friend f and the face index image F of the friend f are associated with each other. The phone book registration information of the friend a and the face index image A of the friend a show how the phone book registration information of the friend e and the face index image E of the friend e are associated with each other.

  The above-mentioned image communication history management function includes communication history information indicating whether photos and videos are attached to e-mails, e-mail address information, communication history information indicating whether uploading or downloading to the web has been performed, The URL information (address information) is stored in the communication history storage area 33 of the memory unit 15 and managed.

  The image browsing history management function is a function of storing browsing history information indicating whether or not a user browses a photograph or video displayed on the screen of the display unit 14 in the history storage unit 33 of the memory unit 15 and manages the browsing history information. It is.

  The history browsing assistance function includes, for example, a case where a phone book, a call outgoing / incoming history, an email transmission / reception history, etc. are displayed on the screen of the display unit 14 and the user is browsing them. When photos and videos related to phone numbers and e-mail addresses displayed as history are saved, the user is notified of the existence of these photos and videos and assists the user in selecting and viewing them. It is a function to do. For example, photos and videos associated with a person specified by a telephone number in a call origination / incoming history or a mail address in an email transmission / reception history are stored in the photo data storage area 34 and the video data storage area 35 of the memory unit 15. If it is stored, it notifies the user of their presence so that the user can select and view them. In particular, in this case, the history browsing assistance function is used when there are photographs or videos that the user has not browsed yet, for example, on the screen 4 of the display unit 14 of the mobile phone terminal 1 as shown in FIG. Is also preferentially notified to the user. In FIG. 5, for example, on the screen 2 on which a call origination / reception history is displayed, a picture or a video associated with a person of a communication partner represented by a certain outgoing / incoming history r1 is a mobile phone of this embodiment. An example of display when the user of the telephone terminal has not yet browsed is shown. In such a case, the history browsing assistance function displays the pop-up image P1 in the vicinity of the display area of the outgoing / incoming call history r1, and asks the user whether to browse photos or videos. For example, when an input for instructing browsing is made by the user (for example, a browsing instruction is input by pressing a key assigned in advance on the terminal), the history browsing assistance function is A photograph or video associated with the person of the communication partner represented by the history r1 is displayed.

  The image communication auxiliary function is, for example, when photos and videos are displayed on the screen 2 of the display unit 14 and the user is viewing them, such photos and videos have been emailed before, It is determined whether the URL on the web when the file is uploaded has been previously notified to another user terminal or the like. Then, the image communication auxiliary function displays, for example, as shown in FIG. 6, when the above-mentioned photos and videos have not been sent by e-mail or the upload destination URL of these files has not been notified to another terminal or the like. A selection menu is displayed on the screen 2 of the unit 14 to attach the photos and videos to an email by a simple operation, or to send an email notification of the URL of the web upload destination by a simple operation. Note that FIG. 6 shows a case where, for example, in the e-mail creation / transmission screen, a photo or video associated with a certain destination e-mail address M has not yet been sent as an e-mail attachment, or a photo or video is displayed. A display example is shown in the case where the URL when uploaded on the web is not yet notified by e-mail or the like. In the case of the example of FIG. 6, in the image communication auxiliary function, a pop-up image P2 indicating the mail address M is displayed, and confirmation of a photograph or a movie and transmission by email attachment or URL transmission of an upload destination are performed to the user. Ask them what to do. Then, when an input for instructing confirmation (“view”) of, for example, a photograph or a video is made from the user (for example, a confirmation instruction is input by pressing a key or the like assigned in advance on the terminal), The image communication auxiliary function displays a photo or a moving image associated with the e-mail address. In addition, when an input for instructing transmission (“send”) of, for example, a photograph or a video is made from the user (for example, a confirmation instruction is input by pressing a key or the like assigned in advance on the terminal), The image communication auxiliary function automatically attaches a data file of a photo or a moving image associated with the e-mail address to the e-mail. At this time, if the size of the data file of the photo or moving image exceeds the size that can be transmitted by e-mail, it is automatically changed to a size that can be transmitted by e-mail.

  The image browsing assistance function is, for example, a case where a picture or video image is displayed on the screen 2 of the display unit 14 and the user is browsing the face index image included in the picture or video. For example, whether the photo or video data file was previously sent as an email attachment to the email address associated with the URL, or whether the URL on the web when the file was uploaded was previously notified to decide. Then, the image browsing assistance function is used when the above-mentioned photo or video has not been sent by e-mail before or the upload destination URL of the file has not been notified to another terminal or the like, for example, as shown in FIG. On the screen 2 on which the image V of the video is displayed, a display for attaching the photo or video to an email by a simple operation or for notifying the web upload destination URL by a simple operation is displayed. Do. In FIG. 7, in the browsing screen for the image V of the photo or video, the data file of the photo or video is not yet sent as an email attachment to the e-mail address associated with the face image in the image V. In such a case, a display example is shown in which the URL at the time of uploading these photos and videos on the web is not yet notified by e-mail or the like. In the case of the example of FIG. 7, the image browsing auxiliary function displays a pop-up image P3 indicating these photos and videos, and transmits the data files of the photos and videos to the user by mail attachment or upload destination Ask whether to send the URL. Then, from the user, the data file such as the photo or the update destination URL is sent to all e-mail addresses associated with all the face images included in the image of the photo or video, or each face When an instruction is given to send the data file or its update destination URL individually to each e-mail address associated with the image, the image browsing assistance function is directed to the e-mail address associated with the face image. The data file of the above-mentioned photograph or moving picture or the upload destination URL thereof is sent by e-mail all at once or individually.

  In addition, the control unit 10 also performs predetermined audio processing such as processing for sending audio data generated by predetermined audio processing such as decoding to the speaker 20 and encoding of audio data supplied from the microphone 21. The control unit 10 also generates characters, symbols, images, various user interface images, web page images, and the like that are displayed on the display unit 14 on the screen.

  Further, although not shown in FIG. 1, the mobile phone terminal of the present embodiment includes an LED (light emitting diode) for key illumination and incoming light and a driving unit thereof, a so-called Bluetooth method (Bluetooth: registered trademark). , UWB (Ultra Wide Band) system, short-range wireless communication unit for short-range wireless communication by wireless LAN (Local Area Network), etc., non-contact communication processing unit with non-contact IC card function and reader / writer function Also, each component provided in a general mobile phone terminal, such as a GPS (Global Positioning System) communication unit, an external memory slot, a digital broadcast receiving tuner unit and an AV codec unit, and a timer (clock unit) .

[Face index image generation processing]
FIG. 8 shows a flow of processing when a face index image is generated from an image of a photograph or a movie in the mobile phone terminal of the present embodiment. Note that the processing of the flowchart of FIG. 8 is realized by the control unit 10 executing the image information management program according to the present invention.

  In FIG. 8, when the process of generating the face index image by executing the image information management program is performed, the control unit 10 is recorded in advance in the photo data storage area 34 and the moving picture data storage area 35 as the process of step S1. Check if the data file of the photo / movie that you are using exists. If there is a recorded photo / moving image file, the control unit 10 advances the process to step S2, and if not, ends the process of the flowchart of FIG.

  When the processing proceeds to step S2, the control unit 10 determines whether or not a face index image has been created before from the recorded photo / moving image file. Then, if there is a photo / movie file for which no face index image has been created, the control unit 10 proceeds to step S3. On the other hand, if a face index image has already been created for all the photo / movie files. Ends the process of the flowchart of FIG.

  When the processing proceeds to step S3, the control unit 10 extracts a face image area using a face recognition technique from the photograph / movie images for which a face index image has not yet been created, and the extracted face image. Is created as a face index image. At this time, the control unit 10 generates a face template image from the face index image, and associates the face index image with the face template image. Then, the control part 10 advances a process to step S4.

  In step S4, the control unit 10 collates the face index image created in step S3 with a personal face image registered in advance in the phone book. The collation of the face image at this time can also be performed by matching the face template image corresponding to the created face index image and an individual face image registered in the phone book in advance. When it is determined that the person is the same person by collating the face images in step S4, the control unit 10 associates the face index image created in step S3 with the information in the phone book. After the process of step S4, the control 10 returns the process to step S2, and determines whether it is necessary to create a face index image with another photo / movie file.

[Image information management process when browsing phonebook and history]
FIG. 9 shows a flowchart of the image information management process when the telephone directory, outgoing / incoming call history, and mail transmission / reception history are being browsed in the mobile phone terminal of this embodiment. Note that the processing of the flowchart of FIG. 9 is realized by the control unit 10 executing the image information management program according to the present invention.

  In FIG. 9, when the phone book and history are being browsed, the control unit 10 first uses the face index image to display the browsed phone book, outgoing / incoming call history, and mail transmission / reception history as processing of step S <b> 11. Determine if there are associated photo and video files. Then, if there is a related photo / moving image file, the control unit 10 proceeds to step S12. If not, the control unit 10 ends the process of the flowchart of FIG.

  When the processing proceeds to step S12, the control unit 10 is set so that the current setting of the mobile phone terminal 1 displays only the photo / video files that are not currently viewed among the related photo / video files. Determine whether or not. The control unit 10 advances the process to step S13 if it is set to display only unviewed photo / movie files, and advances the process to step S14 if it is not set.

  When the processing proceeds to step S13, the control unit 10 proceeds to step S14 when there is a photo / video file that is not currently viewed among the related photo / video files, and proceeds to step S14. The process of the flowchart of FIG. 9 ends.

  When the process proceeds from step S13 to step S14, the control unit 10 presents to the user by pop-up display or the like that there is a photo / movie file that has not been browsed. On the other hand, when the process proceeds from step S12 to step S14 without going through step S13, the control unit 10 displays a pop-up display indicating that there is the related photo / movie file based on a preset priority. To the user. After the process of step S14, the control unit 10 advances the process to step S15.

  In step S15, the control unit 10 determines whether or not the user has selected the pop-up display. If selected, the process proceeds to step S16. If not selected, the control unit 10 proceeds to step S15. The process of the flowchart of FIG. 9 ends.

  When the process proceeds to step S16, the control unit 10 displays the selected photograph / moving image on the screen to show to the user.

[Image information management process when creating e-mail]
FIG. 10 shows a flowchart of image information management processing at the time of creating an e-mail in the mobile phone terminal of this embodiment. Note that the processing of the flowchart of FIG. 10 is realized by the control unit 10 executing the image information management program according to the present invention.

  In FIG. 10, when an e-mail is being created, the control unit 10 sends the destination e-mail address input (or selected from the phone book) by the user on the e-mail creation screen as the process of step S <b> 21. Is determined whether there is a photo / movie file associated with the face index image. Then, the control unit 10 advances the process to step S22 when there is a related photo / moving image file, and ends the process of the flowchart of FIG.

  When the processing proceeds to step S22, the control unit 10 determines whether or not the setting is such that only the photo / moving image files that have not yet been sent by e-mail are displayed. The control unit 10 advances the process to step S23 if the setting is such that only the photo / moving image file that has not been mailed is displayed, and advances the process to step S24 if it is not set. .

  When the process proceeds to step S23, the control unit 10 proceeds to step S24 if there is a photo / moving image file that has not been transmitted by mail, and ends the process of the flowchart of FIG. To do.

  When the process proceeds from step S23 to step S24, the control unit 10 presents to the user by a pop-up display or the like that there is a photo / moving image file that has not been emailed. On the other hand, when the process proceeds from step S22 to step S24 without going through step S23, the control unit 10 displays a pop-up display indicating that there is a related photo / movie file based on a preset priority. To the user. After the process of step S24, the control unit 10 advances the process to step S25.

  When the process proceeds to step S25, the control unit 10 proceeds to step S26 when the user selects transmission in the pop-up display menu, and on the other hand, when not selected by the user, the control unit 10 proceeds to FIG. The process of the flowchart in FIG.

  When the process proceeds to step S26, the control unit 10 determines whether the related photo / moving image file has been uploaded on the web. Then, the control unit 10 proceeds to step S27 when the upload has been completed, and proceeds to step S30 when the upload has not been performed yet.

  When the process proceeds to step S27, the control unit 10 displays an option for asking the user whether to notify only the upload destination URL, and then proceeds to step S28.

  When the process proceeds to step S28, the control unit 10 determines whether or not the notification of only the URL has been selected by the user. If the selection is made, the process proceeds to step S29. Then, the process proceeds to step S30.

  When the process proceeds to step S29, the control unit 10 includes the upload destination URL in the mail body, and ends the process of the flowchart of FIG.

  If the processing proceeds to step S30, the control unit 10 attaches the related photo / moving image file to the mail, and ends the processing of the flowchart of FIG.

[Image information management processing when browsing photos / videos associated with phone book registrants by face index images]
FIG. 11 and FIG. 12 show a flowchart of the image information management process when the mobile phone terminal of this embodiment is browsing a photo or video associated with the phone book registrant by the face index. 11 and 12 is realized by the control unit 10 executing the image information management program according to the present invention.

  In FIG. 11, when an e-mail is being created when a photo or video associated with the phone book registrant is being browsed using the face index, the control unit 10 performs the process of step S41 as follows: It is determined whether the image size of the browsed photo / moving image is greater than or equal to a preset threshold value or less than the threshold value. If the control unit 10 determines that the value is equal to or greater than the threshold, the process proceeds to step S42. If the control unit 10 determines that the value is less than the threshold, the process proceeds to step S52 in FIG.

  In step S42, the control unit 10 determines whether or not there is a history of uploading the browsed photo / movie file to the web. If there is an uploaded history, the control unit 10 proceeds to step S43. If there is no uploaded history, the control unit 10 proceeds to step S44.

  When the processing proceeds to step S43, the control unit 10 has notified the upload destination URL to all of the persons by the face recognition of the person image in the photograph / movie file being browsed. Judge whether. Then, the control unit 10 ends the process of this flowchart when all the people have been notified, and on the other hand, when there is a person who has not notified, the process proceeds to step S47.

  When the process proceeds to step S44, the control unit 10 displays a menu as to whether to upload the browsed photo / movie file to the web, and then proceeds to step S45.

  When the process proceeds to step S45, the control unit 10 proceeds to step S46 if the user selects to upload via the display menu, and on the other hand, if the user does not select, the control unit 10 The process ends.

  When the processing proceeds to step S46, the control unit 10 uploads the above-not uploaded photo / moving image file on the web. At this time, the control unit 10 stores the upload destination URL in the memory unit 15 in association with the photo / moving image file, and proceeds to step S47.

  When the process proceeds to step S47, the control unit 10 displays a selection menu for asking the user whether to notify the upload destination URL by e-mail, and proceeds to step S48.

  When the process proceeds to step S48, the control unit 10 proceeds to step S49 when the user selects to notify, and ends the process of this flowchart when the user does not select it.

  When the processing proceeds to step S49, the control unit 10 displays a mail editing screen in which the URL of the upload destination is automatically input in the mail text, and proceeds to step S50.

  When the process proceeds to step S50, the control unit 10 waits for a transmission instruction after the mail is edited by the user, and proceeds to step S51 when the transmission instruction is input after the mail is edited. If an end instruction is input without a transmission instruction, the process of this flowchart ends.

  When the process proceeds to step S51, the control unit 10 transmits the mail and ends the process of this flowchart.

  When the process proceeds to step S52 in FIG. 12, the control unit 10 determines whether there is a history in which the browsed photo / moving image file has been transmitted with an e-mail attached before that. And the control part 10 advances a process to step S53, when there exists the said transmission log | history, and advances a process to step S54 when there is no transmission log | history.

  When the process proceeds to step S53, the control unit 10 has transmitted the photo / moving image file to all the persons obtained by face recognition from the human image reflected in the browsed photo / moving image file. Determine if there is. Then, the control unit 10 ends the process of this flowchart when it has been transmitted to all of those persons, and on the other hand, if there is a person who has not yet transmitted, the process proceeds to step S54.

  In step S54, the control unit 10 attaches the browsed photo / moving image file to the e-mail and transmits it to the e-mail address associated with the face index image obtained from the photo / moving image. Displays a menu of whether or not.

  Next, the control unit 10 advances the processing to step S55, determines whether or not the user has selected transmission of the e-mail attached with the file, and if the transmission is selected, the processing proceeds to step S56. If the transmission is not selected, the process of this flowchart is terminated.

  When the processing proceeds to step S56, the control unit 10 displays a mail editing screen for an e-mail with the photo / moving image file attached, and proceeds to step S57.

  When the process proceeds to step S57, the control unit 10 waits for a transmission instruction after the mail is edited by the user. When the transmission instruction is input after the mail is edited, the control unit 10 proceeds to step S58. If an end instruction is input without a transmission instruction, the process of this flowchart ends.

  When the process proceeds to step S58, the control unit 10 transmits the mail and ends the process of this flowchart.

[Summary]
As described above, according to the embodiment of the present invention, the face index image associated with the phone book information can be created, and the person can be associated with the person from the outgoing / incoming history or the mail transmission / reception history without troublesome operations. You can browse photos and videos immediately. In this way, for example, when a topic of a certain person comes out, you can immediately browse photos and videos, remember the scenes at that time, and show other people about the photos and videos Will be very easy to do.

  In addition, according to the embodiment of the present invention, when a photograph or video showing a person is attached to an e-mail or uploaded to the web, the photo / video file is opened to check who is shown each time. This means that users can share photos and videos with friends as soon as they think.

  In addition, according to the embodiment of the present invention, the user determines whether the photograph / video is sent to the person shown in the photo or video by email attachment or whether the photo / video can be shared by uploading to the web. Since it is possible to know immediately, for example, it is not possible to forget to send only to a certain person.

  Furthermore, according to the embodiment of the present invention, a face template image is generated from a face image and stored in the terminal, or when the address is exchanged with another terminal, for example, the face stored in the terminal If you send the template image, you can easily detect your face from the photo / video file that the person took. Similarly, when acquiring address information from a friend, if you acquire a face template image from the other party, you can easily detect the face of the friend when you take a photo / video file on your device. become able to. In this way, if a face template image is shared with a friend or the like, the faces of friends can be easily detected, and the reusability of the face template image can be improved.

  The above description of the embodiment is an example of the present invention. Therefore, the present invention is not limited to the above-described embodiments, and various modifications can be made according to the design and the like as long as they do not depart from the technical idea according to the present invention. . For example, in the above-described embodiment, a mobile phone terminal has been described as an example, but the present invention is naturally applicable to other mobile information terminals.

  DESCRIPTION OF SYMBOLS 1 Mobile phone terminal, 2 Display screen, 10 Control part, 11 Communication circuit, 12 Communication antenna, 13 Operation part, 14 Display part, 15 Memory part, 20 Speaker, 21 Microphone, 22 Imaging control part, 23 Camera, 24 Charge control Unit, 25 charging terminal, 26 battery, 31 program storage area, 32 phone book storage area, 33 history storage area, 34 photo data storage area, 35 moving image data storage area, 36 index / template image storage area, 37 e-mail data storage region

Claims (6)

A face information extraction unit that extracts face information from an image captured or acquired and stored;
A face information storage unit that stores the face information extracted by the face information extraction unit in association with the extraction source image;
An address association storage unit that stores the face information extracted by the face information extraction unit in association with the address information;
A usage history storage unit for storing information on usage history of the image and usage history of the image through the address information;
When using the image, the usage history of the image is referred to the usage history of the address information associated with the face information corresponding to the image from the usage history storage unit. When notifying the user of the unused status or the unused status of the image through the address information and using the address information, use of the image corresponding to the face information associated with the address information An information control unit that refers to the history from the usage history storage unit and notifies the user of the unused status of the image through the address information or the unused status of the image;
A portable information terminal.   The usage history storage unit includes at least one of a usage history related to browsing of the image, a usage history related to transmission / reception of the image, a usage history related to uploading the image on the web, and a usage history related to browsing of the address information. The portable information terminal according to claim 1, wherein the portable information terminal stores the usage history information. The face information extraction unit extracts a face image from the image by face recognition processing,
The portable information terminal according to claim 1, wherein the face information storage unit stores the face image as the face information. The face information extraction unit generates a face image obtained by converting the face image extracted by the face recognition process into a grayscale image having a predetermined number of pixels,
5. The mobile phone according to claim 4, wherein the face information storage unit stores the face image extracted by the face recognition process and a face image obtained by converting the face image into a grayscale image having a predetermined number of pixels as the face information. Information terminal. A step in which a face information extraction unit extracts face information from an image captured or acquired and stored; and
Associating the face information extracted by the face information extraction unit with the extraction source image, and storing the storage information in the face information storage unit;
A step of storing the face information extracted by the face information extraction unit in the address association storage unit in association with the address information;
A step in which a storage control unit stores information on the usage history of the image through the usage history of the image and the address information in a usage history storage unit;
When using the image, the information control unit refers to the usage history of the image and the usage history of the address information associated with the face information corresponding to the image from the usage history storage unit. When notifying the user of the unused state of the image or the unused state of the image through the address information and using the address information, the information control unit is associated with the address information. Referring to the usage history of the image corresponding to the face information from the usage history storage unit, and notifying the user of the unused status of the image through the address information or the unused status of the image;
A method for managing image information. Face information extraction processing for extracting face information from images that have been shot or acquired and stored;
A face information storage control process in which the face information extracted in the face information extraction process is stored in the face information storage unit in association with the extraction source image;
Address related information storage control processing for storing the face information extracted in the face information extraction processing in the address association storage unit in association with the address information;
A usage history storage control process for storing the usage history of the image and the usage history information of the image through the address information in a usage history storage unit;
When an image is used, the usage history of the image and the usage history of the address information associated with the face information corresponding to the image are referred to from the usage history storage unit, When notifying the user of the usage status or the usage status of the image through the address information and using the address information, the usage history of the image corresponding to the face information associated with the address information Is used from the usage history storage unit, and information control processing for notifying the user of the unused status of the image through the address information or the unused status of the image,
An image information management program to be executed by a portable information terminal.