JP2010277441A - Portable terminal and facial identification system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To identify a face relatively simply and quickly in a portable terminal without a search server. <P>SOLUTION: A first terminal 100a detects a face in an image acquired by imaging and transmits the face data to peripheral portable terminals 100b-100g by short-range radio communications. The transmitted face data is compared (facial identification) with the face image of a user stored in the second portable terminals 100b-100g. The first terminal 100a receives the user identification information of the user from the second terminals 100b-100d storing the corresponding face images by short-range radio communications, and displays them on a display screen and/or stores the user identification information in association with the face image on a display screen. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a face identification system using communication between a plurality of portable terminals.

  Nowadays, mobile phone terminals that are widely used have advanced functions and high performance, and most models have a camera photographing function. Some have a short-range wireless communication function such as Bluetooth (registered trademark).

  On the other hand, there is also known a technique for detecting a face area of a person present in an image obtained by camera photography, and further extracting feature point data of the face image to perform face identification. (Refer nonpatent literature 1).

  In such a situation, if a person (or people) is photographed using a mobile phone terminal and information about the person can be obtained immediately based on the face image captured in the image, a new application of the camera is expanded. it is conceivable that.

  In Patent Document 1, when you want to know the name, attributes, characteristics, breeding method, etc. of an object (plant, insect, etc.), the object is photographed with the camera attached to the mobile phone, and the image and the content of the information you want are displayed. A system is disclosed that makes a request to a search server via the Internet and provides desired information for a question entered by a user from the server side.

JP 2003-132062 A

David G. Lowe, "Distinctive Image Features from Scale-Invariant Keypoints", Accepted for publication in the International Journal of Computer Vision, 2004, Computer Science Department, University of British Columbia, Vancouver, B. C., Canada, January 5, 2004

  However, the conventional face identification technology has the following problems.

  In general, face data necessary for identification needs to be held in the mobile terminal in advance. In particular, it is necessary to have face data for the number of faces to be identified. In addition, in order to identify a person who is not registered, it is necessary to update the face data in the mobile terminal. Furthermore, when performing the identification process, it is necessary to perform a matching process between the input data and the registered data. It takes a considerable amount of processing time to execute such a process in one portable terminal, especially in a state where a large number of face data are registered.

  The technique described in Patent Document 1 requires a search server on the Internet. In addition, it is not realistic to prepare a database in which face images of real people are concentrated.

  The present invention has been made in such a background, and makes it possible to perform face identification relatively easily and quickly in a portable terminal without requiring a search server.

  The portable terminal according to the present invention transmits a face detection unit for detecting a face existing in an image obtained by photographing, and transmitting the face data corresponding to the image data of the face to surrounding portable terminals by short-range wireless communication. Means for receiving the user identification information of the user by short-range wireless communication from the portable terminal storing the face image corresponding to the transmitted face data, and corresponding the received user identification information to the face image And a processing means for displaying on the display screen and / or storing in the storage unit.

  In the present invention, processing for detecting a face present in an image obtained by photographing is performed in the first portable terminal. However, the face identification processing is executed by the second portable terminal existing in the vicinity by transmitting the face data corresponding to the image data of the face to the nearby portable terminal by short-range wireless communication. The user identification information of the user is received by the short-range wireless communication from the portable terminal that stores the face image corresponding to the transmitted face data among the second portable terminals existing around. Thereby, the identification of the face in an image is performed rapidly. When the user of the first portable terminal photographs another person (people) as a photographer, the photographed person is naturally present in the vicinity. In addition, the photographed person is highly likely to have a portable terminal. The present invention functions extremely effectively on such a premise.

  Another portable terminal according to the present invention as a second portable terminal is a receiving means for receiving face data from another portable terminal, and a storage for storing user identification information of the user of the own terminal including data of a plurality of items. And a face match detection means for confirming whether or not the received face image of the face data matches the face image of the user of the terminal when receiving the transmission of the face data, and the match of the face image Sometimes, it is provided with a transmitting means for returning only the data of the permitted item to the other portable terminal.

  Note that the first mobile terminal and the second mobile terminal can actually be provided with the same mobile terminal.

  The face identification system according to the present invention is a face identification system using a first portable terminal and a second portable terminal. The first portable terminal is a short-distance wireless communication unit that detects a face existing in an image obtained by photographing, and a face data corresponding to the image data of the face to a second portable terminal that exists in the vicinity. Transmitting means for transmitting by communication; receiving means for receiving user identification information of the user by short-range wireless communication from a portable terminal storing a face image corresponding to the transmitted face data; and received user identification information Processing means for displaying on the display screen and / or storing it in the storage unit in association with the face image. A second mobile terminal receiving means for receiving the face data from the first mobile terminal; storage means for storing user identification information of the user of the terminal including data of a plurality of items; and transmission of the face data Receiving the face data of the face data and the face image of the user of the user terminal to confirm whether or not the face image of the permitted item is matched when the face image matches. Transmitting means for returning only data to the first portable terminal.

  According to the present invention, face recognition can be performed relatively easily and quickly on a mobile terminal without requiring a search server. In addition, since it is not necessary to store face data other than the face data of the owner itself in the mobile terminal, application to an existing mobile terminal is easy.

It is a figure which shows schematic structure of the communication system temporarily constructed | assembled with the some portable terminal in embodiment of this invention. It is a figure which shows the schematic hardware constitutions of a typical mobile telephone terminal as a portable terminal shown in FIG. It is the sequence diagram which showed the exchange between the portable terminals in embodiment of this invention. It is a figure of the scene where the user (owner) of the 1st terminal photographs other users in the embodiment of the present invention. It is a figure which shows the example of the image obtained by imaging | photography of FIG. It is a figure of a mode that the face detection was made | formed from the image shown in FIG. It is a figure of a mode that the face data detected in the scene of FIG. 6 was broadcast from the 1st terminal to the 2nd terminal. It is a figure which shows the mode of the matching process of the face data in the 2nd terminal in embodiment of this invention. It is a figure which shows the example of the user identification information in embodiment of this invention. It is a figure which shows the example of the user identification information inquiry setting screen and reply availability setting screen in the 2nd terminal of embodiment of this invention. It is a figure which shows a mode that user identification information is returned to the 1st terminal from the 2nd terminal in embodiment of this invention. It is a figure which shows the state which displayed the said user identification information under each person's face area in the picked-up image in embodiment of this invention. It is a flowchart which shows the process of the 1st terminal which inquires in embodiment of this invention. It is a flowchart showing the process of the 2nd terminal which received the inquiry in embodiment of this invention.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows a schematic configuration of a face identification system as a communication system temporarily constructed with a plurality of portable terminals in the present embodiment. Here, a mobile phone terminal will be described as an example of the mobile terminal. In this example, the present system is configured by a first mobile terminal 100a that is the subject of the operation, and second mobile terminals 100b, 100c, 100d, and the like located in the vicinity of this terminal. Although the number of second mobile terminals is three in the figure, it may be one or four or more. The portable terminals 100a, 100b, 100c, 100d, etc. are collectively referred to as the portable terminal 100. Each of these mobile terminals has a short-range wireless communication means, and has a function of temporarily constructing a short-range wireless network that can perform wireless data communication with each other. As the short-range wireless network, for example, Bluetooth (registered trademark) can be used. If a Bluetooth (registered trademark) PAN (Personal Area Network) profile is used, a network with a radius of about 10 m can be constructed. However, the present invention is not limited to this, and any available wireless network can be adopted.

  The first portable terminal has an imaging means such as a digital camera. The second portable terminal does not necessarily need to have an imaging unit. Further, it is assumed that the second terminal stores user identification information as meta information regarding the user together with the face image data of the user. A mobile phone terminal usually has a function of storing user identification information as profile information of a user of the terminal.

  FIG. 2 shows a schematic hardware configuration of a typical mobile phone terminal as the mobile terminal 100. This configuration can function as both the first and second portable terminals.

  The mobile terminal 100 includes a control unit 101, a communication unit 102, an antenna 103, a display unit 104, an operation unit 105, a storage unit (storage means) 106, an audio processing unit 108, and an imaging control unit 111 that are connected to each other via a bus 117. A short-range wireless communication unit 113, an antenna 114, and a clock unit 115. A speaker 109 and a microphone 110 are connected to the audio processing unit 108. A camera 112 (imaging means) is connected to the imaging control unit 111.

  The control unit 101 includes a CPU, and controls each unit of the mobile terminal 100 and performs necessary data processing. In the present embodiment, the control unit 101 constitutes “face matching detection means” and “processing means” of the present invention. The communication unit 102 performs wireless communication with the base station via the antenna 103. The display unit 104 includes a display device such as an LCD or an organic EL, and displays information on the display screen. The operation unit 105 includes a numeric keypad and various control keys, and receives an instruction and data input from a user. The storage unit 106 includes storage means such as a ROM, a RAM, and a flash memory, and stores an OS, computer programs such as various application programs, and various data. The various data includes the user's face image and user identification information in the present embodiment. The audio processing unit 108 is connected to a speaker 109 that outputs received voice, alarm, audio of a moving image file, audio of music data, and the like, and a microphone 110 that collects transmitted audio and the like, and encodes and decodes an audio signal. Predetermined audio processing including conversion is performed. The short-range wireless communication unit 113 is a unit that performs wireless communication with another short-range wireless communication unit via the antenna 114, and constitutes the “transmission unit” and the “reception unit” of the present invention. In the present embodiment, the short-range wireless communication unit 113 temporarily establishes a wireless communication network with the short-range wireless communication unit of another terminal. The clock unit 115 constitutes the clock means of the present invention and includes an RTC (Real Time Clock) and the like, and provides a clock function for generating date and time information and a timer function for setting a timer.

  Hereinafter, the operation of the present embodiment will be described.

  For user recognition based on a face image, as a first stage, a face detection or face detection stage for detecting a face image (ie, a face area) in an image acquired by photographing, and a second stage As a stage, there is a face identification stage for identifying who the face is based on the detected face image. In the present invention, the face detection process in the first stage is performed by the first terminal, while the face identification process in the second stage is performed by the second terminal. Therefore, it is assumed that people whose faces are identified carry (carry) a portable terminal and their face data is registered in the portable terminal.

  Now, as a typical application in the present embodiment, when the user of the first portable terminal takes a picture of another person (or people) using the camera function, user identification information (namely, personal) Information) is acquired, and at least a part of the user identification information is assumed to be displayed on a display screen such as a viewfinder in association with the person on the captured image.

  FIG. 3 is a sequence diagram showing exchanges between portable terminals 100a to 100d in the present embodiment. Hereinafter, the mobile terminal is also simply referred to as a terminal. In this figure, in addition to the terminals 100b to 100d, the terminals 100e to 100g are also included as representatives of the terminals that exist by chance (within the range where short-range wireless communication is possible).

  Now, the user (owner) of the terminal 100a, which is the first portable terminal, takes a picture of another user (S1). Here, as shown in FIG. 4, it is assumed that user A has photographed users B to D as a photographer. It is assumed that the users B to D have terminals 100b to 100d, respectively. The “photographing” in the application of the portable terminal 100 in this embodiment is not limited to a mode in which a single still image is acquired by pressing the shutter button, but also a so-called viewfinder display screen of the display unit before the shutter button is pressed. A form in which camera-through images are displayed continuously (periodically) can also be included.

  FIG. 5 shows an example of an image obtained by the terminal 100a. In this example, a captured image 40 in which the users B to D having the terminals 100 b to 100 d are shown is shown as an example.

  Thereafter, the terminal 100a performs face detection from the captured image 40 (input image) from the camera, that is, determines a face area and extracts each face image (S2). In the above example, as shown in FIG. 6, the face area including the face images 44 to 46 detected from the captured image 40 is a rectangular area.

  The terminal 100a broadcasts the face data corresponding to the extracted face image to terminals owned by people around using the short-range wireless network (S3). The data for face identification processing transmitted to the second terminal may be the face image data of the extracted face area itself, or may be feature point data extracted from the face image data. Ordinary face identification techniques often convert data into a data format (face identification data) for matching before matching whether they are the same face. If the processing necessary for performing such face identification is performed in advance at the transmitting terminal, the receiving terminal only needs to perform data comparison processing, and the processing speed is further improved. In this specification, these data are collectively referred to as “face data”.

  In the above example, as shown in FIG. 7, the face data broadcast from the terminal 100a can be received by the terminals 100e to 100g that exist by chance in addition to the terminals 100b to 100d.

  The mobile terminal that has received the face data uses face data that has been registered in advance to check whether the face data that has been sent corresponds to the face of the user. Face identification) processing is performed (S4b to S4g). As the feature point data matching processing, for example, a technique as disclosed in Non-Patent Document 1 can be used. According to this technique, object matching can be performed with high reliability even when the size of the image of an object (including a face), the direction of the object, and the like are different.

  In the above example, as illustrated in FIG. 8, for example, the face image 45 ′ stored in the terminal 100 c of the user C matches the face image 45 among the transmitted face images 44 to 46. The face image 49 ′ stored in the terminal of the user E does not match any of the transmitted face images 44 to 46. Although the face data transmitted as described above may not be the face image itself, it is shown here as “face image” for convenience. The face image 45 ′ stored in the terminal is an image of the face of the same person as the face image 45 in the captured image, but the image data should not be exactly the same. However, in the example of the figure, the same image is used for convenience.

  Such a matching process can be performed at high speed because even if there are a plurality of sent face data, it is only necessary to match the face data with one person's own face data. That is, matching processing is performed in a distributed manner at terminals around the photographing terminal.

  Here, FIG. 9 shows an example of user identification information in the present embodiment. FIG. 9 shows an example of user identification information stored in each of the terminals 100b, 100c, and 100d corresponding to the above example.

  Each user identification information includes items such as “name”, “nickname”, “telephone number”, “e-mail address”, “image”, “date of birth”, and the like. “Image” assumes a face image of the user (owner) of the terminal. The image may include a plurality of images, for example, an image 1 as a face image, an image 2 as an avatar, and the like.

  FIGS. 10A and 10B show examples of the user identification information inquiry setting screen 501 and the reply availability setting screen 502 in the second terminal, respectively. The user of the second terminal may not want to respond to an inquiry for user identification information from the first terminal. Such a user does not respond to the inquiry of the user identification information from the first terminal by setting OFF in the user identification information inquiry setting screen 501 in advance. Even when ON is set, it is assumed that a response is automatically made unconditionally at the time of response. Alternatively, a message such as “There was a user identification information inquiry. Can I respond?” Can be sent on condition that the user's confirmation is obtained. It is.

  The reply permission / prohibition setting screen 502 allows the user of the second terminal to set permission / non-permission (“permitted” or “not permitted”) for each item of user identification information. This reply permission / inhibition setting screen 502 can be used in combination with the user identification information inquiry setting screen 501 so that it can be set repeatedly when the reply permission / inhibition setting screen 502 is set to ON. Alternatively, the reply availability setting screen 502 can be used alone. In this case, if all items are set to “non-permitted”, a result substantially equivalent to setting “OFF” on the user identification information inquiry setting screen 501 can be obtained.

  Returning to FIG. 3, the terminal that performed the face identification processing by matching returns the user identification information that can be disclosed to the terminal 100 a when the face match is confirmed (S <b> 5 b to S <b> 5 d). As described above, this user identification information is personal information of an item for which the user has set reply permission. A terminal whose face match is not confirmed does not send any reply. In the above example, as shown in FIG. 11, only the terminals 100b to 100d in which the face images 44 ′, 45 ′, and 46 ′ respectively matching the face images 44, 45, and 46 are stored are stored inside. Each openable user identification information is returned to the terminal 100a.

  Upon receiving these replies, the terminal 100a displays the user identification information in association with the person on the captured image (S6). In the above example, as shown in FIG. 12, the user identification information is displayed under the face area of each person in the captured image 40. The position where the user identification information is displayed is not limited to the position below the face area. If the correspondence with the face image is known, the display position is not limited. The items of the displayed user identification information of each person correspond to those permitted by the user. The “age” is calculated by comparing the current date of birth when the date of birth is returned. Instead of “age”, the date of birth may be displayed as it is. If the item “age” originally exists, such a calculation process is unnecessary.

  In addition to or instead of displaying such user identification information on the display screen, the acquired user identification information may be stored inside.

  The user identification information to be returned is typically text data, but bitmap image data in which the text is expanded may be returned. In this case, in order to use age as the user identification information, the age is calculated on the second terminal side before the bitmap image is generated. The user may be able to select whether to reply by text or image.

  FIG. 13 shows a flowchart of processing of the first terminal (terminal 100a) that makes an inquiry. This process is realized by the control unit 101 (FIG. 2) of the terminal 100 reading and executing the program in the storage unit 106.

  In the present embodiment, the captured image to be processed is typically an image in the viewfinder (S11a) when the camera 112 is activated after the camera 112 of the terminal is activated (S10a) or in the viewfinder ( Assume an image obtained when it is selected to use a so-called camera-through image (S11b). The camera-through image is also considered as a still image that is continuously updated, and one of the still images can be processed. In addition, this processing target can be sequentially updated as the camera through image is updated. Note that an image to be processed may be selected from the terminal (for example, the storage unit 106) without starting the camera (S10b).

  When a captured image to be processed is given, a face detection process is performed to detect a face (face area) in the image and extract a face image (S12). Such a method of the face detection process itself is well known and will not be described in detail here. If no face is detected (S13, No), this process ends.

  If a face is detected (S13, Yes), the face data is transmitted (broadcast) to nearby terminals (S14). In response to this, it waits for receiving user identification information from a nearby terminal (S15). It waits for responses from the number of terminals corresponding to the number of detected face images. Even if there is no response from all terminals when a predetermined time has passed since transmission, it is determined that a timeout has occurred and the standby is terminated.

  If there is a response to the user identification information even in one terminal (S16, Yes), the user identification information is displayed on the captured image in association with the corresponding face image (S17). As described above, instead of or in addition to the display, the user identification information may be stored (registered) in the storage unit 106 (for example, a phone book) together with the face image.

  FIG. 14 is a flowchart showing processing of the second terminal that has received the inquiry. This process is realized by the control unit 101 (FIG. 2) of the terminal 100 reading and executing the program in the storage unit 106.

  First, the second terminal confirms the setting contents set on the user identification information inquiry setting screen 501 as shown in FIG. 10A (S21). If the setting is “OFF”, this process ends.

  If the setting is “ON”, the following loop processing is repeatedly executed until there is no unprocessed face data (S22).

  In the loop process, first, if there are a plurality of received face data, one face data is selected from them (S23). If the face data is the face image data itself, face identification data (feature point data) is generated from the face image data (S24). If the face data is already face identification data, this process is unnecessary. A matching process is performed for comparing the face identification data with the face data stored in the terminal (the face identification data obtained from the data) (S25). If they match, the process proceeds to step S26. If they do not match, the process returns to step S22.

  In step S26, the contents of the reply permission / non-permission setting set on the reply permission / non-permission setting screen 502 in FIG. User identification information is returned to the first terminal for the items permitted by this setting (S27). At this time, if at least a plurality of pieces of face data are received, the second terminal also adds image identification data indicating which received face data corresponds to the returned user identification information. As such data, if the image ID is originally added to each face data when the face data is received, it can be used. If no image ID is added, a number indicating the order in which the face data is received can be used as the image identification data.

As described above, according to the present embodiment, the following effects can be obtained.
(1) It is sufficient to hold the owner's own data (face data and user identification information) on the portable terminal, and it is not necessary to have other person's data. Therefore, it is not necessary to perform the update.
(2) Since it is not necessary to have the face data and user identification information of the other party to perform face identification in advance, the first face-to-face partner can be identified.
(3) The user who wants to perform face identification (first terminal) need only perform face detection processing.
(4) Since face identification processing is left to surrounding terminals, face identification can be performed at high speed.

  The present invention can be realized by a combination of existing technologies. Further, it is considered to be very useful as a technique for realizing augmented reality (AR) as well as face identification. Augmented reality is a concept that is paired with virtual reality, and refers to the composite presentation of virtual objects as additional information in the real environment as electronic information. When a display device (HMD: Head Mount Display) worn on the user's head becomes practical, it keeps track of the face (individual) once recognized from the camera image by linking with the camera device. It is possible to realize AR with a relatively small amount of calculation.

  The preferred embodiments of the present invention have been described above, but various modifications and changes other than those mentioned above can be made. For example, although the mobile terminal of the present invention has been described by taking a mobile phone terminal with a camera as an example, the present invention can also be applied to other devices such as a digital camera with a communication function.

44, 45, 45 ', 46, 49, 49' ... face image, 100, 100a to 100g ... portable terminal, 101 ... control unit, 102 ... communication unit, 103 ... antenna, 104 ... display unit, 105 ... operation unit, DESCRIPTION OF SYMBOLS 106 ... Memory | storage part, 108 ... Audio | voice processing part, 109 ... Speaker, 110 ... Microphone, 111 ... Imaging control part, 112 ... Camera, 113 ... Near field communication part, 114 ... Antenna, 115 ... Clock part, 117 ... Bus, 501 ... User identification information inquiry setting screen, 502 ... Reply availability setting screen

Claims (7)

Face detection means for detecting a face present in an image obtained by photographing;
Transmitting means for transmitting face data corresponding to the image data of the face to surrounding mobile terminals by short-range wireless communication;
Receiving means for receiving user identification information of the user by short-range wireless communication from a mobile terminal storing a face image corresponding to the transmitted face data;
A mobile terminal comprising: processing means for displaying received user identification information on the display screen in association with the face image and / or storing the received user identification information in the storage unit in the storage unit.   The mobile terminal according to claim 1, wherein the face data is image data in a face area in the image.   The mobile terminal according to claim 1, wherein the face data is face identification data obtained from a face image in a face area in the image. The mobile terminal according to claim 1, wherein the user identification information is text data or image data obtained by developing text. Receiving means for receiving face data from another mobile terminal;
Storage means for storing user identification information of a user of the terminal including data of a plurality of items;
Face coincidence detecting means for confirming whether or not the face image of the received face data matches the face image of the user of the terminal when receiving the transmission of the face data;
A mobile terminal comprising: a transmission unit that returns only the data of the permitted item to the other mobile terminal when the face images match. When the transmission of the face data is received, it has a reply permission setting unit that determines in advance whether or not to permit a reply for each item, and the transmission unit stores the data of the permitted item when the face images match. The mobile terminal according to claim 5, wherein only the mobile terminal is returned to the other mobile terminal. A face identification system using a first portable terminal and a second portable terminal,
The first mobile terminal
Face detection means for detecting a face present in an image obtained by photographing;
Transmitting means for transmitting face data corresponding to the image data of the face to a second portable terminal existing in the vicinity by near field communication;
Receiving means for receiving user identification information of the user by short-range wireless communication from a mobile terminal storing a face image corresponding to the transmitted face data;
Processing means for displaying the received user identification information on the display screen in association with the face image and / or storing it in the storage unit,
The second mobile terminal
Receiving means for receiving the face data from the first portable terminal;
Storage means for storing user identification information of a user of the terminal including data of a plurality of items;
Face coincidence detecting means for confirming whether or not the face image of the received face data matches the face image of the user of the terminal when receiving the transmission of the face data;
Transmitting means for returning only the data of the permitted item to the first mobile terminal when the face images match, a face identification system.