JP5801690B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to a technique for presenting information on a subject shown in a captured image.

  In recent years, AR (Augmented Reality) technology that adds and synthesizes virtual object using CG (Computer Graphics) and attribute information of objects in the real environment to images of the real environment taken with cameras etc. It is thriving. For example, augmented reality software for mobile phones has appeared that displays related tag information superimposed on real images from GPS (Global Positioning System) position information over a camera.

  In such an AR system, it is desired to display information related to the subject in the vicinity of the subject. In other words, when various subjects are included in the photographed image, it is desirable to display in such a way that the user can know which subject information each piece of information related to the subject is. For this purpose, it is necessary to uniquely identify the subject from the captured image, and to link the identified subject and information to be displayed in the vicinity of the subject.

  Patent Document 1 discloses a technique for adding information notified by a wireless tag owned by a subject to attached images as attached information. More specifically, an image retrieval apparatus that includes image feature information that can identify the face of the wireless tag owner and the like in the notification information of the wireless tag and that can identify the subject by comparing with the captured image is disclosed. doing.

  In Patent Document 2, when a subject having an RF tag passes through the area of the directional antenna, the subject is photographed with a camera whose angle of view is linked to the directional antenna area. An apparatus for automatically assigning video attributes that can be detected is disclosed.

  Patent Document 3 discloses a method in which a control device notifies a camera of a location where a subject exists, and the camera calculates a relative position between the subject and the camera to identify the location of the subject.

JP 2007-052646 JP 2007-228195 A JP 2008-066963

  However, when a plurality of subjects are close to each other or depending on the orientation of the subject, it may be difficult to specify the subject individually.

  For example, in the technique disclosed in Patent Document 1, it is difficult to specify a subject from image feature information such as a face when the subject is a captured image facing backward. Also, with the techniques disclosed in Patent Document 2 and Patent Document 3, when there are many subjects in the same area such as a downtown area, it is difficult to distinguish between adjacent subjects.

  The present invention has been made in view of such a problem, and an object of the present invention is to provide a technique for uniquely identifying a subject even when the subject cannot be identified from feature information of the subject. .

In order to achieve the above-described object, the image processing apparatus of the present invention includes means for acquiring a captured image,
Receiving means for receiving sensor information representing the state of the user obtained by sensing the user with a sensor;
Acquisition means for acquiring the attribute information of the user and an identification parameter for identifying the user in the captured image;
An identification unit for identifying the user from among subjects in the captured image based on the sensor information received by the reception unit and the identification parameter acquired by the acquisition unit;
And output means for combining and outputting the attribute information acquired by the acquisition means at a position on the captured image corresponding to a result of identification by the identification means.

  According to the configuration of the present invention, even when the subject cannot be identified from the feature information of the subject, the subject can be uniquely identified.

The figure which shows the structure of a system. The figure explaining the information which the server 109 hold | maintains. FIG. 2 is a block diagram illustrating an example of a functional configuration of the digital camera 101. The block diagram which shows the function structural example of a mobile telephone. The figure which shows a process sequence. The figure which shows the structural example of identification information. The flowchart which shows the detail of the process in step S510. The figure which shows the structural example of the table produced | generated in step S905. The flowchart of the process performed in step S703. The figure which shows the structural example of identification information. The figure which shows the structural example of the table produced | generated in step S905. The figure which shows the structural example of a table. The figure which shows the structural example of a table. The figure which shows the structural example of a table.

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

[First Embodiment]
The image processing apparatus according to the present embodiment is a digital camera that synthesizes and displays attribute information of a subject appearing in the captured image on the captured image. FIG. 1 shows the configuration of a system including this digital camera.

  The digital camera 101 has a wireless LAN function conforming to IEEE 802.11 in addition to a function of generating, recording, and displaying a captured image by capturing an image of a subject. In FIG. 1, there are persons 102, 104, and 106 around the digital camera 101, and the persons 102, 104, and 106 hold the mobile phones 103, 105, and 107 in their hands, respectively. Each of the mobile phones 103, 105, and 107 is a device having a wireless LAN function conforming to IEEE 802.11, and enables wireless communication with the digital camera 101. Each of the mobile phones 103, 105, and 107 is an example of a mobile terminal. Any mobile terminal can be used as long as the mobile phone 103, 105, and 107 can perform each process described below. , 107 may be applied. In FIG. 1, it is assumed that there are three mobile phones around the digital camera 101, but the number is not limited to this.

  Each of the mobile phones 103, 105, and 107 transmits identification information including information related to the own device and information related to the owner (user) of the own device to the digital camera 101 by the wireless LAN function periodically or irregularly. The identification information may be transmitted in response to a transmission request from the digital camera 101. This identification information is given and transmitted as an element of a frame conforming to IEEE 802.11. Details of this identification information will be described later.

  The digital camera 101 can also perform data communication with a server 109 as an external device via a network 108 such as the Internet. However, the digital camera 101 uses the identification information received from each of the mobile phones 103, 105, and 107, and uses various identifications described later relating to the persons 102, 104, and 106 who are owners of the mobile phones 103, 105, and 107. Information is acquired from the server 109. The digital camera 101 uses the acquired information and the identification information to identify a subject (one or more of the persons 102, 104, and 106) in the captured image captured by the digital camera 101, and information about the identified subject. Are superimposed on the captured image and displayed.

  Next, a functional configuration example of the digital camera 101 will be described with reference to the block diagram of FIG. The control unit 320 executes various processes using computer programs and data stored in the storage unit 312 and controls the operation of each unit constituting the digital camera 101.

  The wireless communication control unit 301 includes an antenna and a circuit for transmitting and receiving wireless signals to and from the mobile phones 103, 105, and 107, and a control unit that controls wireless communication by controlling them.

  The shutter button 302 is a button for starting imaging. When the user of the digital camera 101 presses the shutter button 302, imaging processing is started in the imaging unit 303 under the control of the control unit 320.

  The imaging unit 303 includes a lens, an RGB sensor, an infrared sensor (thermal sensor), an X-ray sensor, an A / D converter, and a control unit that controls them, and the imaging unit 303 performs imaging processing to perform imaging. An image can be generated.

  The display unit 304 has a display area for displaying a captured image captured by the imaging unit 303 and other information (image information and character information), and includes a liquid crystal display and a control unit that controls the display. The identification information acquisition unit 305 receives the identification information from each of the mobile phones 103, 105, and 107 by controlling the wireless communication control unit 301.

  The face feature information acquisition unit 306 controls the wireless communication control unit 301, and the identification information acquisition unit 305 uses the identification information acquired from each of the mobile phones 103, 105, and 107 to send the mobile phone 103, 105, 107 to the server 109. Queries the facial features of each owner. In this embodiment, a facial feature quantity is used as an example of an identification parameter that can be used to identify the owner. However, any parameter that can be used to identify the owner is not a facial feature quantity. These parameters may be adopted.

  The attribute information acquisition unit 307 controls the wireless communication control unit 301, and the identification information acquisition unit 305 uses the identification information acquired from each of the mobile phones 103, 105, and 107 to send the mobile phone 103, 105, 107 to the server 109. Queries attribute information of each owner. In this embodiment, this attribute information is the name of the owner, but may be information representing other attributes.

  The subject specifying unit 308 identifies a subject in the captured image generated by the imaging unit 303 using the feature amount of each owner's face acquired by the face feature information acquisition unit 306. When this identification fails, the subject specifying unit 308 identifies the subject in the captured image generated by the imaging unit 303 again using various information described later obtained from the identification information.

  The acceleration information acquisition unit 309 includes an acceleration sensor that acquires acceleration in three axis directions, and a control unit that controls the acceleration sensor. The position information acquisition unit 311 controls the GPS, three-axis geomagnetic sensor, gyro sensor, and the like that acquire information such as latitude, longitude, azimuth, altitude, and up / down for specifying the current position of the digital camera 101. And a control unit.

  The storage unit 312 includes a RAM, a ROM, and the like, and temporarily stores identification information received by the wireless communication control unit 301 from the mobile phones 103, 105, and 107 and various information received from the server 109. Have an area. Further, the storage unit 312 has an area for temporarily storing computer programs and data to be executed by the control unit 320, and a work area used when the control unit 320 executes various processes. That is, the storage unit 312 can provide various areas as appropriate.

  The posture information acquisition unit 316 extracts a pitch angle, a roll angle, and a yaw angle from the three-axis azimuth information acquired from the position information acquisition unit 311 and the three-axis acceleration information acquired from the acceleration information acquisition unit 309. . Here, the pitch angle is a rotation angle around the X axis, the roll angle is a rotation angle around the Y axis, and the yaw angle is around the Z axis. It is a rotation angle. That is, the attitude of the digital camera 101 is calculated. Further, the posture information acquisition unit 316 obtains motion information of the digital camera 101 from the triaxial acceleration information and the roll angle, pitch angle, and yaw angle.

  The image composition unit 310 synthesizes and outputs information related to the owner identified by the subject specifying unit 308 on the captured image generated by the image capturing unit 303. The combining position is not particularly limited, and the combining position may be combined with a position in the vicinity of the image area of the owner identified by the subject specifying unit 308 or may be combined with a specified position on the captured image. The output destination of the image by the image composition unit 310 is not particularly limited, and may be output to the display unit 304 or output to the storage unit 312.

  Next, functional configuration examples of the mobile phones 103, 105, and 107 will be described using the block diagram of FIG. The wireless communication control unit 401 includes an antenna and a circuit for transmitting and receiving wireless signals to and from the digital camera 101, and a control unit that controls wireless communication by controlling them.

  The identification information transmission unit 402 transmits identification information unique to the own device to the digital camera 101 by controlling the wireless communication control unit 401. The mobile phone control unit 403 controls the operation of each unit constituting the own device, and executes each process described later as what the own device performs.

  The position information acquisition unit 404 includes a GPS and a triaxial geomagnetic sensor that acquire information such as latitude, longitude, azimuth, and altitude for specifying the position of the own device, and a control unit that controls these. The acceleration information acquisition unit 405 includes an acceleration sensor that acquires acceleration in three axis directions, and a control unit that controls the acceleration sensor.

  The posture information acquisition unit 406 extracts a pitch angle, a roll angle, and a yaw angle from the three-axis azimuth information acquired by the position information acquisition unit 404 and the three-axis acceleration information acquired by the acceleration information acquisition unit 405. . Here, the pitch angle is a rotation angle around the X axis, the roll angle is a rotation angle around the Y axis, and the yaw angle is around the Z axis. It is a rotation angle. That is, the attitude of the own device is calculated. Further, the posture information acquisition unit 406 obtains the motion information of the own device from the three-axis acceleration information and the roll angle / pitch angle / yaw angle. The biological information acquisition unit 407 includes sensors that collect biological information such as a heart rate sensor, a voice sensor, and a body temperature sensor, and acquires biological information such as the heart rate, voice, and body temperature of the owner of the own device.

  Here, a configuration example of identification information transmitted by the identification information transmission unit 402 will be described with reference to FIG. FIG. 6 shows the configuration of identification information transmitted by each of the mobile phones 103, 105, and 107 in a table format.

  The identification information includes information on an identifier (column 601), terminal position (column 602), time (column 603), movement (column 604), heart rate (column 605), body temperature (column 606), and sound (column 607). including.

  The “identifier” is an Ethernet (registered trademark) MAC address used by the wireless communication control unit 401 as a unique value for uniquely determining the owner of the mobile phone, and is appropriately set for the mobile phone. is there. The “terminal position” is a place where the owner of the mobile phone holds the mobile phone. “Time” is the time when the identification information is generated (the time is measured by the mobile phone control unit 403).

  “Movement” is information indicating the degree of movement obtained from the acceleration information obtained by the posture information acquisition unit 406 or the movement obtained from the acceleration information. In FIG. 6, “motion” indicates whether the motion represented by the acceleration information is “still”, “walking”, or “running”. For example, if the motion (acceleration) represented by the acceleration information is less than or equal to the first threshold, it is “still”, and if it is greater than the first threshold and less than or equal to the second threshold, it is “walking”. If it is larger than the second threshold, it is said that “running”.

  As shown in FIG. 10, this “movement” may be a function obtained by Fourier transforming acceleration information into a four-dimensional function with each mobile phone as the origin and the azimuth, up and down, and time as axes.

  “Heart rate” is the heart rate of the owner of the mobile phone acquired by the biometric information acquisition unit 407. “Body temperature” is the body temperature of the owner of the mobile phone acquired by the biological information acquisition unit 407. “Sound” is voice information of the owner of the mobile phone acquired by the biometric information acquisition unit 407, and this voice information may be compressed as appropriate.

  Next, information held by the server 109 will be described with reference to FIG. In the database managed by the server 109, as shown in FIG. 2, for each owner (person 102, 104, 106) of the mobile phone, a set of “identifier, name, comment, facial feature amount” for the owner is stored. It is registered.

  The “identifier” corresponds to the above “identifier” set on the mobile phone side. For example, the “identifier” of the mobile phone 103 in FIG. 6 is the same as the “identifier” of the person 102 in FIG. ing.

  “Name” is a name as attribute information of the owner of the mobile phone. However, as the attribute information of the owner of the mobile phone, various things such as an address and affiliation can be considered in addition to the name, and other attribute information may be registered. “Comment” is a comment entered as appropriate. Note that “name” and “comment” may be registered as data in text format, for example.

  The “face feature amount” is an example of an identification parameter used in the face identification process when the face of the owner of the mobile phone is shown in the image. The “face feature amount” may be registered as binary data, for example.

  Next, processing sequences by the digital camera 101, the mobile phones 103, 105, 107, and the server 109 will be described with reference to FIG. When the user of the digital camera 101 presses the shutter button 302, in step S501, the control unit 320 detects this and instructs the imaging unit 303 to start imaging. As a result, the imaging unit 303 images a subject (including the persons 102, 104, and 106) and generates a captured image. This captured image is stored in the storage unit 312 or the like.

  In step S502, the control unit 320 controls the wireless communication control unit 301 and issues a wireless signal indicating an acquisition request for identification information. More specifically, the wireless communication control unit 301 transmits a probe request frame according to IEEE 802.11 by broadcast. In the case of FIG. 1, since the mobile phones 103, 105, and 107 exist around the digital camera 101, as a result, an acquisition request for identification information is sent to the mobile phones 103, 105, and 107. Will be sent.

  As described above, the identification information may be transmitted periodically or irregularly by each mobile phone. In that case, there is no need to transmit an acquisition request for the identification information, and the digital camera 101 It is only necessary to receive the identification information transmitted from the mobile phone.

  Steps S503, S504, and S505 are processes for generating identification information in the mobile phones 103, 105, and 107, respectively. When the wireless communication control unit 401 of each of the mobile phones 103, 105, and 107 receives the identification information acquisition request, the mobile phone control unit 403 includes a position information acquisition unit 404, an acceleration information acquisition unit 405, an attitude information acquisition unit 406, The biometric information acquisition unit 407 is operated. Thereby, information included in the identification information as shown in FIG. 6 or 10 can be collected.

  In step S <b> 506, the identification information transmission unit 402 of each of the mobile phones 103, 105, and 107 controls the wireless communication control unit 401 to transmit identification information to the digital camera 101. This transmission may be each responded with a probe response frame.

  As a result, the digital camera 101 can acquire the identification information transmitted from each of the mobile phones 103, 105, and 107. The identification information is received by the identification information acquisition unit 305 controlling the wireless communication control unit 301, and the identification information acquisition unit 305 stores the received identification information of each mobile phone in the storage unit 312. .

  In step S507, the facial feature information acquisition unit 306 transmits the identifier in the identification information received from each of the mobile phones 103, 105, and 107 to the server 109, thereby determining the facial feature amount corresponding to the identifier. Make an acquisition request. Furthermore, the attribute information acquisition unit 307 transmits an identifier in the identification information received from each of the mobile phones 103, 105, and 107 to the server 109, thereby requesting acquisition of a name corresponding to the identifier.

  In step S508, when the server 109 receives the identifier transmitted in step S507, the server 109 acquires the name and facial feature quantity registered as a set with this identifier from the database managed by the server 109.

  In step S509, the server 109 transmits the name and facial feature quantity managed in the database as a set with the identifier transmitted in step S507 to the digital camera 101.

  As a result, the digital camera 101 can receive the name and facial feature amount corresponding to the identifier transmitted to the server 109. The face feature information is received by the face feature information acquisition unit 306, and the name is received by the attribute information acquisition unit 307.

  In step S <b> 510, the subject specifying unit 308 uses the facial feature amount acquired from the server 109 to determine which subject in the captured image the persons 102, 104, and 106 have generated in step S <b> 501 and stored in the storage unit 312. Identify if there is. Details of the processing in step S510 will be described later.

  In step S <b> 511, the image composition unit 310 composes the name of the owner identified by the subject specifying unit 308 on the captured image generated in step S <b> 501 and stored in the storage unit 312, and outputs it to the display unit 304. To do.

  Next, details of the processing in step S510 will be described using the flowchart of FIG. In step S <b> 701, the subject specifying unit 308 uses the facial feature amount acquired from the server 109 to indicate which subject in the captured image the persons 102, 104, and 106 have generated in step S <b> 501 and stored in the storage unit 312. To identify. Since this identification process is a general technique, a description thereof will be omitted.

  In step S702, the subject specifying unit 308 determines whether all of the persons 102, 104, and 106 have been identified from the captured image. As a result of the determination, if any one of the persons 102, 104, and 106 is not identified from the captured image, the process proceeds to step S703. If all the persons are identified, the process proceeds to step S705. In step S705, the subject specifying unit 308 determines that the subject identification process has been successful, and ends the process of the flowchart of FIG.

  In step S703, the subject identifying unit 308 has not identified any of the subjects 102, 104, and 106 among the subjects in the captured image using the information in the identification information received from each of the mobile phones 103, 105, and 107. Identification processing is performed on the subject. In this identification process, the identification process using the “movement” information in the identification information, the identification process using the biological information (“heart rate” or “body temperature”) in the identification information, and the “sound” in the identification information One of identification processing using information is executed. Details of the processing in step S703 will be described later.

  In step S704, the subject specifying unit 308 determines whether all of the persons 102, 104, and 106 have been identified from the captured image by the identification processing in step S703. As a result of this determination, if any one of the persons 102, 104, and 106 is not identified from the captured image, the process proceeds to step S706. If all the persons are identified, the process proceeds to step S705. In step S706, the subject specifying unit 308 determines that the subject identification processing has failed, and the processing of the flowchart of FIG. 7 ends.

  Next, the “identification process using the“ motion ”information in the identification information” performed in step S703 will be described with reference to FIG. 9 showing a flowchart of the process. In the following description, although the person 106 is shown in the captured image, the person 106 is in a state where the back face is directed to the digital camera 101 as shown in FIG. Assume that no face is shown.

  That is, which subject in the captured image is the person 102 using the facial feature amount corresponding to the identifier of the mobile phone 103, and which subject in the captured image is the facial feature amount corresponding to the identifier of the mobile phone 105. Whether the subject is the person 104 has been successfully identified. However, identification of which subject in the captured image is the person 106 using the facial feature amount corresponding to the identifier of the mobile phone 107 has failed.

  In step S <b> 901, the subject specifying unit 308 detects an object (humanoid object) that is estimated to be a person from the captured image (the identified humanoid object is omitted). Since this detection process can be realized by a general recognition process, a description thereof will be omitted.

  Here, the imaging unit 303 always generates a captured image just by not recording an image even if the user does not press the shutter button 302. However, in step S902, the subject specifying unit 308 detects a humanoid object from each captured image continuously generated after the processing in step S901, in the same manner as in step S901.

  In step S903, the subject specifying unit 308 performs the following process. That is, using the position on the captured image of the humanoid object detected in step S901 and the position on the captured image of the humanoid object detected in step S902, the person between the captured images whose imaging times are adjacent is used. Find the difference (movement amount) of the position of the type object.

  At this time, the mobile phone 107 may specify the movement amount of only the head of each object with respect to the humanoid object in the image because the terminal position indicates that it is the head. Further, in order to determine the movement amount, the movement amount of the humanoid object may be specified by image processing for a plurality of image data. In determining the movement amount of the humanoid object in the image, the movement and rotation of the digital camera 101 is acquired from the posture information acquisition unit 316, and the movement amount of the humanoid object affects the movement or rotation of the digital camera 101. You may consider not to be.

  In step S904, the subject specifying unit 308 obtains acceleration information of the humanoid object in the captured image from the amount of change in the movement amount when the movement amounts obtained in step S903 are arranged in time series. Then, it is specified whether the motion represented by the acceleration information is “still”, “walking”, or “running”.

  In step S905, for each humanoid object, the movement specified from the acceleration information obtained for the humanoid object is registered in the table. Here, FIG. 8 shows a configuration example of the table generated in step S905 when the humanoid objects A, B, and C are detected from the captured image.

  In FIG. 8, for each humanoid object, any one of “still”, “walking”, and “running” is registered as the movement specified from the acceleration information obtained for the humanoid object. ing.

  In step S905, the subject specifying unit 308 further matches the “movement” (“walking”) registered in the identification information as a set with the identifier of the person 106 in the table generated in step S905 ( Identify human objects that match. The identified humanoid object is identified as a person 106. In the case of FIG. 8, since the “movement” of the humanoid object C is “walking”, the humanoid object C is identified as the person 106.

  In step S904, the acceleration information is converted into vector information using the azimuth information acquired from the posture information acquisition unit 316 as the origin, the azimuth and the vertical as axes, and the azimuth, vertical and time as axes. Fourier transform may be performed on the four-dimensional function. In this case, the configuration of the table generated in step S905 is as shown in FIG.

  The table in FIG. 11 is a table generated in step S905 when the humanoid objects A, B, and C are detected from the captured image, and is generated from the acceleration information obtained for the humanoid object for each humanoid object. The absolute value of the obtained four-dimensional function is registered.

  In this case, in step S905, the subject specifying unit 308 matches the absolute value of the four-dimensional function registered in the identification information as a set with the identifier of the person 106 in the table generated in step S905. Is identified. The identified humanoid object is identified as a person 106.

  Next, the “identification process using the biological information in the identification information” performed in step S703 will be described. A table having a configuration illustrated in FIG. 12 is registered in the storage unit 312. In this table, a set of heart rate and body temperature for each of the movement states of “still state”, “walking state”, and “running state” is registered. This set may be measured in advance or may be created by the user.

  In this case, the subject specifying unit 308 generates, for each humanoid object as illustrated in FIG. 8 according to the flowchart of FIG. 9, a table in which the movement specified from the acceleration information obtained for the humanoid object is registered. .

  Next, the subject specifying unit 308 specifies the movement state corresponding to the “heart rate” registered in the identification information as a set with the identifier of the person 106 from the table of FIG. In the case of FIG. 6, since the “heart rate” registered in the identification information as a set with the identifier of the person 106 is 120, the motion state corresponding to this heart rate is “walking state” from the table of FIG. It becomes. In the table of FIG. 8, since the humanoid object corresponding to this “walking state” is the humanoid object C, the humanoid object C is identified as the person 106.

  The subject specifying unit 308 may perform the determination process using “body temperature” instead of the heart rate. In this case, the movement state corresponding to the “body temperature” registered in the identification information as a set with the identifier of the person 106 is specified from the table of FIG. In the case of FIG. 6, since “body temperature” registered in the identification information as a set with the identifier of the mobile phone 107 is 37.5, the movement state corresponding to this body temperature is “walking state” from the table of FIG. " In the table of FIG. 8, since the humanoid object corresponding to this “walking state” is the humanoid object C, the humanoid object C is identified as the person 106.

  Note that information other than “heart rate” and “body temperature” may be used as the biological information. In this case, it is necessary to register biological information corresponding to each movement state in the table of FIG. However, it is necessary to include biometric information corresponding to the identification information.

  Further, as shown in FIG. 1, the subject 104 is specified when the person 104 is located relatively far from the digital camera 101, and thus the resolution necessary for obtaining the facial feature determination cannot be secured by image processing. How to do it.

  Since the person 104 is located far away, the subject cannot be specified in step S701. As described with reference to FIG. 7, subject identification in step S <b> 701 is performed for the person 104. The person 104 can apply any of the above-described subject determination based on the motion state, subject determination based on the acceleration, and subject determination based on the biological state. However, unlike the person 106, the person 104 faces the digital camera 101. Subject determination based on audio information is also possible. In the following description, it is assumed that the identification of which subject in the captured image is the person 104 using the facial feature amount corresponding to the identifier of the mobile phone 105 has failed.

  In this case, the shape of the mouth is recognized from the captured image for each humanoid object, and the sound (movement of the mouth) generated from the recognized mouth shape is recognized. Since this recognition process is also a well-known technique, a description thereof will be omitted. For each humanoid object, information indicating the sound generated by the humanoid object is registered in the table. Here, FIG. 13 shows a configuration example of a table generated when the humanoid objects A, B, and C are detected from the captured image.

  In FIG. 13, information indicating the sound generated by the humanoid object is registered for each humanoid object. For the humanoid object A, “n” is displayed as information indicating the sound, and the humanoid object B Is registered as information indicating sound. For the humanoid object C, “unknown” is registered because information indicating a sound was not obtained because the recognition process failed.

  Then, the subject specifying unit 308 specifies a humanoid object that matches the “sound” (“A”) registered in the identification information as a set with the identifier of the person 104 from this table. Then, the identified humanoid object is identified as the person 104. In the case of FIG. 13, since the “sound” of the humanoid object B is “A”, the humanoid object B is identified as the person 104.

  Each of the functional units shown in FIGS. 3 and 4 may be configured with hardware, but may be partially configured with software (computer program). In this case, the computer program is stored in the storage unit 312 or the memory of the mobile phone.

  As described above, a specific user is identified using not only the result obtained by image processing but also sensor information obtained by sensing the user with a sensor, so that the identification accuracy of the specific user is improved. To do. Therefore, it is possible to increase the possibility that information related to a specific user can be presented in the vicinity of the specific user in the captured image.

[Second Embodiment]
In the following, only the differences of the present embodiment from the first embodiment will be described. In the present embodiment, infrared image data is used in step S703 described above. However, it is necessary to add a configuration for acquiring infrared image data to the digital camera 101 according to the present embodiment.

  In the following, although the person 106 is reflected in the captured image, the person 106 is in a state where the back face is directed to the digital camera 101 as shown in FIG. Assume that no face is shown.

  That is, which subject in the captured image is the person 102 using the facial feature amount corresponding to the identifier of the mobile phone 103, and which subject in the captured image is the facial feature amount corresponding to the identifier of the mobile phone 105. Whether the subject is the person 104 has been successfully identified. However, identification of which subject in the captured image is the person 106 using the facial feature amount corresponding to the identifier of the mobile phone 107 has failed.

  In this case, the subject specifying unit 308 detects the humanoid object from the image captured by the imaging unit 303 as described above, and further processes the temperature information of the humanoid object from the thermographic data acquired by performing image processing on the separately acquired infrared image data. To get. Then, as illustrated in FIG. 14, for each humanoid object, a table in which the temperature information of the humanoid object is registered is generated. However, in FIG. 14, the humanoid object having the body temperature closest to the “body temperature” (37.5) registered in the identification information as a set with the identifier of the mobile phone 107 is the humanoid object B (37.6). Therefore, it is identified that the humanoid object B is the person 106.

[Third Embodiment]
In the above embodiment, the process is performed on a still image. However, this process may be performed on each frame constituting a moving image.

  Further, communication related to transmission and acquisition of identification information is not limited to wireless LAN communication conforming to IEEE 802.11, but may be Bluetooth (registered trademark), passive / active RFID, or the like. Communication regarding identification information may be simultaneously performed by a plurality of wireless communication interfaces such as a wireless LAN and a passive RFID. Further, the identification information may be transmitted and acquired using a directional wireless system such as millimeter wave.

  In the first and second embodiments, each identifier is associated with a person. However, the identifier does not have to be a person, and may be associated with an animal, a car, or a specific object.

  In the first and second embodiments, the method of subject determination based on the motion state, subject determination based on the acceleration, subject determination based on the biological state, and subject determination based on the audio information have been described. Also good. That is, when the time information of the image data obtained from the imaging unit 303 and the time of the identification information notified by the mobile phones 103, 105, and 107 are not close to each other, the object match determination may be determined as “No”. Good.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (8)

Means for acquiring a captured image;
Receiving means for receiving sensor information representing the state of the user obtained by sensing the user with a sensor;
Acquisition means for acquiring the attribute information of the user and an identification parameter for identifying the user in the captured image;
An identification unit for identifying the user from among subjects in the captured image based on the sensor information received by the reception unit and the identification parameter acquired by the acquisition unit;
An image processing apparatus comprising: output means for combining and outputting the attribute information acquired by the acquisition unit at a position on the captured image corresponding to a result of identification by the identification unit.   The image processing apparatus according to claim 1, wherein the output unit synthesizes and outputs the attribute information in the vicinity of a position in the captured image of the user identified by the identification unit. When the identification unit cannot identify the user from the captured image using the identification parameter acquired by the acquisition unit,
Obtains a motion of the subject in the captured image, according to claim 1 or 2 subject movement matching the motion of the user which the receiving means is included in the received sensor information, wherein the identifying as the user An image processing apparatus according to 1. When the identification unit cannot identify the user from the captured image using the identification parameter acquired by the acquisition unit,
The movement of the subject in the captured image is obtained, and the subject having the movement that matches the movement previously associated with the biological information of the user included in the sensor information received by the receiving unit is identified as the user. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. When the identification unit cannot identify the user from the captured image using the identification parameter acquired by the acquisition unit,
The movement of the mouth of the subject in the captured image is obtained, and the subject having the mouth movement that matches the movement of the user's mouth included in the sensor information received by the receiving unit is identified as the user. The image processing apparatus according to claim 1 or 2. When the identification unit cannot identify the user from the captured image using the identification parameter acquired by the acquisition unit,
An infrared image is acquired, a temperature of a subject is obtained from the infrared image, and a subject having a temperature closest to the user's temperature included in sensor information received by the receiving unit is identified as the user. The image processing apparatus according to claim 1. An image processing method performed by an image processing apparatus,
Obtaining a captured image;
A receiving step of receiving sensor information representing the state of the user obtained by sensing the user with a sensor;
An acquisition step of acquiring the attribute information of the user and an identification parameter for identifying the user in the captured image;
An identification step for identifying the user from among subjects in the captured image based on the sensor information received in the reception step and the identification parameter acquired in the acquisition step;
An image processing method comprising: an output step of combining and outputting the attribute information acquired in the acquisition step at a position on the captured image corresponding to a result of identification in the identification step.   A computer program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 6.

Images