JP4501531B2 - Image recognition apparatus and program - Google Patents

Description

  The present invention relates to an image recognition device and a program for recognizing a subject by comparing a subject portion included in a photographed image with a registered image registered in advance for subject recognition.

Conventionally, as a technique for analyzing a photographed image taken by a camera and extracting a person portion shown in the image and recognizing who the person is, for example, features such as eyes and nose without being affected by changes in luminance There has been known a person authentication device that extracts a point stably and performs person authentication with high accuracy (see Patent Document 1). There is also known an image processing apparatus that displays information on the name or photographing year of a person whose image has been recognized in association with the image (see Patent Document 2).
JP 09-251534 A JP 2003-150932 A

However, in the above-described techniques of Patent Documents 1 and 2, since it is necessary to compare all images (dictionary images / reference images) to be compared with the person portion in the captured image, The larger the number of images, the longer the recognition time and the lower the recognition efficiency, and the more often the images to be compared are similar to each other, leading to erroneous recognition and a reduction in recognition rate. It was.
By the way, when a plurality of persons are group-photographed, each person who is a subject often has a common human relationship with the photographer, such as a family relationship, a friendship relationship, a company relationship, or a relative relationship. There is a high possibility that the persons are associated with each other through the photographer.

An object of the first invention is to make it possible to recognize each subject efficiently in consideration of the relationship between the subjects when individually recognizing a plurality of subjects included in the captured image. is there.
An object of the second invention is to make it possible to accurately recognize each subject in consideration of the relationship between the subjects when individually recognizing a plurality of subjects included in the captured image. is there.

The invention according to claim 1 (second invention) recognizes the face image by comparing the face image portion included in the photographed image with a registered image registered in advance for face image recognition. An attribute information setting unit for setting at least one attribute information indicating an attribute of the face image corresponding to a registered image registered for face image recognition in the image recognition device; In the process of sequentially specifying a plurality of face images included in the captured image and individually recognizing each of them, a plurality of recognition candidates are obtained for each face image, and a plurality of face images obtained for each face image are obtained. A recognition means for determining a recognition candidate set with attribute information common to each face image from among recognition candidates as a recognition result of each face image; a registered image registered for the face image recognition; and the at least one Attribute information and email Storage means for associating and storing a response and transmission means for transmitting the captured image, wherein the transmission means is associated with each face image of a recognition candidate determined by the recognition means. The photographed image is transmitted to a destination .
Furthermore, a program for realizing the main functions shown in the above-described invention according to claim 1 is provided to the computer (the invention according to claim 3 ).
The invention described in claim 1 may be as follows.
A mode setting unit configured to set a transmission mode in which the captured image can be transmitted. In this case, when the transmission mode is set by the mode setting unit, the recognition unit includes a plurality of modes included in the captured image. The face images are sequentially designated and recognized individually (the invention according to claim 2 ).

According to the present invention, when a plurality of subjects included in a captured image are individually recognized, each subject can be recognized while also considering the relevance between the subjects. In addition, when sending captured images to e-mails and sending them to recipients, you can send them properly to the recipients, checking each photographed image for each destination. This eliminates the troublesome task of searching for images and prevents forgetting to send images.

(First embodiment)
A first embodiment of the present invention will be described below with reference to FIGS.
FIG. 1 is a block diagram showing basic components of a camera-equipped mobile terminal in this embodiment.
This camera-equipped mobile terminal is a mobile terminal such as a camera-equipped mobile phone, PDA, etc., and has a function of sending a photographed image by e-mail via a network (terrestrial / satellite mobile communication network, Internet network) have. In this case, digitized (packetized) captured images are transmitted / received between the camera-equipped mobile terminal and the destination terminal by the HTTP protocol using the TCP / IP communication protocol.
Prior to detailed description of the features of this embodiment, the hardware configuration of this embodiment will be described below.

  The CPU 1 is a central processing unit that controls the overall operation of the camera-equipped mobile terminal in accordance with the operating system and various application software in the storage device 2. The storage device 2 has a program storage area and a data storage area, and is configured by a drive system such as a magnetic, optical, or semiconductor memory. The recording device 2 may have a configuration in which a removable storage medium such as a CD-ROM or a DVD can be mounted in addition to a fixed memory such as a hard disk. The program area in the storage device 2 stores an application program for realizing the present embodiment in accordance with the operation procedure shown in FIGS. 5 to 8 described later, and the data area includes a diagram described later. 2 to 4 shown in FIG. 4 (person database, image file, delivery database) are stored. The program and data in the storage device 2 are loaded into a RAM (for example, static RAM) 3 as necessary, or the data in the RAM 3 is saved in the storage device 2. The RAM 3 has a program execution area and a work area.

  On the other hand, the communication device 4, the input device 5, the display device 6, and the camera device 7, which are input / output peripheral devices, are connected to the CPU 1 through the bus line, and the CPU 1 performs these operations according to the input / output program. Control. In addition to various buttons such as a shutter button that are normally provided, the input device 5 includes a special mode button, an image registration button, a transmission setting button, a transmission button, and the like described later as buttons unique to this embodiment. . The camera device 7 is a digital still camera that uses a CCD image sensor or the like as an image sensor, and a photographed image (color image) photographed by the camera device 7 is stored in the storage device 2 as well as an e-mail. And sent to the destination terminal.

FIG. 2 is a diagram showing the contents of each person registration record RT constituting the person database.
The person registration record RT stores and manages a subject image and a destination in association with each subject (person) registered in advance. For each person, a “person image”, “name”, and “email address” are stored. "," Person ID ", and" person attribute ". The “person image”, “name”, and “mail address” are, for example, arbitrarily read from a destination mail address book (not shown) and set in the person registration record RT. Is a face image taken centering on the face of a person, and is a registered image registered in advance for each person for person recognition. “Person ID” is person identification information (registration record identification information) issued each time a person registration record RT is generated. “Person attribute” indicates the human relationship between the user (photographer) and the person who is the subject. The attribute value “00” is a kinship relationship, the attribute value “01” is a school age relationship, and the attribute value “ “02” indicates a company relationship, and its attribute value “03” indicates a business relationship. In this case, the present invention is not limited to registering any one of the attribute values. For example, two or more attribute values may be registered such as a school age relationship or a business relationship.

FIG. 3 shows the contents of the image file GF, and FIG. 4 shows the contents of the delivery record AD constituting the delivery database.
The image file GF stores and manages each captured image in the form of one image and one file. The image file GF is configured to store and manage each captured image in association with a shooting number (series No.) updated for each shooting. ing. The sent record AD is used to classify and manage each captured image by person ID, and is configured to store a plurality of “shooting Nos” corresponding to “person ID”. Here, the camera-equipped mobile terminal recognizes whether or not the “person image” registered in the person registration record RT is included in the photographed image by analyzing the photographed image, and the registered image is included. If so, the “shooting No” is set in the delivery record AD in association with the person ID. Further, the image file GF is accessed based on each “imaging No” in the sending record AD to read out the corresponding photographed image, and the person registration record RT is accessed based on the “person ID” in the sending record AD. The corresponding “mail address” is read out, and each photographed image is sent to this mail address.

  Next, the operation concept of the camera-equipped mobile terminal in the first embodiment will be described with reference to the flowcharts shown in FIGS. Here, each function described in these flowcharts is stored in the form of a readable program code, and operations according to the program code are sequentially executed. It is also possible to sequentially execute operations according to the program code transmitted via the transmission medium. The same applies to other embodiments to be described later. In addition to a recording medium, an operation peculiar to this embodiment can be executed using a program / data supplied externally via a transmission medium.

FIG. 5 to FIG. 8 are flowcharts showing processing relating to the time of camera photographing and the time of sending a photographed image mail.
First, the CPU 1 checks the presence / absence of a button operation for shifting from the normal shooting mode to the special mode (step S1 in FIG. 5). If the special mode button is operated, the CPU 1 checks the presence / absence of other button operations. (Step S2). In this special mode, setting / reset is repeated each time the special mode button is operated. In the special mode, when any of the image registration button, transmission setting button, and transmission button is operated ( If YES in step S2, the type of this operation button is determined, and if it is an image registration button (step S3), the contents of the destination mail address book are displayed in a list (step S4).

  When an arbitrary person is selected and designated from the list display (step S5), the “person image”, “name”, and “mail address” corresponding to the designated person are read from the address book (step S6). The “person ID” is generated and issued (step S7), and a new person registration record RT including the “person image”, “name”, “mail address”, and “person ID” is generated and stored (step S7). S8). Thereafter, until the end of registration is instructed (step S9), the above-described registration record generation process is repeated for each record (steps S5 to A9).

  When the transmission setting button is operated in the state where the registration records corresponding to persons are generated and managed in this way (step S10), it is checked whether or not the transmission mode is currently set (step S11). ). This transmission mode is an operation mode in which a captured image is set in a transmittable state, and the set / reset is repeated each time the transmission setting button is operated. If the transmission setting button is operated while the transmission mode is set, the transmission mode is reset (step S13). However, the transmission setting button is operated when the transmission mode is reset. If so, the transmission mode is set (step S12).

  On the other hand, when the shutter button is operated in the state where the above-described special mode is released (reset) (step S14), image shooting is performed in response thereto (step S21 in FIG. 6). No is updated and a new shooting number is issued (step S22), and the shot image with the shooting number added is stored as an image file GF (step S23). Then, it is checked whether or not the transmission mode is set (step S24). If it is reset, the process returns to the first step S1 in FIG. 5, but if the transmission mode is set, this captured image is displayed. By analyzing (step S25), the face portions of all persons included in the photographed image are extracted (step S26).

  Then, the process of recognizing each person individually while sequentially specifying the face portions of all the persons extracted as described above is performed, but prior to that, the process for specifying the person first designated as the recognition target is performed. (Steps S27 to A33). First, all the extracted face parts are evaluated, and a good face image (person) excluding those with extremely bad face orientation, face hidden state, etc. is selected (step S27). Then, it is determined whether or not a face image is located at the center of the photographed image among the selected face images (step S28). If there is a face image at the center of the photographed image (at step S29). (YES), the person of this face image is designated as the first recognition target (step S30).

  If there is no face image at the center of the photographed image (NO in step S29), the sizes of the selected face images are compared to check whether there is a significant difference in the sizes (step S31). Here, if there is a difference in the size of each selected face image, the face image of the maximum size is selected from them (step S32), and the person of the face image is designated as the first recognition target ( Step S30) If the size of each face image is substantially the same (NO in Step S31), the face image with the best reflection is selected from the selected face images (Step S33), and the face image is selected. A person is designated as the first recognition target (step S30).

When the person as the first recognition target is designated in this way, each person registration record RT constituting the person database is searched based on this face image and compared with each “person image” as the registered image. (Step S34). In this case, after the correct complement the orientation of the face image, the size, the face contour, eyes, mouth, nose, the shape of the forehead and the like, the position, size, hairstyle, comparing separately the presence or absence and type of eyeglasses In addition, image comparison is performed by comprehensively comparing them (hereinafter the same). As a result, when it is recognized that the two persons are characteristically matched and the person is specified, the “person ID” and the “person attribute” associated with the registered image are read from the person registration record RT. Then, “person ID” and “person attribute” are specified (step S35).

  Next, the process proceeds to step S36 in FIG. 7, and it is checked whether or not there is an unspecified face image among the face images extracted from the photographed image. The first is because the first face image is specified. Any one of the undesignated face images is designated as the second recognition target (step S38). Then, person recognition is performed based on the designated face image. At the time of second and subsequent person recognition, person recognition is performed in consideration of the person attributes obtained at the time of the first person recognition described above (step S39). That is, each person registration record RT that constitutes the person database is searched based on the person attribute obtained at the time of the first person recognition, each registered image having the same person attribute is selected as a comparison target, and within this selection range The person recognition is performed by comparing each of the registered images with the face image designated second. In other words, instead of comparing all the registered images in the person database, refer to the comparison target within the range narrowed down based on the person attribute (human relation) obtained at the time of the first person recognition. Perform person recognition.

  As a result, when any registered image within the range narrowed down based on the human relationship of the first person matches the second designated face image at a predetermined level or higher, and the person is specified ( “YES” in step S40), the “person ID” associated with the registered image is read from the person registration record RT to identify the “person ID” (step S41). On the other hand, if each registered image within the narrowed range does not include a person that matches the second designated face image (NO in step S40), it is the same as usual (same as the first person). ), The person recognition is performed using all the registered images in the person database as comparison targets (step S42). As a result, when the person is specified, the “person ID” associated with the registered image is registered as the person. The “person ID” is specified by reading from the record RT (step S41).

  Thereafter, returning to step S36, the presence or absence of an undesignated face image is checked, and the above-described operation is repeated until all face images are designated. Here, when all the face images have been designated, the shooting number is set in the person-corresponding sending record AD for each specified person ID (step S37). Thereafter, the process proceeds to the first step S1 in FIG. In this case, if a delivery record AD corresponding to the specified person exists (if it has been created), this shooting number is additionally registered in the corresponding record, but if the corresponding record AD does not exist, it is specified. A new delivery record AD corresponding to the person is created.

  On the other hand, when the send button is operated in the state where the special mode is set (step S51 in FIG. 8), one sending record AD is designated, and the “person ID” in the sending record AD is designated. The person registration record RT is accessed based on the above, and the corresponding “mail address” is read and set as the mail destination (step S52). Next, each “shooting No” is read out from the designated sending record AD, the corresponding image file GF is acquired based on this “shooting No” (step S53), and each shot image file is converted into a compressed file. Is attached to the mail (step S54). Then, after sending the mail (step S55), the sent delivery record AD is deleted (step S56). Thereafter, if there is another delivery record AD (YES in step S57), the above-described delivery process is repeated while sequentially specifying the delivery record AD (steps S51 to A57).

 As described above, in the camera-equipped mobile terminal according to the first embodiment, the CPU 1 performs the camera device 7 in a state where the person attribute is stored and managed corresponding to the registered image (person face image) in the person database. In the process of sequentially specifying a plurality of persons included in a captured image captured by, and recognizing each individually, the attribute corresponding to this person is specified and recognized. The registered image having the same attribute is preferentially selected as a comparison target based on the image, and the person is recognized by comparing each registered image within this selection range with another person. When recognizing multiple persons individually, the comparison target can be narrowed down based on the human relationship between each person (family relationship, friendship relationship, company relationship, relative relationship, etc.), and the number of registered images increases. Even if it is large, it becomes possible to recognize each person efficiently.

  In this case, the CPU 1 first designates a person included in the central portion of the photographed image from among a plurality of persons included in the photographed image, or selects a plurality of persons contained in the photographed image. Among them, since the person with the best size or the best image quality is designated first, it becomes possible to easily specify the person serving as the reference when judging the relevance between the persons.

  The name and email address corresponding to the recognized person is acquired and sent by mail in the state where the name and email address of this person are stored and managed in association with the registered image in the person database. Therefore, when you send a photographed image attached to an e-mail and send it to the other party, you can send the photographed image to the other party properly, and send each while confirming what each photographed image shows. This eliminates the troublesome work of searching for a destination and prevents forgetting to send an image.

  In the above-described first embodiment, kinship relations, school age relations, company relations, business relations, etc. have been exemplified as the attribute values. More detailed relationships such as circle relationships may be registered, or the relationships may be associated in a hierarchical structure. In this case, the comparison target may be selected step by step from the lower hierarchy to the upper hierarchy.

  In the first embodiment described above, a person is exemplified as a subject. However, for example, a plurality of pets, a plurality of buildings in a sightseeing spot, or the like may be used. For example, a person and a pet may be used. Further, after recognizing a plurality of subjects in the photographed image, the recognition result may be guided and displayed as a selection candidate.

On the other hand, a recording medium (for example, a CD-ROM, a flexible disk, a RAM card, etc.) on which program codes for executing the above-described units are recorded may be provided to the computer. That is, a recording medium having a computer readable program code, storing and managing attribute information indicating the attribute of the subject corresponding to a registered image registered for each subject for subject recognition, and shooting In the process of sequentially specifying a plurality of subjects included in an image and individually recognizing each of them, when a certain subject is specified and recognized, the same is based on attribute information corresponding to this subject A registered image having an attribute is selected as a comparison target, and a program for realizing a function of recognizing the subject by preferentially comparing each registered image within the selection range with another subject portion is recorded. A computer-readable recording medium may be provided.
(Example 2)

The second embodiment of the present invention will be described below with reference to FIGS.
In the first embodiment described above, when the first person is designated and recognized, a registered image having the same attribute is preferentially selected as a comparison target based on the attribute corresponding to this person. In this second embodiment, when a plurality of persons included in the captured image are individually recognized, each person is selected even if there are many similar registered images. It is designed to be recognized with high accuracy. That is, in the second embodiment, a plurality of recognition candidates are obtained for each person, and among the plurality of recognition candidates obtained for each person, recognition candidates having attributes common to each person are assigned to each person. Each person can be accurately recognized by determining the recognition result.
Here, the same or the same names in both embodiments are denoted by the same reference numerals, the description thereof will be omitted, and the following description will focus on the features of the second embodiment. .

FIG. 9 is a diagram illustrating a plurality of recognition candidates obtained for each person.
In this case, a case where a plurality of person IDs and their attributes corresponding to the persons A, B, and C are obtained as recognition candidates is shown. That is, when recognizing a plurality of persons individually, a plurality of registered images having high similarity for each person are identified as recognition candidates, and the person ID and person attribute corresponding to each registered image are read from the person registration record RT. Is. A person ID having a common person attribute is selected from the recognition candidates corresponding to each person and determined as a recognition result of each person. For example, in the example of FIG. 9, since the person attribute common to the persons A, B, and C is “001”, the person ID associated with the person attribute “001” is determined as the recognition result of each person. The

FIG. 10 is a flowchart showing the characteristic part of the second embodiment, and is a flowchart replacing FIG. 6 in FIGS. 5 to 8 described above. FIGS. Since this embodiment is the same as the embodiment, the characteristic portion of the second embodiment will be described below with reference to FIG.
First, when the shutter button is operated in a state in which the special mode is canceled (reset) (step S14 in FIG. 5), the CPU 1 performs image capturing in response thereto (step in FIG. 10). In step S61, the shooting number is updated to issue a new shooting number (step S62), and the shot image with the shooting number added is stored as an image file GF (step S63). Then, it is checked whether or not the transmission mode is set (step S64). If it is reset, the process returns to the first step S1 in FIG. 5, but if the transmission mode is set, this captured image is displayed. By analyzing (step S65), the face portions of all persons included in the photographed image are extracted (step S66).

  Then, processing for recognizing each person individually is performed while sequentially specifying the face portions of all the extracted persons. First, after designating any one of the extracted face parts of each person as a recognition target (step S67), the person registration record RT constituting the person database is searched based on the face image. Then, each “person image” as the registered image is compared (step S68). In this case, it is checked whether or not there is a registered image having a matching degree (similarity) that is characteristically equal to or higher than a predetermined level. If there is no corresponding similar image (NO in step S69), the next step S71 is performed. If there is at least one corresponding similar image, the three person IDs and person attributes are read from the person registration record RT in the descending order of the degree of coincidence (similarity) and specified as the candidate for recognition of the person. (Step S70).

  Next, it is checked whether there is an undesignated face image (step S71). Since the first face image is designated first, any one of the undesignated face images is selected as 2 The second recognition target is designated (step S67). Thereafter, the above-described operation is repeated until all face images are designated (steps S67 to A71). Here, when all the face images have been designated, among the plurality of person IDs (recognition candidates) specified for each person, a person attribute common to each person is specified (step S72), and this A person ID corresponding to the person attribute is specified (step S73). For each person ID specified in this way, the photographing number is set in the person-corresponding delivery record AD (step S74). Thereafter, the process proceeds to the first step S1 in FIG.

As described above, in the camera-equipped mobile terminal according to the second embodiment, the CPU 1 performs the camera device 7 in a state where the person attribute is stored and managed corresponding to the registered image (person face image) in the person database. In the process of sequentially specifying a plurality of persons included in the photographed image captured by the method and recognizing each individually, a plurality of recognition candidates are obtained for each person, and the plurality of persons obtained for each person are obtained. Recognition candidates with attributes common to each person are determined as recognition results for each person, so that the human relationships common to each person (family relationship, friendship relationship, company relationship, relative relationship, etc.) ) Can be recognized, and each person can be accurately recognized even if there are many similar registered images.

  In the second embodiment described above, the attribute value is not limited to a kinship relationship, a school age relationship, a company relationship, a business relationship, etc. For example, a college relationship, a high school age friend relationship, a circle relationship, etc. A more detailed relationship may be used. The subject is not limited to a person, and may be a plurality of pets or a plurality of buildings in a sightseeing spot. The plurality of subjects in a captured image are not limited to people and pets. And may be a pet.

 On the other hand, a recording medium having a computer-readable program code, storing and managing attribute information indicating the attribute of the subject corresponding to a registered image registered for each subject for subject recognition, and shooting In the process of sequentially specifying a plurality of subjects included in an image and individually recognizing each of them, a plurality of recognition candidates are obtained for each subject, and among the plurality of recognition candidates obtained for each subject From the above, it is possible to provide a computer-readable recording medium in which a program for realizing a function for determining a recognition candidate having attribute information common to each subject as a recognition result of each subject is recorded.

In addition, in the first and second embodiments described above, each time a shutter operation is performed, the captured image is analyzed to recognize the image and create the delivery record AD. The sending record AD is not created every time the shutter operation is performed. For example, the stored captured images may be sequentially read and executed in the idle time when the shutter is not operated.
In the first and second embodiments described above, one type of image is registered corresponding to one person in the registration record RT. In order to improve the recognition rate at the time of image comparison, shooting is performed. A plurality of images having different directions may be registered.

In the first and second embodiments described above, the photographed image is transmitted by attaching it to an e-mail. However, the color-printed photographed image may be fax-transmitted by a facsimile apparatus, and the transmission form is as follows. It doesn't matter. The photographed image to be sent is not limited to a still image, and may be a moving image.
In the first and second embodiments described above, a case where the present invention is applied to a camera-equipped mobile terminal has been exemplified. However, each captured image captured by another camera device is received and acquired and transmitted to a destination. It may be an image transmission device.

The block diagram which showed the basic component of the portable terminal with a camera. The figure which showed the content of the registration record RT which comprises a registration database. The figure which showed the content of the image file GF. The figure which showed the content of the delivery record AD which comprises a delivery database. The flowchart which showed the process regarding the mail transmission of camera imaging | photography and a captured image. The flowchart following FIG. The flowchart following FIG. The flowchart following FIG. The figure which illustrated a plurality of recognition candidates obtained for every person in a 2nd embodiment. The flowchart which shows the characteristic part of 2nd Embodiment.

Explanation of symbols

1 CPU
2 storage device 4 communication device 5 input device 6 display device 7 camera device RT registration record GF image file AD sending record

Claims (3)

An image recognition device for recognizing a face image by comparing a face image portion included in a captured image with a registered image registered in advance for face image recognition,
Attribute information setting means for the user to set at least one attribute information indicating the attribute of the face image corresponding to the registered image registered for face image recognition;
In the process of sequentially specifying a plurality of face images included in the captured image and individually recognizing each of them, a plurality of recognition candidates are obtained for each face image, and a plurality of face images obtained for each face image are obtained. A recognition means for determining a recognition candidate set with attribute information common to each face image from recognition candidates as a recognition result of each face image;
Storage means for storing the registered image registered for the face image recognition, the at least one attribute information, and an email address in association with each other;
Transmission means for transmitting the captured image,
The image recognizing apparatus , wherein the transmitting unit transmits the captured image to the mail address associated with each face image of the recognition candidate determined by the recognizing unit. A mode setting means for setting a transmission mode capable of transmitting a photographed image is provided.
The recognition means is
When the transmission mode by said mode setting means is set, according to claim 1, wherein the recognizing each sequentially specifying the plurality of face images contained in the captured image individually Image recognition device. Against the computer,
A function for the user to set at least one attribute information indicating the attribute of the face image corresponding to the registered image registered for face image recognition;
In the process of sequentially specifying a plurality of face images included in the captured image and individually recognizing each of them, a plurality of recognition candidates are obtained for each face image, and a plurality of face images obtained for each face image are obtained. A function for determining a recognition candidate in which attribute information common to each face image is set from among recognition candidates as a recognition result of each face image;
A function of storing the registered image registered for the face image recognition, the at least one attribute information, and an email address in association with each other;
A function of transmitting the captured image to the e-mail address associated with each face image of the determined recognition candidate;
A program to realize

Images