JP5180116B2 - Nationality determination device, method and program - Google Patents

Description

  The present invention relates to a technique for providing information according to attributes of a target person.

  In Patent Document 1, a person attribute identification model for individually identifying these data corresponding to a plurality of types of person attribute identification data is prepared in advance. After creating a plurality of types of person attribute identification data from the frame image to be processed obtained by shooting the image, the index value calculation processing unit 37 uses the person attribute identification model to estimate each person attribute identification data. Index values such as degrees are calculated individually, and thereafter, a plurality of index values are integrated by the identification result information calculation processing means.

  In Patent Document 2, the face recognition unit shoots a person in front of the camera, detects a face portion from the image, and the detected face image includes age, sex, occupation, nationality, birthplace, facial expression, etc. With respect to the attribute, it is determined which category the facial features are similar to. The production control unit is responsible for overall control of the entire system, such as input processing related to operations from visitors including people, management of visitor history information files, ranking aggregation, transmission of shooting commands to the face recognition unit, and email transmission. The display of instructions regarding operations to visitors, the display of attribute determination results, and the presentation display control such as ranking display are performed. The visitor information storage unit stores history information files and photographed face image files created for each visitor in the storage area.

  Patent Documents 3 and 4 are examples of conventional multilingual speech recognition systems. Patent Document 5 is an example of conventional clothing recognition.

  Patent Document 6 is an example of a technique for detecting facial feature points, and a first feature point detector having a high tolerance generated by machine learning for a plurality of types of feature points of a predetermined object on a detection target image. The plurality of types constrained by the positional relationship using a group and a first positional relationship model having a high tolerance that defines the positional relationship between the multiple types of feature points generated by statistical learning After tentatively determining the feature points, a second feature point detector group having a low tolerance generated by machine learning in the vicinity of each tentative feature point and a statistical learning, Using the second positional relationship model having a low tolerance, the plurality of types of final feature points constrained by the positional relationship are determined.

  Patent Document 7 is an example of extraction of a human body region, detects a face region F in an image, determines a candidate region C that will include a human body region from position information of the detected face region F, and It is determined whether or not each unit area constituting the candidate area includes a human body area, and a set of unit areas determined to include the human body area is determined as an inferred area E that is estimated to include the human body area. The human body region Hu in the estimation region E is extracted, and the estimation region E is expanded and updated until it is determined that the human body region Hu does not exist in the contour peripheral region in the estimation region E. Repeatedly extracting the human body region.

  Patent Document 8 is an example of face extraction, and extracts the shape of a face, the contours of eyes and mouth according to the degree of matching with a template.

  Patent Document 9 is an example of pupil region extraction. Eyes are extracted using edge detection, shape pattern detection, position information, and the like, and a low luminance region is extracted from the luminance histogram of the eye image data. The brightness area is contracted to extract the pupil area.

  Patent Document 10 is an example of nose shape extraction, and a template having a predetermined shape is matched on an edge video in a predetermined search area. Among the detected candidates having a large matching value, a pair satisfying the symmetry condition is selected and determined as the nose wing surface position. Each nasal flank is represented by a polynomial curve, which matches the detected nasal wing surface template and three points interpolated between the nasal wing surface and the corners of the eyes by a predetermined factor. Eventually, the nose tip and the lower nose curved surface are located using a predetermined interpolation coefficient.

  Patent Document 10 is an example of mouth shape extraction. First, a mouth square is initialized. The initial square is reduced to a more precise bounding box by analyzing the moments of non-skin color pixels. A lip function image is constructed to measure the probability that a pixel belongs to the lips or skin using the pixels in the refined bounding box. Initialize the outline of the lips with an ellipse by measuring the secondary central moment of the pixel with a high lip function value. The outline of the lips is dynamically moved by external force and internal force. A curve expression is generated by approximating a polynomial to the movement result point.

  Patent Document 11 shows an example of hairstyle detection, and a hairstyle is identified by matching a hairstyle model shape pattern with an image.

  Patent Document 12 shows an example of estimating age from a face detected from an image, and a reference feature vector for each reference face image captured from a plurality of different face orientation directions for a plurality of reference persons having different genders and ages. Among them, those similar to the target feature vector of the target face image obtained by capturing the face of the customer whose sex and age are unknown from any face direction are determined. The range in the face direction in which the reference face image is captured with respect to the reference feature vector is estimated to be the range in the face direction in which the target face image is captured. Further, by determining a feature vector similar to the target feature vector from the feature vectors of the reference face image captured in the estimated face orientation direction, the reference person of the reference face image corresponding to the reference feature vector is identified. Estimate that the gender or age is that of the customer.

  Patent Document 13 is an example of a technique for estimating character recognition and the language of a recognized character from an image. A mobile phone with a camera recognizes a character image included in one of a plurality of languages and converts it into a character code. A recognition processing unit. The recognition processing unit includes, for each of the plurality of languages, a language possibility value estimation unit that stores estimated history information indicating that a character recognition result is estimated to be the language, in a history information storage unit, An operation unit that stores in the history information storage unit operation history information indicating that an operation for designating that the language to which the target character belongs belongs is the language, and refers to the history information in the history information storage unit. To convert the character image into a character code.

  Patent Document 14 is an example of a technique for selecting an acoustic model corresponding to a speaker's sex and age from speech, and after performing conversion processing to a known speech recognition feature such as MFCC, categorization is performed according to gender or age group. The matching processing is performed using the plurality of acoustic models and the language model. Here, among the recognition results when matching is performed using each acoustic model, an acoustic model having the highest average value of the top N likelihoods or word reliability is selected.

JP-A-2005-250712 Japanese Patent Laid-Open No. 2007-80057 JP 2001-188556 A JP-A-10-116093 JP 2007-272896 A JP 2008-3749 A JP 2008-15641 A JP 2001-209802 A JP-A-2005-122287 JP-A-2005-78646 JP-A-11-169357 JP 2008-282089 A JP 2006-331354 A JP 2008-96577 A

  When sending information to an unspecified number of people, such as digital signage at the airports visited by people from various countries and the main streets of large cities, the nationality of the individual (in this specification, Information is effective if it can be selected or changed according to the information that represents an international category to which an individual belongs rather than a specific nationality.

  In Patent Documents 1 to 5, gender, age, facial sample similarity, language recognition, and clothing recognition are performed, respectively, but it is not accurate to select information transmission contents by each element alone.

  The present invention objectively analyzes biological characteristics such as individual race and trait, cultural anthropological characteristics such as ethnicity, linguistic characteristics such as native language or dialect of the spoken language, and other external characteristics of the individual. , And automatically and comprehensively determine the nationality of the individual based on the measurement results, and perform actions accordingly.

  The nationality determination device extracts an image input unit for inputting an image, and a plurality of feature amounts related to the attributes of the person from the image input by the image input unit, and handles each feature amount based on the extracted individual feature amounts Image nationality determination unit that individually determines the nationality of the person to be performed, and final nationality determination that determines the nationality of the final person based on the nationality of the person corresponding to each feature amount individually determined by the image nationality determination unit A section.

  Preferably, each feature amount includes a feature amount of the face area detected from the image and a feature amount of the peripheral area of the face region.

  Preferably, the feature amount of the face area includes the color, position, and shape of the face part.

  Preferably, the feature amount of the peripheral region of the face region includes at least one of character information and clothing information.

  Preferably, a voice input unit that inputs voice and a voice nationality determination unit that determines the nationality of a person corresponding to the voice based on the voice input by the voice input unit, and the final nationality determination unit includes an image nationality determination The final nationality of the person is determined based on the nationality of the person corresponding to each feature amount determined by the department and the nationality corresponding to the voice determined by the voice nationality determination part.

  Preferably, the nationality of the person in the image can be comprehensively determined from the image and the content of the utterance.

  Preferably, the voice nationality determination unit recognizes an utterance language from the voice and determines a nationality corresponding to the voice based on the recognized utterance language.

  Preferably, the image nationality determination unit extracts a feature amount related to the universal attribute of the person from the image, and based on the feature amount related to the extracted universal attribute of the person, the universal attribute of the person corresponding to the image The voice nationality determination unit determines a universal attribute of the person corresponding to the voice based on the voice, and the final nationality determination unit determines the universality of the person corresponding to the image determined by the image nationality determination unit. The universal attribute of the final person is determined based on the attribute and the universal attribute of the person corresponding to the voice determined by the voice nationality determination unit.

  Preferably, the universal attributes of the final person include at least one of gender and age.

  Preferably, the final nationality determination unit has a priority defined in advance corresponding to the nationality of the person corresponding to each feature amount determined by the image nationality determination unit and the nationality corresponding to the sound determined by the audio nationality determination unit. Based on this, determine the nationality of the final person.

  Preferably, a final determination result output unit is provided that outputs information indicating the nationality of the final person determined by the final nationality determination unit to a predetermined playback device.

  Preferably, the reproduction information storage unit that stores the nationality of the final person and desired reproduction information in association with each other, and the reproduction information storage unit that stores the reproduction information corresponding to the nationality of the final person determined by the final nationality determination unit And a reproduction information output unit that outputs the extracted reproduction information to a predetermined reproduction device.

  The nationality determination method according to the present invention includes a step in which a computer inputs an image, and extracts a plurality of feature amounts relating to a person's attributes from the input image, and converts each feature amount into individual feature amounts based on the extracted individual feature amounts. A step of individually determining the nationality of the corresponding person, and a step of determining the final nationality of the person based on the nationality of the person corresponding to the individually determined individual feature amount.

  A program for causing a computer to execute the nationality determination method is also included in the present invention.

  According to the present invention, the nationality of a person in an image can be comprehensively determined from each feature amount obtained from the image and further from voice. In addition, information according to the nationality can be reproduced, and information corresponding to the nationality of the person can be provided.

Schematic diagram of nationality determination system Flow chart of nationality determination processing The figure which illustrated the information of nationality information DB

  FIG. 1 is a schematic configuration diagram of a nationality determination system 100 according to a preferred embodiment of the present invention. This system includes an image input device 1, an image analysis device 2, a voice input device 3, a voice analysis device 4, a nationality determination device 5, a nationality information DB 6, a nationality correspondence information DB 7, and a display device 8. The nationality determination device may be a single personal computer (equipped with an arithmetic circuit, a data input / output circuit, a display circuit, an operation device, a communication circuit, etc.), or may be configured by connecting each personal computer via a network. For example, the nationality information DB 6 and the nationality correspondence information DB 7 can be server computers, and the other devices can be client computers. Therefore, it is not necessary for these devices to be integrally installed at the same place. For example, the image input device 1, the voice input device 3, and the display device 8 can be used in airport lobbies, department stores, underpass walls, train Installed in a conspicuous place such as the upper part of the entrance door, the image analysis device 2, the voice input device 3, the voice analysis device 4, the nationality determination device 5, the nationality information DB6, and the nationality correspondence information DB7 are the airport and underpass control room. It can be placed in an invisible place.

  The image input device 1 inputs an image (still image or moving image) to the image analysis device 2. The image input device 1 may be an imaging device itself or an interface that transfers and inputs an image captured by another imaging device to the image analysis device 2. The subject of the input image is a person photographed by an imaging device installed at a place where there are many unspecified people. For example, if an imaging device built in or connected to the image input device 1 as described above is installed in an airport lobby, department store, underpass wall, upper part of a train entrance door, the subject is These are people who use airport lobbies, department stores, underpasses and trains.

  The image analysis device 2 analyzes the input image and outputs the analysis result to the nationality determination device 5. The method of the image analysis apparatus 2 is arbitrary. For example, face detection is performed and a face area is extracted. As a specific method of face detection, a known method can be adopted. For example, a face detection method by edge detection or shape pattern detection, a feature point vector method by vectorizing feature points that are the coordinates of a feature part, and detecting a feature point vector approximately, a region detection method by hue detection or skin color detection, Alternatively, a known method such as face discrimination based on a correlation value with a template as in Japanese Patent No. 4127521 can be used. Then, as will be described later, feature amounts relating to various human attributes are calculated from the detected faces. When the image analysis device 2 and the image input device 1 are remotely connected, it is necessary to transmit and receive images between them. However, if both are installed at the same or close locations, this is not necessary.

  The voice input device 3 includes a microphone that collects voice and converts it into an analog voice signal, an amplifier that amplifies the analog signal output from the microphone, and a converter that converts the amplified analog voice signal into digital voice data. An input voice speaker is a person photographed by an imaging device installed at a place where there are a large number of unspecified people like the subject of the image. In other words, the image input device 1 and the voice input device 3 are installed at the same or close locations. Even if both of the image input device 1 and the voice input device 3 do not function, it is possible to determine the nationality based on only one of the information, and the present invention relates to the image input of the image input device 1 and the voice input device 3. This is not impossible unless both voice inputs are always present.

  The voice analysis device 4 identifies the language of the utterances collected by matching the voice database storing the utterance voice patterns of a plurality of languages (including dialects) and the digital voice data output from the voice input device 3 and the voice database. Including a language identification device. When the voice analysis device 4 and the voice input device 3 are remotely connected, it is necessary to transmit and receive voices between them. However, if both are installed in the same or close locations, this is not necessary.

  The nationality determination device 5 includes circuits necessary for arithmetic processing such as a CPU, a RAM, and a ROM, and determines the nationality of a person based on the analysis result of the image analysis device 2 and the identification language by the voice analysis device 4. Information serving as a determination criterion for nationality is stored in the nationality information DB 6. The nationality determination device 5 can also control the operation of the nationality determination system 100 in an integrated manner.

  The display device 8 is a device that displays an image to be output according to the nationality determined by the nationality determination device 5, and includes a liquid crystal display or the like. In the display device 8, the image input device 1 and the voice input device 3 are installed at the same location or close to each other. Therefore, when there is a person in the vicinity of the image input apparatus 1 and the voice input apparatus 3, an image corresponding to the nationality of the person determined by the nationality determination apparatus 5 can be shown to the person.

  The nationality correspondence information DB 7 stores in advance the final determination nationality and output information corresponding to it (one of video and audio including characters and images, or a combination of both). The output information may be further associated with universal information that does not constitute a nationality classification standard, such as gender and age, and the output information corresponding to nationality may be further classified and individualized for men and women. . The nationality determination device 5 extracts output information corresponding to the determined nationality from the nationality correspondence information DB 7 and outputs the output information to the display device 8. The nationality information DB 6 and the nationality correspondence information DB 7 are configured by a storage medium such as an HDD.

  Hereinafter, the nationality determination process executed by the nationality determination system 100 will be described with reference to the flowchart of FIG. This process is controlled by the nationality determination device 5, and a program for executing the control is stored in a computer-readable storage medium such as a ROM provided in the nationality determination device 5. Read and execute.

  In S <b> 1, the image input apparatus 1 inputs an image. The input method is arbitrary, and may be input from a camera or input of existing image (still image / moving image) data. However, in order to immediately reflect the attribute of the subject of the image in the reproduction content of the display device 8, it is desirable to input the captured image in real time. The nationality determination device 5 may be able to command image capturing and input timing. For example, the nationality determination device 5 instructs to take one still image every minute, or instructs to take a moving image for 10 seconds every minute.

  In S <b> 2, the image analysis device 2 detects a face area from the input image, and calculates a feature amount from the detected face. When the image is a moving image, the face of the same person is detected from a plurality of images with different orientations and expressions detected from frames constituting the moving image within a certain time.

  When multiple faces are included in the same image, or when multiple images are included in different images, the face to be subject to subsequent nationality determination is determined according to the following conditions at the time the image is input. Also good.

  (1) Target all faces included in the image.

  (2) Of the faces included in the image, those that meet a predetermined condition (face size is equal to or greater than a threshold, for example, 16 × 16 pixels or greater)

  (3) Of the faces included in the image, the target face is selected by the authorized user via the operation device 9 (configured with a keyboard, mouse, touch panel, etc.).

  If there are a plurality of faces that meet the above conditions, the subsequent processing is executed for each face.

  In S3, the image analysis device 2 calculates the facial feature amount from the extracted face. The facial feature amount and its calculation method are as follows, for example.

  (1) Contour shape. See, for example, US Pat.

  (2) Face skin color. For example, as in paragraph 0054 of Patent Document 7, color information of an area (skin color area) determined that the color information satisfies a predetermined conditional expression is set as the skin color.

  (3) Eye position, shape, and pupil color. The eye position can be extracted by extracting the feature points (the eye corners and the eye heads) of Patent Document 6. The eye shape is extracted by using the eye contour extracted in Patent Document 8 as the eye shape. Regarding the extraction of the pupil color, the color information of the pupil region extracted as in Patent Document 9 is used as the pupil color.

  (4) The position and shape of the nose. Extraction of the position of the nose can be performed by feature point extraction (left nose, right nose, etc.) of Patent Document 6. Extraction of the shape of the nose is possible in Patent Document 10.

  (5) The position and shape of the lips. The position of the lips can be extracted by feature point extraction (left mouth corner, right mouth corner, middle point of upper lip, middle point of lower lip, etc.) in Patent Document 6. Extraction of the shape of the lips is possible in Patent Document 10.

  (6) Hair style, hair position and color. Extraction of hairstyle is possible in Patent Document 11. The position and color of the hair can be extracted from the position and color of the identified hairstyle.

  Alternatively, the age and sex are estimated based on the facial feature amount (see Patent Document 12). Other known techniques may be used to calculate various feature quantities indicating human attributes that can be analyzed from the image, and the feature quantities are not limited to the above.

  In S <b> 4, the image analysis apparatus 2 extracts a feature amount related to the attribute of the person from the peripheral area of the face area of the input image (the face area itself is excluded). For example, as in Patent Document 5, clothes are extracted. Not only clothes worn from the neck down, but also ornaments such as hats and mufflers may be extracted. Alternatively, as in Patent Document 12, using a character recognition technique from an image, around a person's face area such as a book, newspaper, magazine, or logo printed on a person's clothes Recognize certain character information and estimate the language of the recognized character information. If the face area is used as a reference, the surrounding area is arbitrary. For example, the face area is removed from the extended area obtained by enlarging the vertical and horizontal sizes of the face area by a predetermined magnification (2 ×, 4 ×, etc.) Is the surrounding area.

  In S5, a voice is input from the voice input device 3. Assume that the nationality determination device 5 controls the image input (imaging timing) and the audio input timing to be synchronized.

  In S6, voice analysis by the voice analysis device 4 is started. For example, as in Patent Documents 3 and 4, the speech language is identified by matching the sample speech pattern of the speech language database with the input speech. Note that the gender and age of the speaker can be determined based on the voice instead of or together with the image analysis. For example, as in Patent Document 14, the age and gender corresponding to the speech model having the highest degree of coincidence are determined as the gender and age of the speaker in accordance with matching between the sex model and the age-specific speech model and the input speech.

  In S <b> 7, the nationality determination device 5 determines the nationality, gender, and age of the person based on the analysis result of the image analysis device 2 and the analysis result of the voice analysis device 4.

  First, the nationality corresponding to each of the patterns that can be output as the analysis result of the image analysis apparatus 2 is stored in the nationality information DB 6 in advance, and the nationality determination apparatus 5 performs the actual individual analysis result (S3 of the image analysis apparatus 2). Alternatively, the nationality information corresponding to S4) is extracted from the nationality information DB6.

  For example, in the nationality information DB 6, the shape of the face and face parts (eyes, nose, lips) (or the feature amount indicating the shape) as in the Caucasian skeleton = Europe, the Mongoloid skeleton = Asia, and the Negroid skeleton = Africa. The nationality determination device 5 identifies from the nationality information DB 6 the birth region corresponding to the shape of the face and face parts, which are the actual analysis results of the image analysis device 2, and this is based on the face base. As nationality information a.

  The nationality information DB 6 associates the skin color (or feature amount indicating the color) of the face region such as brown = Asia, white = Europe, black = Africa, and the region of origin (see FIG. 3). 5 designates a region of origin corresponding to the skin color of the face region, which is an actual analysis result of the image analysis device 2, from the nationality information DB 6, and extracts this as skin color-based nationality information b.

  The nationality information DB 6 may associate a combination of the shape of the face and face parts and the skin color with the region of origin. For example, a Caucasian skeleton and brown skin color = Hispanic (Latin America) can be used.

  Alternatively, the nationality information DB 6 stores the image feature pattern of national costumes of each country such as Sally = India, Chimachogori = Korean Peninsula, Turban = Middle East, Kimono = Japan, and the region from which they are associated. The determination device 5 identifies the region of origin corresponding to the feature quantity indicating the clothing, which is the actual analysis result of the image analysis device 2, from the nationality information DB 6, and extracts this as the costume-based nationality information c.

  Alternatively, the nationality information DB 6 includes Japanese = Japan, English (American English) = United States, English (British English) = United Kingdom, Spanish = Spain or Latin America, Mandarin = Beijing, Cantonese = Guangdong, Like Hong Kong and Macau, the language used and the region of origin are associated with each other, and the nationality determination device 5 identifies the region of origin corresponding to the language that is the actual analysis result of the image analysis device 2 from the nationality information DB 6. This is extracted as nationality information d based on image peripheral information. The language used here is written language, but the same correspondence is possible with spoken language.

  That is, the nationality determination device 5 identifies a region of origin corresponding to the language that is the analysis result of the speech analysis device 4 from the nationality information DB 6 and extracts this as utterance-based nationality information e. There may or may not be a subdivision of the region of origin in the same language dialect, but especially in languages with a large native population (Chinese, English, Spanish, etc.), the range that can be distinguished by spelling of spoken language and spoken language accents If you subdivide your home region, your nationality will be more accurate.

  As described above, the nationality determination device 5 extracts nationality information corresponding to each analysis result by the image analysis device 2 or the voice analysis device 4 from the nationality information DB 6. However, if the individual analysis itself fails or the nationality information corresponding to the analysis result is not present in the nationality information DB 6, the nationality information is “unknown”.

  Next, the nationality determination device 5 determines the final nationality of the person based on the individual nationality information extracted from the nationality information DB 6. For example, the priority is defined in advance in a storage medium such as the nationality information DB 6 for individual nationality information corresponding to each analysis result, and the individual nationality information having the highest priority is defined as the final nationality.

  For example, a 40-year-old Japanese man in a suit with light orange skin has an English newspaper with an article spelled in American English and speaks Japanese. As a result of taking an image as a subject and collecting speech, nationality information a = Asia, nationality information b = Asia, nationality information c = unknown, nationality information d = English, nationality information e = Japan, gender = Suppose a man is age = early 40s.

  The nationality information DB 6 stores the order of utterance-based nationality information e> face-based nationality information a> costume-based nationality information c> skin color-based nationality information b> image peripheral information-based nationality information d. Suppose that Then, since the priority of the nationality information is “Japan” which is the speech-based nationality information e, the nationality determination device 5 finally determines the nationality of the person as “Japan”. Gender and age are not used for final nationality determination.

  In the above-mentioned ordering, the accuracy of determination becomes high when highly accurate nationality information is brought to the top. However, in order to enable appropriate determination according to the usage environment such as the installation location of the nationality determination system 100, the order of the order may be arbitrarily set by the authorized user from the operation device 9. For example, when the nationality determination system 100 is installed in a noisy place where people are busy, it is considered that the accuracy of nationality determination by voice is considered to be low, so the priority of the utterance-based nationality information e may be lowered. .

  In addition, nationality information a to e for all genres may not always be available, but nationality information that does not exist is excluded from priority determination targets. For example, when recording is possible but recording is not possible, the utterance-based nationality information e does not exist, and the highest-ranked nationality information excluding the non-existent utterance-based nationality information e is finally obtained. Judged as nationality.

  Final nationality determination does not have to be based on ordering. For example, the majority (the most common nationality) in the individual nationality information may be determined as the final nationality. Nationality information a = Asia, Nationality information b = Asia, Nationality information c = Unknown, Nationality information d = English, Nationality information e = In Japan, there are 2 Asia, 1 English, 1 Japan, Asia is the final nationality because of the majority. However, since Japan is included in Asia, it is considered that Japan is also included in the majority Asia, and in this case, Japan, which is nationality information of a lower concept than Asia, may be determined as the final nationality. .

  Universal information (gender and age) is also determined based on the gender and age obtained from the results of image analysis and speech analysis. The age may not be a unique number, but may be a numerical range (age group) such as early teens, late 10s, early 20s. For example, regarding the gender, if the genders obtained from the results of the image analysis and the voice analysis match, the matching gender is determined as the final gender. If they do not match, it is determined that the gender is unknown. Regarding the age, a range obtained by taking a logical sum (OR) of the age groups obtained from the results of the image analysis and the voice analysis is set as the final age group. The logical product (AND) of both analysis results, that is, the overlapping age of both may be determined as the final age group, but if there is no overlapping, it is determined as “age unknown”.

  In S8, the nationality finally determined in S7 is displayed on the display device 8. Alternatively, instead of displaying the final determined nationality itself or together with the determined nationality, output information (video) corresponding to the final determined nationality is acquired from the nationality correspondence information DB 7 and the acquired output information is displayed. Good. Although illustration is omitted, if the nationality determination system 100 includes a known audio reproduction device 10 such as an audio decoder, an amplifier, and a speaker, and the output information includes audio information, the audio corresponding to the final determined nationality. Information may be reproduced by the audio reproduction device 10. Of course, the output information may include both video and audio, and both may be displayed synchronously on the display device 8 and the audio reproduction device 10.

  For example, when the nationality of the person is finally determined as “Japan” as described above, the output information (Japanese airport guide or advertisement message) corresponding to “Japan” is reproduced by video or audio. When the output information corresponding to “Japan” is subcategorized for each universal information (gender and age) in the nationality correspondence information DB 7, the final determination nationality “Japan”, the determination age “40s”, and the determination gender “ Output information corresponding to “male” is acquired from the nationality correspondence information DB 7 and reproduced.

  Alternatively, if the nationality of the person is finally determined to be “Asia”, the output information corresponding to “Asia” (airport information and advertisement messages in both Japanese, Korean, and Mandarin) will be displayed in video and / or audio. To play.

  When there are a plurality of persons whose final nationality is determined, the final determined nationality of each person and / or output information corresponding thereto may be displayed in association with the subject person in the input image. This can be a still image or a moving image. If there are a plurality of images, each image is displayed one by one or dividedly on one screen for a predetermined display period, and nationality and / or output information corresponding to the subject person in the image is displayed. For the correspondence between the two, a balloon or the like arranged near the subject person may be used.

  Also, when determining the final nationality with the majority mentioned above, the final determination should be made uniquely because there are multiple majoritys of the same number or because there is no single majority due to conflicting individual nationality information. Is impossible, the individual nationality information is displayed as a list of nationality candidates on the display device 8 in the order of accuracy of the individual nationality information, and the authorized user arbitrarily selects the nationality candidate from the nationality candidates via the operation device 9. The nationality candidate may be determined as the final nationality. Alternatively, when each nationality information is “unknown”, default information stored in the nationality correspondence information DB 7, for example, guidance or advertisement message written in English is acquired as output information, and displayed or reproduced by voice. .

  When the reproduction of the output information acquired from the nationality correspondence information DB 7 is completed, the process returns to S1 and the same processing can be repeated. The start / end of this process may be automatically controlled by the timer function, or an authorized user may arbitrarily command the start / continuation / end of the process from the nationality candidate via the operation device 9.

  Through the above processing, the nationality of the person can be accurately and comprehensively determined, information on the content that matches the determined nationality of the person can be output, and the information can be presented in a form that is easy for the person to understand.

1: Image input device, 2: Image analysis device, 3: Audio input device, 4: Audio analysis device, 5: Nationality determination device, 6: Nationality information DB, 7: Nationality correspondence information DB, 8: Display device

Claims (13)

An image input unit for inputting an image;
An image nationality that extracts a plurality of feature quantities related to the attributes of a person from an image input by the image input unit, and individually determines nationality information of the person corresponding to the individual feature quantities based on the extracted individual feature quantities A determination unit;
Based on the nationality information of the person corresponding to the individual feature amounts individually determined by the image nationality determination unit, and the priority order of the nationality information defined in advance , the ranking of the determined individual nationality information is determined. A final nationality determination unit that determines and determines the nationality information of the highest ranking as the final nationality of the person,
Equipped with a,
The final nationality determination unit is a nationality determination device that excludes nationality information that cannot be determined from the determination target of the ranking of the nationality information .   The nationality determination device according to claim 1, wherein the individual feature amount includes a feature amount of a face area detected from the image and a feature amount of a peripheral area of the face region.   The nationality determination device according to claim 2, wherein the feature amount of the face area includes a color, a position, and a shape of a face part.   The nationality determination device according to claim 2 or 3, wherein the feature amount of the peripheral region of the face region includes at least one of character information and clothing information. A voice input unit for inputting voice;
A voice nationality determination unit that determines nationality information of a person corresponding to the voice based on the voice input by the voice input unit;
With
The final nationality determination unit includes nationality information corresponding to the individual features determined by the image nationality determination unit and nationality information corresponding to the sound determined by the audio nationality determination unit, and the predefined nationality The order of the determined individual nationality information is determined based on the priority order of information, and the nationality of the final person is determined based on the highest nationality information . Nationality determination device.   The nationality determination device according to claim 5, wherein the voice nationality determination unit recognizes an utterance language from the voice and determines a nationality corresponding to the voice based on the recognized utterance language. The image nationality determination unit extracts a feature amount related to the universal attribute of the person from the image, and determines a universal attribute of the person corresponding to the image based on the extracted feature amount related to the universal attribute of the person. Judgment,
The voice nationality determination unit determines a universal attribute of the person corresponding to the voice based on the voice,
The final nationality determination unit is configured to determine a final attribute based on the universal attribute of the person corresponding to the image determined by the image nationality determination unit and the universal attribute of the person corresponding to the sound determined by the audio nationality determination unit. The nationality determination device according to claim 5 or 6, wherein a universal attribute of a typical person is determined.   The nationality determination device according to claim 7, wherein the universal attribute of the final person includes at least one of gender and age. The final nationality determination unit is defined in advance corresponding to the nationality information of the person corresponding to the individual feature amount determined by the image nationality determination unit and the nationality information corresponding to the sound determined by the audio nationality determination unit. The nationality determination device according to any one of claims 5 to 8, wherein a rank of the determined individual nationality information is determined based on priority, and the highest nationality information is determined as the final nationality of a person. .   The nationality determination device according to claim 1, further comprising a final determination result output unit that outputs information indicating the nationality of the final person determined by the final nationality determination unit to a predetermined playback device. A reproduction information storage unit for storing the final person's nationality and desired reproduction information in association with each other;
A reproduction information output unit that extracts reproduction information corresponding to the nationality of a final person determined by the final nationality determination unit from the reproduction information storage unit and outputs the extracted reproduction information to a predetermined reproduction device. The nationality determination device according to any one of 1 to 10. Computer
Inputting an image;
Extracting a plurality of feature quantities related to the attributes of the person from the input image, and individually determining the nationality of the person corresponding to the individual feature quantities based on the extracted individual feature quantities;
Based on the nationality information of the person corresponding to the individually determined individual feature amount and the priority order of the nationality information defined in advance , the ranking of the determined individual nationality information is determined, and the highest ranking and nationality and determining the final person highly nationality information of,
Excluding indeterminate nationality information from the determination target of the nationality information,
Nationality determination method including   A program for causing a computer to execute the nationality determination method according to claim 12.

Images