JP2021015362A - Interactive device, information processing method, and information processing program - Google Patents

Abstract

To alleviate a user load when updating teacher data of a model used for estimation of a person.SOLUTION: The interactive device estimates a user photographed by a camera, and interacts with the estimated user, and includes a person estimation model generated by using a plurality of pieces of imaging data obtained by photographing persons under a plurality of photographing conditions as teacher data, an estimation unit which estimates a user photographed by the camera by using the person estimation model, an interaction unit which outputs an inquiry containing information indicating the estimated person, and an updating unit which replaces one piece of the teacher data with a piece of imaging data captured when the user is photographed by the camera so that a variation of appearance frequency of the imaging data being the teacher data decreases for each of the photographing conditions when the response to the inquiry indicates that the user and the estimated person are different.SELECTED DRAWING: Figure 2

Description

The present invention relates to an interactive device, an information processing method and an information processing program.

An information processing device that estimates a person using a model generated as teacher data of imaged data obtained by photographing a person is used (see, for example, Patent Documents 1 and 2).

Japanese Unexamined Patent Publication No. 2010-061465 Japanese Unexamined Patent Publication No. 2008-282089

The estimation of a person is used in various aspects, and for example, a dialogue device for performing a dialogue with the estimated person has been proposed. In such a dialogue device, in order to improve the estimation accuracy of a person, a model is generated in which imaging data under various shooting conditions such as posture, direction, clothes and brightness around the person are used as teacher data for person estimation. At the same time, it is preferable to update the teacher data for the generated model as appropriate. However, updating the teacher data requires shooting the user under various shooting conditions, which is a burden on the user.

One aspect of the disclosed technique is to provide a dialogue device, an information processing method and an information processing program that can reduce the burden on the user in updating the teacher data of the model used for estimating a person.

One aspect of the disclosed technique is exemplified by the following dialogue device. This dialogue device is a dialogue device that estimates a user captured by a camera and performs a dialogue with the estimated user. This dialogue device estimates a person estimation model generated by using a plurality of imaging data obtained by photographing a person under a plurality of shooting conditions as teacher data, and an estimation that estimates a user captured by the camera using the person estimation model. When the unit, the dialogue unit that outputs a question including information indicating the estimated person, and the response to the question indicate that the user and the estimated person are different, one of the teacher data and the camera It is provided with an update unit that replaces the imaging data acquired when the user is photographed so as to reduce the variation in the appearance frequency of the imaging data as the teacher data for each imaging condition.

The disclosed technique can reduce the burden on the user in updating the teacher data of the model used for estimating the person.

FIG. 1 is a diagram showing an example of a hardware configuration of a communication device according to an embodiment. FIG. 2 is a diagram showing an example of a processing block of the communication device according to the embodiment. FIG. 3 is a diagram showing an example of a shooting condition management table stored in the management database in the embodiment. FIG. 4 is a diagram showing an example of shooting conditions for face image data shot by the shooting unit in the embodiment. FIG. 5 is a diagram showing an example of dispersion when the teacher data is exchanged. FIG. 6 is a diagram showing a processing flow of initial construction of the determination database in the embodiment. FIG. 7 is a first diagram showing an example of a processing flow of the determination data model update processing in the embodiment. FIG. 8 is a second diagram showing an example of a processing flow of the determination data model update processing in the embodiment. FIG. 9 is a diagram showing an example of a processing block of the communication device according to the first modification.

The dialogue device according to the embodiment is a dialogue device that estimates a user captured by a camera and performs a dialogue with the estimated user. The dialogue device has, for example, the following configuration.
A person estimation model generated by using multiple imaging data of a person photographed under multiple shooting conditions as teacher data, and
An estimation unit that estimates the user captured by the camera using the person estimation model,
A dialogue section that outputs questions that include information indicating the estimated person,
When the response to the question indicates that the user and the estimated person are different, one of the teacher data and the imaging data acquired when the user is photographed by the camera are used as the teacher data. It is provided with an update unit that is replaced so that the variation in the appearance frequency of the data for each shooting condition is reduced.

The dialogue device, for example, performs voice communication with the user. The dialogue device includes a person estimation model generated by using a plurality of imaging data obtained by photographing a person under a plurality of photographing conditions as teacher data. The person estimation model is generated by machine learning such as deep learning. The camera is, for example, a digital camera equipped with a Charge Coupled Device (CCD) sensor and a Complementary Metal Oxide Sensor (CMOS) sensor. The camera generates imaging data in which the captured user's image is used as a digital image.

The estimation unit estimates the user captured by the camera using a person estimation model. The estimation unit acquires, for example, information indicating the estimated person as a result of estimation using the person estimation model. Examples of the information indicating the person include a name, a surname, a first name, a nickname, and the like.

The dialogue unit outputs a question including information indicating the estimated person. Question is, for example, a greeting and schedule of the notification, such as, in particular, "Hello, Mr. A", "Mr. A, tomorrow there is a XX of the event" can be mentioned wording such.

As a case where the response to the question indicates that the user and the estimated person are different, a response denying the information indicating the estimated person included in the question can be mentioned. For example, for the question of "Hello, Mr. A", a response of "No, I am B", that has denied in response to "Mr. A" that is included in the question, was the estimated and the user Indicates that the person is different. When the user and the estimated person are different in this way, it is considered that the estimation accuracy of the person using the person estimation model is low, so that the update unit replaces the teacher data of the person estimation model.

By exchanging the teacher data so that the variation in the appearance frequency for each shooting condition is reduced, the updating unit 16 can assume that the imaging data used for generating the person estimation model is captured under various shooting conditions. .. As a result, the dialogue device can improve the estimation accuracy of the person under various conditions. In addition, since this dialogue device replaces the teacher data using the imaging data acquired during the dialogue with the user, it is not necessary to instruct the user about the posture, etc., so that when the teacher data is replaced, the teacher data is replaced. The burden on the user can be reduced.

In the dialogue device, the dialogue unit may output a question including information indicating the estimated person as a response to an instruction from the user. That is, the first voice in the dialogue with the user may be from the user, not from the dialogue device.

In the present dialogue device, the estimation unit may make the camera take a picture of the user who responded to the question again, and further estimate the user using the imaged data taken again. The user who responded to the question is considered to have a different posture and position than before responding to the question. By estimating the user whose posture and position have changed in this way and updating the teacher data based on the estimation result, the teacher data under various shooting conditions can be used to generate a person estimation model.

In the present dialogue device, when there is a plurality of teacher data capable of reducing the variation in the appearance frequency for each shooting condition by replacing the imaging data captured by the camera, the updating unit appears for each shooting condition. The teacher data having a large amount of reduction that reduces the variation in frequency may be prioritized and replaced with the imaging data captured by the camera. By having such a feature, the imaging data collected in a well-balanced manner for each shooting condition can be used as the teacher data of the person estimation model.

In this dialogue device, the updating unit is photographed at an older time when there is a plurality of teacher data capable of reducing the variation in the appearance frequency for each shooting condition by exchanging the imaging data captured by the camera. The teacher data may be prioritized and replaced with the imaging data captured by the camera. Since the user's face changes over time, it is considered that the old imaging data is not valid as teacher data. By having such a feature, the old teacher data can be replaced with the new imaging data, and the person estimation model can be suitable for the estimation of the current user.

In this dialogue device, even if the response to the question is the same between the user and the estimated person, the update unit is used when a predetermined period has passed since the previous replacement of the teacher data. May replace any of the teacher data with the imaging data captured by the camera so as to reduce the variation in the appearance frequency of the imaging data as the teacher data for each imaging condition. These features also allow the old teacher data to be replaced with new imaging data, making the person estimation model suitable for the estimation of the current user.

The embodiment can also be grasped as an information processing method and an information processing program executed by the dialogue device.

Hereinafter, embodiments will be further described with reference to the drawings. The configurations of the embodiments shown below are examples, and the disclosed technology is not limited to the configurations of the embodiments.

<Embodiment>
In the embodiment, a communication device that estimates the user captured by the camera and interacts with the estimated user will be described. FIG. 1 is a diagram showing an example of a hardware configuration of a communication device according to an embodiment. The communication device 1 illustrated in FIG. 1 includes a Central Processing Unit (CPU) 101, a main storage unit 102, an auxiliary storage unit 103, a camera 104, a microphone 105, a speaker 106, a timekeeping unit 107, a sensor 108, and a connection bus B1. It is an information processing device. The CPU 101, the main storage unit 102, the auxiliary storage unit 103, the camera 104, the microphone 105, the speaker 106, the timekeeping unit 107, and the sensor 108 are connected to each other by the connection bus B1. The communication device 1 is an example of a “dialogue device”.

The CPU 101 is also called a microprocessor unit (MPU) or a processor. The CPU 101 is not limited to a single processor, and may have a multiprocessor configuration. Further, a single CPU 101 connected by a single socket may have a multi-core configuration. At least a part of the processing executed by the CPU 101 is performed by a processor other than the CPU 101, for example, a dedicated processor such as a Digital Signal Processor (DSP), a Graphics Processing Unit (GPU), a numerical arithmetic processor, a vector processor, or an image processing processor. You may. Further, at least a part of the processing executed by the CPU 101 may be executed by an integrated circuit (IC) or other digital circuit. Further, at least a part of the CPU 101 may include an analog circuit. Integrated circuits include Large Scale Integrated Circuits (LSIs), Application Specific Integrated Circuits (ASICs), and Programmable Logic Devices (PLDs). PLDs include, for example, Field-Programmable Gate Array (FPGA). The CPU 101 may be a combination of a processor and an integrated circuit. The combination is called, for example, a microcontroller unit (MCU), a system-on-a-chip (SoC), a system LSI, a chipset, or the like. In the communication device 1, the CPU 101 expands the program stored in the auxiliary storage unit 103 into the work area of the main storage unit 102, and controls the peripheral devices through the execution of the program. As a result, the communication device 1 can execute a process that meets a predetermined purpose. The main storage unit 102 and the auxiliary storage unit 103 are recording media that can be read by the communication device 1.

The main storage unit 102 is exemplified as a storage unit that is directly accessed from the CPU 101. The main memory 102 is a Random Access Memory (RAM) and a Read.
Includes Only Memory (ROM).

The auxiliary storage unit 103 stores various programs and various data in a readable and writable recording medium. The auxiliary storage unit 103 is also called an external storage device. The auxiliary storage unit 103 stores an operating system (Operating System, OS), various programs, various tables, and the like. External devices and the like include, for example, other information processing devices and external storage devices connected by a computer network or the like.

The auxiliary storage unit 103 is, for example, an Erasable Programmable ROM (EPROM), an Embedded MultiMediaCard (eMMC), a solid state drive (Solid State Drive, SSD), a hard disk drive (Hard Disk Drive, HDD), or the like. Further, the auxiliary storage unit 103 is, for example, a Compact Disc (CD) drive device, Digital.
Versail Disc (DVD) drive device, Blu-ray®
It may be a Disc (BD) drive device or the like.

The camera 104 is an imaging device capable of capturing at least one of a moving image and a still image. The camera 104 is, for example, a digital camera including a CCD sensor or a CMOS sensor.
The camera 104 uses, for example, an Exchange Image of captured image data.
The shooting date and time acquired from the time measuring unit 107 may be stored in the File Format (Exif) data. The camera 104 is arranged, for example, on the front side (the side facing the user) of the communication device 1.

The microphone 105 is an input unit that receives voice operation instructions and the like from the user. The voice input by the microphone 105 is voice-analyzed to extract information such as a name. The communication device 1 may include another input unit in addition to or in place of the microphone 105. Examples of other input units include a keyboard, a pointing device, a touch panel, an acceleration sensor, and the like.

The speaker 106 is an output unit that outputs audio. The communication device 1 may include another output unit in addition to the speaker 106 or in place of the speaker 106. Examples of other output units include output devices such as a Cathode Ray Tube (CRT) display, a Liquid Crystal Display (LCD), a Plasma Display Panel (PDP), an Electroluminescence (EL) panel, and an organic EL panel. The communication device 1 according to the present embodiment is provided with the microphone 105 and the speaker 106 to enable voice communication (dialogue) with the user.

The timekeeping unit 107 is a circuit that generates time information. The timekeeping unit 107 is, for example, a clock built in the communication device 1.

The sensor 108 is a sensor that detects the approach of a person. As the sensor 108, a motion sensor can be mentioned. The motion sensor detects the approach of a person by using, for example, infrared rays, ultrasonic waves, and visible light. When detecting the approach of a person using visible light, an illuminance sensor may be adopted as the sensor 108.

<Processing block of communication device 1>
FIG. 2 is a diagram showing an example of a processing block of the communication device according to the embodiment. The communication device 1 includes a photographing unit 11, an estimation unit 12, a dialogue unit 13, a determination unit 14, a calculation unit 15, an update unit 16, an initial construction unit 17, a determination data model 18, and a management database 19. In the communication device 1, the CPU 101 executes a computer program executably deployed in the main storage unit 102, so that the communication device 1 has a photographing unit 11, an estimation unit 12, a dialogue unit 13, a determination unit 14, and a calculation unit. The processing as each part of 15, the update unit 16, the initial construction unit 17, the determination data model 18, the management database 19, and the like is executed.

The determination data model 18 is a learning model constructed by machine learning such as deep learning, using face image data obtained by extracting a face region from imaged data taken by a user as a set of teacher data (teacher data set). The determination data model 18 is constructed by inputting, for example, a teacher data set into a classifier such as a support vector machine (SVM). The determination data model 18 is used for user estimation by the estimation unit 12, which will be described later. The determination data model 18 is constructed for each user, for example, and the user's name can be used as the model name. The user's name is acquired by collating the imaging data acquired by the user's shooting with the determination data model 18. The determination data model 18 is an example of a “person estimation model”.

The management database 19 is a database that manages face image data as teacher data of the determination data model 18. FIG. 3 is a diagram showing an example of a shooting condition management table stored in the management database in the embodiment. The shooting condition management table 191 illustrated in FIG. 3 is created for each user who has been shot, for example. For example, a user's name can be attached to each of the shooting condition management tables 191 as a label. Each of the shooting condition management tables 191 is specified by the name of the user attached as a label. Note that the label attached to each of the shooting condition management table 191 is not limited to the user's name, and if the information is uniquely determined for each user, it is information such as a numerical value or a character string other than the user's name. You may. The shooting condition management table 191 manages the shooting conditions of each of the face image data used as the teacher data of the determination data model 18 among the face image data extracted from the imaged data acquired by the shooting unit 11.

The shooting condition management table 191 illustrated in FIG. 3 includes each item of "PID", "shooting date", and "shooting condition". The "PID" stores ID information that uniquely identifies the face image data extracted from the imaging data acquired by the camera 104 taking a picture of the user. The “shooting date” stores the date on which the shooting was performed, that is, the date on which the imaging data from which the face image data was extracted was generated. In the "shooting condition", information indicating the shooting condition at the time of shooting is stored. In the example of FIG. 3, the "shooting condition" includes three categories of "direction", "distance" and "light and darkness". In the "orientation", information indicating the orientation of the photographed user's face is stored. In FIG. 3, five directions of "front", "left", "right", "up" and "down" are exemplified as the "direction". In the "distance", information indicating the distance between the communication device 1 and the user is stored. In FIG. 3, three stages of “far”, “medium”, and “near” are exemplified as “distance”. Information indicating the brightness of the user's surroundings at the time of shooting is stored in "brightness". In FIG. 3, three stages of “light”, “medium”, and “dark” are exemplified as “light and dark”. The criteria for each of the five directions of "direction", the three steps of "distance", and the three steps of "light and darkness" may be appropriately determined according to the characteristics such as the f-number and focal length of the camera 104, for example. In FIG. 3, for each of “direction”, “distance”, and “brightness”, the bit of the item corresponding to the imaging data is turned on. For example, in the imaging data in which the "PID" is 1, it can be understood that the orientation of the face is "front", the distance between the communication device 1 and the user is "medium", and the brightness around the user is "medium". ..

The shooting condition management table 191 further includes each item of "total", "dispersion", and "variance total". In "total", the total number of corresponding imaging data is stored for each item of shooting conditions. In the "dispersion", information (index) indicating the variation in the appearance frequency for each shooting condition is stored. Examples of the index showing the variation for each shooting condition include variance, standard deviation, coefficient of variation, and the like. As the dispersion, either unbiased dispersion or sample dispersion may be used. In the present embodiment, "dispersion" is adopted as an index showing the variation for each shooting condition, but other indexes may be used. In the example of FIG. 3, as "dispersion", among the three categories included in "shooting conditions", the variance for "direction", the variance for "distance", and the variance for "light and darkness" are exemplified. .. The total value of these variances is stored in the "variance total". That is, the smaller the value of "total variance", the more equal the number of teacher data belonging to each of the "direction", "distance" and "light and dark" categories is. Also, the smaller the value of "variance", the more the number of teacher data belonging to each condition in the category (for example, "far", "medium", "near" in the "distance" category) is possible. It is shown that they are equal.

When the sensor 108 detects the approach of the user, the photographing unit 11 photographs the user using the camera 104. The photographing unit 11 detects the face area included in the imaged data acquired by the image taken by the camera 104, and acquires the face image data obtained by extracting the face area. The face area is an area including at least the user's face (head), and may include the user's face and its surroundings. The photographing unit 11 stores the acquired face image data in the auxiliary storage unit 103 in association with a UID that uniquely identifies the face image data.

The estimation unit 12 estimates the person photographed by the photographing unit 11 using the camera 104 using the determination data model 18. For example, the estimation unit 12 collates the face image data acquired by the photographing unit 11 with the determination data model 18 and estimates the name of the person. The estimation unit 12 is an example of the “estimation unit”.

The dialogue unit 13 has a dialogue with the user estimated by the estimation unit 12. The dialogue unit 13 outputs, for example, a question including the name of the person estimated by the estimation unit 12. For example, the dialogue unit 13 outputs a greeting including the user's name as a question from the speaker 106. The dialogue unit 13 acquires a response from the user to the question via the microphone 105. For example, the dialogue unit 13 voice-analyzes the acquired response from the user and acquires the name included in the response. The dialogue unit 13 is an example of the “dialogue unit”.

The determination unit 14 determines whether or not the responding user and the person estimated by the estimation unit 12 match, based on the response from the user acquired by the dialogue unit 13. When the name included in the question and the name acquired by the dialogue unit 13 do not match, the determination unit 14 determines that the responding user and the person estimated by the estimation unit 12 do not match. Further, when the name included in the question and the name acquired by the dialogue unit 13 match, the determination unit 14 determines that the responding user and the person estimated by the estimation unit 12 match. When the dialogue unit 13 receives a response that does not deny the name included in the question, the determination unit 14 may determine that the responding user and the person estimated by the estimation unit 12 match.

As a specific example in which the responding user and the person estimated by the estimation unit 12 do not match, the dialogue unit 13 outputs a question "Good morning, Mr. A mountain B Taro", and the response to the question is "No, I'm a C field D man ", for example, when I get a response from a user. In this case, the name "C field D man" included in the user's response does not match the name "A mountain B Taro" included in the question. In other words, the user response includes a name different from the name estimated by the estimation unit 12. Therefore, the determination unit 14 determines that the responding user and the person estimated by the estimation unit 12 do not match.

Further, as a specific example in which the responding user and the person estimated by the estimation unit 12 match, the dialogue unit 13 outputs a question "Good morning, Mr. A mountain B Taro" and responds to the question. For example, when a response such as "Yes, good morning" or "Yes, good morning, A mountain B Taro" is obtained from the user. In this case, the user's response does not deny the name included in the question. Therefore, the determination unit 14 determines that the responding user and the person estimated by the estimation unit 12 match.

The calculation unit 15 refers to the shooting condition management table 191 and calculates an index indicating the variation in the appearance frequency of the face image data used as the teacher data of the determination data model 18 for each shooting condition. As the index showing the variation in the appearance frequency for each shooting condition, the variance, standard deviation, coefficient of variation, etc. can be mentioned as described above, and “variance” is adopted in the present embodiment. The calculation unit 15 stores the calculated variance in the shooting condition management table 191.

When the determination unit 14 determines that the responding user and the person estimated by the estimation unit 12 do not match, the update unit 16 updates the teacher data used for constructing the determination data model 18. The update unit 16 acquires the name included in the response from the user. The update unit 16 reads the determination data model 18 corresponding to the acquired name from the auxiliary storage unit 103. The update unit 16 uses the face image data acquired by the photographing unit 11 to update the teacher data of the read determination data model 18 so that the dispersion of the photographing conditions of the teacher data becomes small. In other words, the update unit 16 updates the teacher data so that the determination data model 18 is constructed using the teacher data shot under various shooting conditions. The teacher data is updated, for example, by replacing any of the teacher data among the teacher data of the determination data model 18 with the face image data acquired by the photographing unit 11.

The update unit 16 identifies the teacher data to be replaced among the teacher data of the determination data model 18 read from the auxiliary storage unit 103. For example, the update unit 16 causes the calculation unit 15 to calculate the variance of the shooting conditions when each of the teacher data of the determination data model 18 is replaced with the face image data acquired by the shooting unit 11. The update unit 16 specifies the teacher data whose variance of the calculated shooting conditions is smaller than that before the replacement as the replacement target. The updating unit 16 preferably specifies the teacher data as the replacement target, which minimizes the dispersion of the shooting conditions by replacing the face image data acquired by the shooting unit 11. The update unit 16 creates a new teacher data set by replacing the specified teacher data with the face image data acquired by the shooting unit 11. The update unit 16 updates the determination data model 18 by inputting a new teacher data set into a classifier such as SVM. If any of the teacher data used for constructing the determination data model 18 is replaced with the face image data acquired by the shooting unit 11, the updating unit 16 replaces the teacher data when the dispersion of the shooting conditions is not reduced. It doesn't have to be.

Here, a specific example of the process for the update unit 16 to specify the teacher data to be replaced will be described. FIG. 4 is a diagram showing an example of shooting conditions for face image data acquired by the shooting unit in the embodiment. In FIG. 4, as in FIG. 3, each item of “PID”, “shooting date” and “shooting condition” is included, and the item of “shooting condition” is each of “direction”, “distance” and “light and darkness”. Includes items. Since each item of "PID", "shooting date" and "shooting condition" is the same as that in FIG. 3, the description thereof will be omitted. The update unit 16 causes the calculation unit 15 to calculate the variance when the imaging data (that is, the teacher data) registered in the shooting condition management table 191 is replaced with the face image data acquired by the shooting unit 11.

FIG. 5 is a diagram showing an example of dispersion when the teacher data is exchanged. In FIG. 5, for the sake of explanation, the total variance when the teacher data is replaced with the face image data acquired by the photographing unit 11 is illustrated together with the photographing condition management table 191 as “updated variance”. With reference to FIG. 5, it can be understood that any teacher data to be replaced is improved (the total variance becomes smaller) than the current total variance "42.7". Therefore, the update unit 16 may specify any of the teacher data as the update target.

Further, referring to FIG. 5, for example, when the teacher data of PID "1" or PID "16" is replaced with the face image data acquired by the imaging unit 11, the total variance becomes "42.7", and the variance after the replacement becomes "42.7". The total value can be the smallest. That is, by replacing them, the variation in the appearance frequency for each shooting condition can be minimized. Therefore, it is more preferable to specify the teacher data of PID "1" or PID "16" as a replacement target.

When there are a plurality of teacher data that minimize the total variance after replacement, such as the teacher data of PID "1" and PID "16", the update unit 16 selects, for example, the teacher data having the oldest shooting date and time. It may be specified as a replacement target. In the case of FIG. 5, the shooting date and time of the teacher data of PID "1" is "July 9, 2017", and the shooting date and time of the teacher data of PID "16" is "July 7, 2017". Since the shooting date and time of the teacher data of PID "16" is older than that of the teacher data of PID "1", the update unit 16 may specify the teacher data of PID "16" as a replacement target. The update unit 16 is an example of the “update unit”.

The initial construction unit 17 initially constructs the determination data model 18 using a plurality of imaging data captured by the camera 104 under various shooting conditions as a teacher data set. The initial construction unit 17 starts the initial construction of the determination data model 18, for example, when the user instructs the start of the initial construction. The initial construction of the determination data model 18 includes, for example, constructing a new model for estimating the user. The initial construction unit 17 outputs imaging data under various shooting conditions to the user, for example, by outputting instructions such as "turn to the right" and "closer" from the speaker 106 to perform shooting. To get. The initial construction unit 17 performs face recognition processing on the imaged data acquired by having the camera 104 photograph the user, and extracts the face image data in which the user's face is photographed from the imaged data. The initial construction unit 17 stores the extracted face image data in the auxiliary storage unit 103 in association with a UID that uniquely identifies the face image data. The initial construction unit 17 inputs the plurality of face image data stored in the auxiliary storage unit 103 as a teacher data set into a classifier such as an SVM, and performs initial construction of the determination data model 18.

The initial construction unit 17 stores the constructed determination data model 18 in the auxiliary storage unit 103. Since the storage capacity of the auxiliary storage unit 103 is finite, the capacity of the auxiliary storage unit 103 that can be used by the determination data model 18 is limited. Therefore, the number of teacher data used for constructing the determination data model 18 is limited. The number of teacher data used for constructing the determination data model 18 may be appropriately determined according to the capacity of the auxiliary storage unit 103. In the present embodiment, for example, the number of teacher data used for constructing the determination data model 18 is "20".

The initial construction unit 17 further determines the shooting conditions for each of the face image data. Examples of the shooting conditions include "direction", "distance", and "light and darkness", which are exemplified with reference to FIG. 3 for the shooting condition management table 191. For example, the initial construction unit 17 determines the front direction, left direction, right direction, upward direction, and downward direction from the positional relationship between the face contour and the eyes, nose, and mouth in the face image data with respect to the shooting condition "direction". For example, the initial construction unit 17 determines whether the shooting condition "distance" corresponds to "far", "medium", or "near" based on the size of the image size (number of pixels) of the face image data. To judge. For example, the initial construction unit 17 determines whether the shooting condition “light / dark” corresponds to “bright”, “medium”, or “dark” by using the luminance distribution of the face image data. The initial construction unit 17 stores the PID of the face image data, the shooting date and time of the imaging data, and the determined shooting condition in association with each other in the shooting condition management table 191.

<Processing flow>
The processing flow of the communication device 1 according to the embodiment described above will be described with reference to the drawings.

(Initial construction of judgment database 18)
FIG. 6 is a diagram showing a processing flow of initial construction of the determination database in the embodiment. The processing flow of FIG. 6 is started, for example, when the user instructs the start of initial construction. Hereinafter, the processing flow of the initial construction of the determination database 18 in the embodiment will be described with reference to FIG.

In T1, the initial construction unit 17 outputs an instruction to the user (for example, "turn right", "closer", etc.) from the speaker 106, and causes the camera 104 to take a picture of the user. In T2, the initial construction unit 17 detects the face region from the imaged data captured in T1. In T3, it is determined whether or not the imaged data captured by the initial construction unit 17 and T1 includes a face. That is, the initial construction unit 17 determines whether or not the face region can be detected by T2. If the captured data includes a face (YES at T3), the process proceeds to T4. If the captured data does not include a face (NO at T3), the process proceeds to T1.

In T4, the initial construction unit 17 extracts face image data from the imaged data captured in T1. In T5, the initial construction unit 17 extracts the shooting conditions of the face image data extracted in T4. For example, the initial construction unit 17 analyzes the face image data extracted in T4 and extracts shooting conditions such as the orientation of the user's face, the distance between the communication device 1 and the user, and the brightness (illuminance) around the user. To do.

In T6, the initial construction unit 17 acquires the shooting date and time of the shooting data shot in T1. The initial construction unit 17 acquires, for example, the shooting date and time from the Exif data of the imaging data. In T7, the initial construction unit 17 stores the face image data extracted in T4 and the UID that uniquely identifies the face image data in the auxiliary storage unit 103 in association with each other.

In T8, the initial construction unit 17 updates the shooting condition management table 191 in association with the shooting conditions extracted in T5, the shooting date and time acquired in T6, and the UID that uniquely identifies the face image data. The processing of T1 to T8 is executed, for example, 20 times repeatedly, so that 20 face image data taken under various shooting conditions are collected.

In T9, the initial construction unit 17 constructs the determination data model 18 by inputting the 20 face image data stored in the auxiliary storage unit 103 in T6 as a teacher data set into a classifier such as SVM.

(Update of judgment data model 18)
7 and 8 are diagrams showing an example of the processing flow of the determination data model update processing in the embodiment. “A” in FIG. 7 is connected to “A” in FIG. 8, and “B” in FIG. 7 is connected to “B” in FIG. Hereinafter, an example of the processing flow of the update processing of the determination data model 18 will be described with reference to FIGS. 7 and 8.

The processing of T1 to T4 is the same as the processing of T1 to T3 in FIG. 6, except that the main body of the processing is replaced by the photographing unit 11 in which the approach of the user is detected by the sensor 108 from the initial construction unit 17. Therefore, the description thereof will be omitted.

In T11, the estimation unit 12 estimates the user's name using the determination data model 18 based on the face image data extracted in T4. The determination data model 18 has already been constructed by, for example, the initial construction process described with reference to FIG.

At T12, the dialogue unit 13 outputs a question including the user's name estimated in T11 from the speaker 106. The question may be, for example, a greeting. The dialogue unit 13 outputs, for example, a voice question "Good morning, Mr. A mountain B Taro" from the speaker 106. The dialogue unit 13 receives a response from the user to the question output from the speaker 106 via the microphone 105. The dialogue unit 13 acquires the user's name from the response received via the microphone 105.

In T13, the determination unit 14 determines whether or not the user's name estimated by the estimation unit 12 in T11 is valid based on the response received by the dialogue unit 13 in T12. If it is valid (YES in T13), the process ends without updating the determination data model 18. If not valid (NO at T13), processing proceeds to T4.

Since the processing of T5 to T7 is the same as the processing of T4 to T6 in FIG. 6 except that the main body of the processing is changed from the initial construction unit 17 to the update unit 16, the description thereof will be omitted. In T14, the update unit 16 identifies the shooting condition management table 191 to which the user's name acquired in T12 is given as a label. The update unit 16 causes the calculation unit 15 to calculate the total variance of the shooting conditions when the teacher data registered in the specified shooting condition management table 191 is replaced with the face image data extracted in T4. The update unit 16 specifies teacher data in which the total variance of the shooting conditions at the time of replacement with the face image data extracted in T2 is equal to or less than the total variance before replacement as the teacher data to be replaced.

Note that the update unit 16 may determine that there is no teacher data to be updated if the total variance of the shooting conditions is larger than that before the replacement regardless of which teacher data is replaced. If there is teacher data to be updated (YES at T14), the process proceeds to T15. If there is no teacher data to be updated (NO in T14), the process ends without updating the determination data model 18.

In T15, the update unit 16 replaces the teacher data specified in T14 with the face image data extracted in T4 among the teacher data of the determination data model 18 using the user's name acquired in T12 as the model name. Determine a new teacher dataset. In T16, the update unit 16 inputs the teacher data set determined in T15 into the SVM and updates the determination data model 18.

<Action and effect of the embodiment>
In the embodiment, the communication device 1 estimates a user based on the imaged data (face image data) captured by the image capturing unit 11, and asks a question including the estimated user's name. When the response to the question indicates that the estimated person and the user are different, the communication device 1 takes an image so that the total variance of the shooting conditions of the teacher data used in the determination data model 18 is equal to or less than the total variance before replacement. Either the imaging data captured by the unit 11 or the teacher data is exchanged. That is, according to the embodiment, the determination data model 18 can be updated in a dialogue with the user without issuing an instruction relating to the shooting conditions to the user, and thus the determination data model 18 is updated. The burden on the user can be reduced.

The communication device 1 according to the embodiment can reduce the burden on the user related to the update of the determination data model 18 even when the display is not provided. For example, when the communication device 1 has a display, the user determines whether or not the face image data used as the teacher data of the determination data model 18 is photographed under various shooting conditions (that is, the distribution of shooting conditions). Is small or not) can be confirmed using a display, and the shooting conditions that are insufficient as teacher data can be grasped. On the other hand, when the communication device 1 does not have a display, it is difficult for the user to confirm the face image data as the teacher data. According to the embodiment, since the update unit 16 updates the face image data taken under various shooting conditions as teacher data, it is suitable for estimating a person without confirmation and correction by the user. Judgment data model 18 can be constructed.

Here, since the variance indicates the variation of the teacher data for each shooting condition, the updating unit 16 may use either the imaging data or the teacher data captured by the imaging unit 11 so as to reduce the variation in the appearance frequency for each shooting condition. It is also possible to replace the heel. By exchanging the teacher data in this way, the determination data model 18 can use the imaging data (face image data) taken under various shooting conditions as the teacher data. Since the determination data model 18 is constructed based on such teacher data, the communication device 1 can improve the estimation accuracy of the user under various conditions.

Further, since the teacher data is replaced as described above, even if the number of teacher data is limited to a predetermined number (20 in the embodiment), the update unit 16 may perform various shootings within the limited number. The determination data model 18 can be constructed using the imaging data taken under the conditions as the teacher data. Therefore, according to the embodiment, the estimation accuracy of the user can be improved even if the number of teacher data is limited. Further, since the determination data model 18 is constructed based on the teacher data limited to a predetermined number, the utilization efficiency of the main storage unit 102 and the auxiliary storage unit 103 is compared with the case where the number of teacher data is not limited. Can be enhanced.

In the embodiment, when there are a plurality of teacher data having the same variance after replacement with the face image data acquired in T2 of FIG. 5, the update unit 16 replaces the teacher data with the oldest shooting date and time. Identify as. The user's face changes with the passage of time due to the effects of growth and aging. In the embodiment, the accuracy of estimating a person by the communication device 1 can be improved by preferentially replacing the teacher data having an old shooting date and time.

<First modification>
In the embodiment, the face image data acquired in T4 of FIG. 7 and the teacher data are exchanged based on the total variance of the shooting conditions. In the first modification, the face image data acquired in T4 of FIG. 7 and the teacher data are exchanged based on the variance calculated for each of a plurality of categories in the shooting conditions. The same components as those in the embodiment are designated by the same reference numerals, and the description thereof will be omitted. Hereinafter, the first modification will be described with reference to the drawings.

FIG. 9 is a diagram showing an example of a processing block of the communication device according to the first modification. The communication device 1a illustrated in FIG. 9 is different from the communication device 1 according to the embodiment in that the update unit 16a is provided in place of the update unit 16.

The update unit 16a refers to the shooting condition management table 191 of the management database 19, and refers to the category with the largest classification among the shooting condition categories (“direction”, “distance”, and “brightness” in the example of FIG. 3). To identify. The update unit 16a specifies teacher data that can be replaced with the face image data acquired in T2 of FIG. 5 so that the classification of the specified category can be set to a value equal to or less than that before the replacement. The update unit 16a replaces the specified teacher data with the face image data acquired in T2 of FIG. 5 to determine a new teacher data set. The update unit 16a inputs the determined new teacher data set into the SVM and updates the determination data model 18.

According to the first modification, by exchanging the teacher data to be exchanged based on the variance of each category of shooting conditions, the estimation accuracy in the category where the estimation accuracy is lower than that of other categories can be improved. It can be carried out.

<Other variants>
In the embodiment, a greeting including the user's name such as "Good morning, Mr. A mountain B Taro" is given as an example of the question by the dialogue unit 13, but the question by the dialogue unit 13 is not limited to the greeting. .. The question asked by the dialogue unit 13 may be, for example, calling out, "Mr. A, B, Taro, I will inform you of yesterday's health data." Further, the imaging unit 11 may take a picture of the user facing the direction of the communication device 1 in response to the call, and the estimation unit 12 may further perform the estimation of the user. When the result of the estimation executed again by the estimation unit 12 is "C field D man" different from "A mountain B Taro", the teacher of the judgment data model 18 corresponding to "C field D man" by the update unit 16 The data may be updated.

The communication device 1 may store information such as a user's schedule and hobbies in the auxiliary storage unit 103, and may ask the user based on the stored information. For example, when the communication device 1 holds in the auxiliary storage unit 103 that the user "A mountain B Taro" went to the concert yesterday, the speaker 106 asks "Did you enjoy the concert yesterday?" Output from. If you receive a response that does not deny that you went to the concert, for example, "Yes, it was fun", you can judge that the presumption that the user was "A mountain B Taro" was valid. .. Further, when the communication device 1 receives a response denying that the user has gone to the concert, for example, "No, the concert has not been performed", the presumption that the user is "A mountain B Taro" is not valid. It can be determined that it was.

As another example of using such information such as schedule and hobbies, the case where the number of steps of the person estimated by the estimation unit 12 yesterday is 10,000 steps can be mentioned. In this case, the dialogue unit 13 outputs a question that "I walked a lot yesterday, it's 10,000 steps." When the response to this question is "Yes, it was a good exercise", the determination unit 14 can determine that the estimation by the estimation unit 12 is appropriate. Further, when the response "No, I haven't walked so much" is received, the determination unit 14 can determine that the estimation by the estimation unit 12 is not valid.

As an example of using information such as schedules and hobbies, there is a case where an event about the user's favorite XX estimated by the estimation unit 12 (A mountain B Taro) is held tomorrow. In this case, the dialogue section 13 outputs a question that "A mountain B Taro's favorite XX event will be held tomorrow." When receiving a response that does not deny the hobby of "A mountain B Taro" that he likes XX saying "Let's pass this time" as a response to this question, the estimation unit 14 is appropriate for the estimation unit 12. It can be determined that there is. In addition, when a response denying the hobby of "A mountain B Taro" that "I do not like XX" that I like XX is received, the determination unit 14 is appropriate for the estimation by the estimation unit 12. It can be determined that there was no such thing. Information such as the date of the event related to XX may be acquired by the communication device 1 by appropriately searching for an information source such as the Internet.

Further, the communication device 1 may include a question in the response to the instruction from the user. For example, the communication device 1 may return a response of "Yes, Mr. A, Mr. B, Taro, I understand" to the instruction "Turn on the lighting" from the user. In this case, when a response denying that it is A mountain B Taro such as "No, I am C field D man" is received from the user, the presumption that the user is "A mountain B Taro" was not valid. Can be judged. If the communication device 1 does not respond from the user within a predetermined time, the user does not deny the estimated name, so it is determined that the estimation that the user is "A mountain B Taro" is appropriate. You may. As a similar example, when receiving an instruction "Tell me the number of steps yesterday" from the user, you may reply "Yes. I will inform you of the number of steps of Mr. A mountain B Taro".

The communication device 1 may instruct the user to exercise in order to acquire face image data under various shooting conditions. As instructions to encourage exercise, for example, "exercise a little. Slowly turn your neck three times" or "exercise a little. Close your eyes tightly for 10 seconds." ..

Even if the estimation by the estimation unit 12 is valid, the communication device 1 may execute the replacement of the teacher data when the teacher data of the determination data model 18 has not been replaced for a predetermined period (for example, one year). Good. As described above, since the user's face changes with the passage of time due to the influence of growth, aging, etc., it is considered that the estimation accuracy by the estimation unit 12 can be maintained high by exchanging the teacher data with the passage of a certain period of time.

In the embodiment, the dialogue unit 13 outputs a question including the name of the person estimated by the estimation unit 12. However, the question output by the dialogue unit 13 may include information indicating the estimated person instead of the user's name. Information indicating the estimated person may include, for example, a user's surname, a user's name, a user's nickname, and the like.

In the embodiment, three categories of shooting conditions are listed, "direction", "distance", and "light and darkness", but instead of or in addition to these, other categories are included in the shooting conditions. May be good. Other categories include, for example, "presence or absence of a hat", "presence or absence of eyeglasses", "expression" and the like. Further, in the embodiment, for each category of shooting conditions, "direction" is set to 5 directions, "distance" is set to 3 levels, and "light and darkness" is set to 3 levels, but each category of shooting conditions may be further subdivided. However, it may be divided more roughly.

The embodiments and modifications disclosed above can be combined with each other.

<< Computer-readable recording medium >>
An information processing program that enables a computer or other machine or device (hereinafter, computer or the like) to realize any of the above functions can be recorded on a recording medium that can be read by the computer or the like. Then, by causing a computer or the like to read and execute the program of this recording medium, the function can be provided.

Here, a recording medium that can be read by a computer or the like is a recording medium that can store information such as data and programs by electrical, magnetic, optical, mechanical, or chemical action and can be read from the computer or the like. To say. Among such recording media, those that can be removed from a computer or the like include, for example, a flexible disk, a photomagnetic disk, a Compact Disc Read Only Memory (CD-ROM), a Compact Disc-Recordable (CD-R), and a Compact Disc-ReWriterable. (CD-RW), Digital Versaille Disc (DVD), Blu-ray Disc (BD), Digital Audio Tape (DAT), 8 mm tape, memory cards such as flash memory, and the like. In addition, there are hard disks, ROMs, and the like as recording media fixed to computers and the like.

1, 1a ... Communication device 11 ... Shooting unit 12 ... Estimating unit 13 ... Dialogue unit 14 ... Judgment unit 15 ... Calculation unit 16, 16a ... Update unit 17 ... Initial construction unit 18 ... Judgment data model 19 ... Management database 191 ... Shooting condition management table 101 ... CPU
102 ... Main storage unit 103 ... Auxiliary storage unit 104 ... Camera 105 ... Microphone 106 ... Speaker 107 ... Timekeeping unit 108 ... Sensor

Claims (8)

It is a dialogue device that estimates the user captured by the camera and performs a dialogue with the estimated user.
A person estimation model generated by using multiple imaging data of a person photographed under multiple shooting conditions as teacher data, and
An estimation unit that estimates the user captured by the camera using the person estimation model,
A dialogue section that outputs questions that include information indicating the estimated person,
When the response to the question indicates that the user and the estimated person are different, one of the teacher data and the imaging data acquired when the user is photographed by the camera are used as the teacher data. It is equipped with an update unit that replaces data so that the variation in appearance frequency for each shooting condition is reduced.
Dialogue device. The dialogue unit outputs a question including information indicating the estimated person as a response to the instruction from the user.
The dialogue device according to claim 1. The estimation unit causes the camera to take a picture of the user who responded to the question again, and further estimates the user using the imaged data taken again.
The dialogue device according to claim 1 or 2. When there are a plurality of teacher data capable of reducing the variation in the appearance frequency for each shooting condition by replacing the imaging data captured by the camera, the updating unit reduces the variation in the appearance frequency for each shooting condition. The teacher data with a large amount of reduction is prioritized and replaced with the imaged data captured by the camera.
The dialogue device according to any one of claims 1 to 3. When there are a plurality of teacher data capable of reducing the variation in the appearance frequency for each shooting condition by exchanging the imaging data captured by the camera, the updating unit gives priority to the teacher data captured at an older time. The target is to be replaced with the imaged data taken by the camera.
The dialogue device according to any one of claims 1 to 4. Even if the response to the question is the same between the user and the estimated person, the update unit will use the teacher data if a predetermined period has passed since the previous replacement of the teacher data. And the imaging data captured by the camera are replaced so that the variation in the appearance frequency of the imaging data as the teacher data for each imaging condition is reduced.
The dialogue device according to any one of claims 1 to 5. A computer that estimates the user captured by the camera and interacts with the estimated user
The user captured by the camera is estimated by using a person estimation model generated by using a plurality of imaging data of a person photographed under a plurality of shooting conditions as teacher data.
Outputs a question containing information indicating the estimated person,
When the response to the question indicates that the user and the estimated person are different, one of the teacher data and the imaging data acquired when the user is photographed by the camera are used as the teacher data. Replace the data so that the variation in the frequency of appearance for each shooting condition is reduced.
Information processing method. To a computer that estimates the user in the camera and interacts with the estimated user
The user captured by the camera is estimated by using a person estimation model generated by using a plurality of imaging data of a person photographed under a plurality of shooting conditions as teacher data.
Output a question containing information indicating the estimated person,
When the response to the question indicates that the user and the estimated person are different, one of the teacher data and the imaging data acquired when the user is photographed by the camera are used as the teacher data. Replace the data so that the variation in appearance frequency for each shooting condition is reduced.
Information processing program.

Images