JP2019217558A - Interactive system and control method for the same - Google Patents

Abstract

To provide an interactive robot system that narrows down an interactive object person in advance by determining whether or not the person has an interactive intention and takes an interest in dialogues, before the person gets close to a robot, when the robot is utilized in an environment where a plurality of persons come and go, and a control method for the interactive robot.SOLUTION: An interactive robot system includes: imaging equipment for taking an image of its surroundings; and a calculator which detects a person from image information from the imaging equipment, which tracks the detected person by a plurality of images taken by the imaging equipment, which calculates the extent of interest in the tracked person on the basis of changes in the orientation of his/her face and the orientation of his/her body in the plurality of images, and which regards him/her as an interactive candidate on the basis of the calculated extent of the interest.SELECTED DRAWING: Figure 7

Description

  The present invention relates to a dialogue system, and more particularly to a dialogue robot system that is used in an environment where a plurality of people come and go, identifies a person who can be a dialogue target, and prepares for a dialogue in advance, and a dialogue robot control method.

  2. Description of the Related Art In recent years, robots have been actively developed in retail stores and public facilities to provide interactive services on behalf of employees to visitors. In particular, when using a robot in an environment with many people, the robot must respond by selecting a person to be a conversation target from a plurality of people moving around.

  A technology in which a robot selects a conversation target from a plurality of persons is described in Patent Literature 1. This patent document 1 defines a degree of interest of a person existing in an area near a robot, and selects a person with a high degree of interest as a conversation target. Here, the degree of interest is scored according to the direction of the face, the direction of the line of sight, the presence or absence of a gesture or an utterance for each person, and ranks the dialogue targets according to the score.

JP 2009-248193 A

  The technique in Patent Literature 1 performs scoring based on the direction of the face, the direction of the line of sight, and the presence or absence of gestures and utterances for each person, and functions in a situation where people have already gathered around the robot.

  However, a dialogue robot is actually installed in an environment where people come and go, selects a person who can be a conversation target from among those people, actively speaks, and can actually speak to the person It is necessary to turn to the body before and prepare for dialogue such as capturing and recognizing a person with a camera.

  The interactive robot cannot judge whether a person can be a conversation target until the person enters a predetermined area or actually speaks to the person. In addition, the robot does not support determining whether a person who has entered a designated area passes through or has a dialogue intention before the person actually speaks to the robot.

  Therefore, the problem of the present invention is that when a robot is used in an environment where a plurality of people come and go, the robot determines the degree of interest of the person to determine whether the person has a dialogue intention or interest, and determines the degree of interest of the person beforehand. An object of the present invention is to provide a dialogue robot system and a control method of a dialogue robot for narrowing down persons.

  A representative aspect for solving the above-described problem is an imaging device that images the surroundings, a person is detected from image information from the imaging device, and the detected person is tracked with a plurality of images of the imaging device. A computer that calculates the degree of interest of the tracked person based on changes in the orientation of the person's face and body in a plurality of images, and sets the degree of interest as a dialogue candidate based on the calculated degree of interest.

  Before the person approaches the robot, the robot can narrow down persons who are willing or interested in talking to the own device.

It is a schematic diagram of an interactive robot system. FIG. 2 is a diagram illustrating an example of a hardware configuration of an interactive robot system. FIG. 3 is a block diagram illustrating a functional configuration example of the interactive robot system. FIG. 9 is a flowchart showing a specific processing procedure of the first estimation processing 01. 11 is a flowchart for selecting a method for a robot to work on a dialogue candidate determined to have a degree of interest. It is a figure showing the flow chart which showed the 2nd presumption processing 02 at the time of acting on the dialogue candidate. It is a figure which shows the flowchart which a conversation robot system identifies the person who becomes a conversation object. It is a figure which shows the relationship between the change of the positional relationship of a robot and a person, and the degree of interest. FIG. 3 is a diagram illustrating a state of a person moving for three frames with respect to a robot. 9 is a table showing an example of calculating a degree of interest for a movement of a person for three frames with respect to a robot.

  Hereinafter, each embodiment will be described with reference to the drawings.

  FIG. 1 is a schematic diagram of the interactive robot system. An example of a use state of a dialogue robot system (hereinafter, dialogue system) 100 in an environment where a person comes and goes will be described. The interactive system 100 includes an interactive robot 110 that interacts with a person (hereinafter, simply referred to as a robot) and a remote server 130 that controls the robot 110 based on a signal from the robot 110.

  The robot 110 includes a camera 120, a speaker 121, a microphone array 122, an internal server 123, a display device 124, a driving device 125, and a first communication interface (hereinafter, referred to as IF) 126. The remote server 130 is a computer that sends a control signal for controlling the operation of the robot 110, and includes a second communication IF 131 that communicates with the first communication IF 126. Note that the first communication I / F 126 and the second communication IF 131 are wireless interfaces, and include a LAN system that transmits and receives data using wireless communication, for example, a LAN system defined by IEEE 802.11. The first communication IF and the second communication IF may be via a network such as the Internet.

  The camera 120 is an imaging device that captures images, and the microphone array 122 captures environmental sounds and voices of people. The display device 124 presents information to a person, and is, for example, a display or a projection image. Further, the face and expression of the robot 110 may be expressed. The drive unit 125 is located at a joint such as an arm or a leg of the robot 110, and is, for example, a motor or a speed reducer that implements an operation for expressing an emotion or moves.

  The internal server 123 is a computer, and transmits data obtained by the camera 120 or the microphone array 122 to the remote server 130 via the first communication IF 126. Further, the remote server 130 includes a second communication IF 131, receives images, audio data, and signals from the first communication IF 126, and transmits a signal for controlling the robot 110 according to the received signal to the second communication IF 131 and the first communication IF 126. To the robot 110.

  In addition, the function of controlling the robot 110 of the remote server 130 may be processed by the internal server 123. In this case, the remote server 130 and the first communication interface 126 are not required, and the robot 110 independently interacts with a person. It can be. The dialogue system having the above configuration is mainly used as a dialogue robot that works on a dialogue with a person around the robot or performs a dialogue.

<System configuration example>
FIG. 2 is a diagram illustrating an example of a hardware configuration of a system of the robot 110 and the remote server 130 included in the interactive system 100.

  The robot 110 includes a camera 120, a microphone array 122, and a first output device 140, and these are connected to an internal server 123 by a bus 129. The first output device 140 includes a speaker 121, a display device 124, and a driving device 125. The internal server 123 has a first processor 127, a first storage device 128, a first communication interface 126, and a bus 129 connecting them. Further, the camera 120 may be a depth sensor.

  The first processor 127 controls the output device 140 provided in the robot 110 and implements the function of the internal server 123. The first storage device 128 is a work area for the first processor 127, and is a non-temporary or temporary storage medium for storing various programs and data for realizing functions. The first storage device 128 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a HDD (Hard Disk Drive), a GPU (Graphics Processing Unit), and a flash memory. The first output device 140 includes, for example, a display device 124 and a speaker 121. The first communication IF 126 performs wireless communication with the remote server 130 or connects via a network (not shown) to transmit and receive data.

  The camera 120 is an imaging device that captures an image of the periphery of the robot 110, and may have, for example, a three-dimensional measurement function that can measure a distance to a subject. The driving device 125 may be, for example, a motor, and is a mechanism that drives the robot 110. For example, the robot 110 may be moved by a walking motion or wheels, the arms or fingers of the robot 110 may be moved to express the emotions of the robot 110, or the driving of changing the direction of the camera 120 by shaking the head. Device.

  The remote server 130 has a second communication IF 131, a second processor 132, a second storage device 133, and a bus 134 connecting these. The second storage device 133 is a work area for the second processor 132, and the second storage device 133 is a non-temporary or temporary storage medium for storing various programs and data for realizing the functions of the remote server. Examples of the second storage device 133 include a ROM, a RAM, an HDD, a GPU, and a flash memory. The second communication IF 131 performs wireless communication with the robot 110 or connects via a network (not shown) to transmit and receive data.

<Example of functional configuration of control system>
FIG. 3 is a block diagram illustrating a functional configuration example of the interactive system 100.

  The internal server 123 includes an image receiving unit 120A for receiving image data from the camera 120, a first communication IF 126 for transmitting and receiving data to and from the remote server 130, and an output device control unit 303 for controlling the speaker 121 and the display device 124. , A drive control unit 304 for controlling the drive device 125. The output device control unit 303 and the drive control unit 304 are realized by the first processor 127 executing a program stored in the first storage device 128. For example, the control of the first output device by the output device control unit 303 and the control of the drive device 125 by the drive control unit 304 are realized so as to have a conversation with a person using the microphone array 122 and the speaker 121. In addition, the operation for expressing the emotion of the robot 110 and the movement of the robot 110 are realized.

  The remote server 130 includes a second communication IF 131, a person detection unit 312, a person feature extraction unit 313, a person tracking unit 314, a time series feature extraction unit 315, an interest behavior identification unit 316, and a reaction confirmation unit 317. And The functions of the human detection unit 312, the human feature extraction unit 313, the human tracking unit 314, the time-series feature extraction unit 315, the interest behavior identification unit 316, and the reaction confirmation unit 317 are stored in the second storage device 133. This is realized by the second processor 132 executing the stored program. The human feature extraction unit 313 includes a head detection unit 321, a head orientation unit 322, and a body orientation unit 323.

  The environment around the robot 110 captured by the camera 120 is transmitted as image information to the remote server 130 via the image receiving unit 120A, the first communication IF 126, and the second communication IF 131.

  The person detection unit 312 estimates an area where a person exists from image information from the camera 120. The region where the person exists may be a rectangular position surrounding the region of the person. The region estimated by the human detection unit 312 or image information in the region is transmitted to the human feature extraction unit 313 and the human tracking unit 314. The human detection process performed by the human detection unit 312 is currently known, and specifically, the presence of a person is estimated by characterizing the contour of the person from the gradient with the surrounding pixel values for each pixel. It may be represented by a rectangle indicating the presence of a person by using Convolutional Neural Network (hereinafter, CNN) using a convolution filter. By using the above-described technology, the human detection unit 312 can acquire area information for each person even when a plurality of persons exist.

  The human feature extraction unit 313 extracts, for example, a feature in an image of a person from a region for each person obtained by the person detection unit 312. The features extracted by the human feature extracting unit 313 are features of each person in a human area in one piece of image information. Specifically, the human feature extraction unit 313 includes a head detection unit 321, a head direction determination unit 322, and a body direction determination unit 323. The feature of the person obtained by the person detection unit 312 may be used when the time-series feature extraction unit 315 extracts the feature of each person that is temporally continuous.

  The head detection unit 321 detects a region of the head in the region extracted by the human detection unit 312. Specifically, the region of the head may be a rectangular position surrounding the head, and a detector based on CNN (Conventional Neural Network) is used. When the person is facing the front, the face may be detected. For face detection, Harr-Like features are used. For face detection, a detector using CNN may be used.

  The head orientation unit 322 estimates the direction of the head using information in the region of the head identified by the head detection unit 321. As a head direction estimating method, if the face can be used, the feature points of the face that are currently known in image processing are extracted, and the face direction is determined from the arrangement of the feature points on the image. May be used. Alternatively, a threshold value may be determined for a value obtained based on the above-described direction estimation result to determine whether the camera 120 faces the camera 120 or not. Further, for example, learning may be performed using a classifier that outputs a head direction by inputting a human head image CNN, or a direction value may be directly output. The learning device may be learned in advance, and the feature value of the intermediate layer of the classifier may be used at the time of execution.

  The torso determining unit 323 estimates the direction of the person's body in the area obtained from the person detecting unit 310. As a means for estimating the direction of the body, if the camera 120 is a depth sensor, the skeleton of the person is estimated based on the machine learning estimation using the depth image as an input, and the body position is estimated based on the estimated position of the skeleton. Determine the direction. Alternatively, a discriminator for estimating the body orientation may be created using a number of cases in which a human image and a corresponding body orientation label are provided, and used for the determination. At this time, the above-described classifier may directly output the direction value, or may output the intermediate layer output of the classifier.

  In the above description, the human feature extraction unit 313 outputs the head direction and the body direction separately, for example. However, the human feature extraction unit 313 can also create a classifier based on an image of a person and a label corresponding to the image. Specifically, images of a person and labels indicating whether or not the person actually interacted may be collected as cases, and the judgment may be made based on the cases.

  The person tracking unit 314 takes the correspondence of the same person between consecutive image frames based on the person region estimated by the person detection unit 313. The human tracking technique between consecutive frames compares the feature amount in the area of the output of the human detection unit 313 with the feature amount in the consecutive frames. Often, correspondence between frames is performed.

  The time-series feature extraction unit 315 extracts a time-series behavior feature for each person from the human feature extraction unit 313 and the person tracking unit 314. Specifically, the movement direction of the person is extracted from the transition of the torso direction over a plurality of time frames of the torso setting unit 323. Further, a time-series feature of a person approaching or passing by the robot 110 may be extracted. This specific processing will be described later.

  The interest behavior identification unit 316 identifies whether each person in the image has a degree of interest in the robot 110 based on the time series feature extracted by the time series feature extraction unit 315, and determines whether the person is a conversation candidate. judge. If it is determined that the interest level of the person is high, the person is determined to be a conversation candidate, and the output device control unit 303 and the drive control unit 304 are controlled via the second communication IF 131 and the first communication IF 126. Send a signal. The control signal will be described later.

  The output device control unit 303 that has received the control signal controls the speaker 121 and the display device 124 of the first output device 140, and the drive control unit 304 controls the drive device 125. The output device control unit 303 changes the display of the display device 124 and calls out from the speaker 121 to the person, for example. The drive control unit 304 performs an action such as an operation such as beckoning by controlling the drive device 125.

  In the reaction confirmation unit 317, for example, the speaker 121 of the first output device 140 controlled by the output device control unit 303, the display device 124, and the drive device 125 controlled by the drive control unit 304 act. Check the response of the person. The output device control unit 303 and the drive control unit 304 may detect whether or not a change in the reaction of a person correlated with the approach is obtained at a time close to the time when the approach is performed.

<Process for identifying conversation target>
FIG. 7 shows a flowchart executed by the dialog system 100 to identify a person to be talked to.

  First, in step S701, the camera 120 of the robot 110 captures an image of the surroundings and transmits image information to the human detection unit 311 of the remote server 130. The human detection unit 311 acquires the transmitted image information.

  Next, in step S702, the person detection unit 312 performs a person detection process based on the acquired image information. In the human detection process, the human detection unit 312 determines whether or not a person exists, and if there is, obtains an area of the person for each person in the form of, for example, a rectangular area.

  Next, in step S703, the person tracking unit 314 receives the output of the person detection unit 312, determines whether the person detected in the current acquisition frame has been detected in the most recent past acquisition frame, and determines between the frames. Perform person correspondence. On the other hand, when there is no person corresponding to the latest past acquired frame, the person tracking unit 314 determines that a new person has been detected and registers the new person in the second storage device 133. The feature is stored for the new person, and the person tracking unit 314 executes the association in the next and subsequent acquired frames. The human tracking technique used here is realized, for example, by measuring the degree of similarity of the image feature amount in the area of the person. In the human tracking technology, it is known that even if a tracking target is lost from an image due to an obstruction or the like, tracking can be continued if the tracking target subsequently appears.

  Next, in step S704, the time-series feature extraction unit 315 and the interest behavior identification unit 316 determine whether the person has a conversation intention or interest in the robot 110, and determines that the person is a conversation candidate. One estimation process 01 is performed. The person determined to have an intention or interest in the dialogue in the first estimation process 01 is a dialogue candidate (S705). The specific processing of step S704 will be described later.

  In step S706, the robot 110 works on the person determined to be a conversation candidate by the first estimation processing 01. The specific processing of the action will be described later.

  In step S707, the reaction confirming unit 317 observes the response of the person to the action of the robot 110 that has acted in step S706, and if it confirms the reaction to the act, determines that the object is a dialogue target.

  In step S708, preparations are made for a dialogue with the person determined to be a dialogue target in step S707. Specifically, for example, if the driving device 125 has a moving function of the robot 110, the user may walk to the conversation target before conducting the conversation. Alternatively, if the driving device 125 has a turning function of the robot 110, the orientation of the body of the robot 110 may be turned to the dialogue target before performing the dialogue. At this time, the camera 120 may be pointed at a person, and an image of the person may be taken from the front. If the first storage device 128 or the second storage device 133 is provided with means for estimating the appearance characteristics of the person, the estimation may be performed before the interaction with the captured image of the person. . Here, the appearance feature is, for example, the age and gender of the person based on the face image.

  In step S709, when it is determined that a person who actually has a dialogue intention does not have a dialogue intention, the robot 110 performs an exception process corresponding to a case where the robot 110 approaches the person and speaks.

  In step S710, for the person determined to be the conversation target by the second estimation process 02, or the person who talked to the robot 110 in step S709, for example, the utterance of the robot 110 by the speaker 121 and the driving The interactive service is provided by the gesture of the robot 110 by the device 125, the information presentation by the display device 124, and the like. In the dialogue with the person, for example, the robot 110 may change the tone, for example, based on appearance characteristics such as age and gender determined from the image of the person captured in step S708.

  In the first embodiment, a person's dialogue intention or interest in the robot 110 is determined in two stages using the first estimating unit in step S704 and the second estimating unit in step S707, thereby determining the person's interaction. The intention of conversation or the degree of interest can be accurately calculated.

<Specific processing of the first estimation processing 01>
FIG. 4 is a flowchart showing a specific processing procedure of the first estimation processing 01. In a first estimation process 01, a person detecting unit 312 detects a person, and a person tracking unit 314 determines whether a person who can be tracked between frames has a dialogue intention or interest level with the robot 110. To estimate dialog candidates.

  First, in step S404, the head detection unit 321 and the head orientation unit 322 estimate the direction of the head from the region of the head of the person and identify whether the head is facing. The determination of whether or not the person is facing is determined by determining the head direction of the person estimated by the head orientation determining unit 322 and a threshold value for determining whether or not the person is facing, Is determined. Further, a case of the face or the head facing this and a case of the face or the head not facing the head or the head may be collected, and a classifier may be created and used for the determination.

  In step S405, the second processor 132 records in the second storage device 133 the time Tf at which the second processor 132 determines in step S404 that the head is facing the user.

  In step S <b> 406, the body direction determination unit 323 and the time-series feature extraction unit 315 determine whether the person moves toward, away from, or passes through the person, or the traveling direction of the person. The determination is made by extracting the movement vector of the person. Alternatively, after a case of a person's movement is collected and a classifier is created, it may be used for determination.

  In step S407, the interest behavior identification unit 316 calculates the degree of interest in the robot 110 for each person in the current frame. For example, the score may be added based on the fact that the head is facing this direction and the torso of the person is approaching.

  Another method of calculating the score is an approaching motion, but when the head is not facing this direction, a score that takes into account the attenuation according to the difference between the time Tf and the current time can be added. If the operation is a straightforward operation that cannot be determined to be an approaching operation, the score may not be added. Alternatively, the interest behavior identification unit 316 may average and add the score calculated in step S407 to a plurality of calculated scores for each person to obtain the degree of interest of the target person. If the person shows the back of the head and keeps moving away for a predetermined time, the interest behavior identification unit 316 may reset or subtract the score.

  FIG. 8 is a diagram illustrating the relationship between the change in the positional relationship between the robot and the person and the degree of interest. The interest behavior identification unit 316 calculates the degree of interest of the person based on the robot 110 and changes in the behavior of the person within a predetermined time.

  FIG. 8A shows an example in which a person has moved from a position 810 to a position 811 via a route 812 toward the robot within a predetermined time. At this time, the direction of the head of the person is in the direction of the robot 110. In this example, the interest behavior identification unit 316 determines that there is a dialogue intention and adds the degree of interest (dialogue will score).

  FIG. 8B shows an example in which a person has moved from a position 820 to a position 821 via a path 822 toward the robot within a predetermined time. At this time, at the position 820, the head of the person is facing the robot 110, but not at the position 821. The interest behavior identification unit 316 adds the interest level score after attenuating it.

  FIG. 8C illustrates an example in which a person moves from a position 830 to a position 831 via a path 832, and the head is not turned to the robot. At this time, the interest behavior identification unit 316 does not add the interest degree score.

  FIG. 8D illustrates a case where the person reaches the position 841 from the position 840 via the path 842 that moves away from the robot, and the head of the person does not face the robot. At this time, the interest behavior identification unit 316 resets the score of the person's intention to interact or the degree of interest. Alternatively, it may be subtracted.

  Next, in step S408, the interest behavior identification unit 316 uses the scores calculated over a plurality of frames (calculated in step S407) as the intention of the person to interact with the robot 110 or the degree of interest. The calculated degree of interest is stored in the second storage device 133 as shown in FIG.

  FIG. 9 shows how a person moves by three frames with respect to the robot 110. FIG. 10 is a table showing an example of calculating the degree of interest with respect to the movement of the person for three frames with respect to the robot 110. In each case, based on the scoring in FIG. 8, an example of a method of calculating a dialogue intention or a degree of interest in the interest behavior identification unit 316 from the behavior of the person using the approaching behavior and head direction of the person. Is shown. In FIG. 10, as the scoring for each frame, the score of the first frame is C1, the score of the second frame is C2, the score of the third frame is C3, or time-averaged as an example of the scoring for three frames. Shows scoring. The interest level in FIG. 10 is stored in the second storage device 133 and can be used as a similar interest level table even when the robot 110 is actually used. In other words, in an environment where a plurality of people come and go, the IDs for identifying the plurality of people captured by the camera 120 are (A) to (D), and the time average interest level of each person is similarly calculated. Manage as a table.

  FIG. 9A shows a state in which a person approaches the robot 110 for three frames. A route 901, a route 902, and a route 903 indicate a moving route of the person in each frame. In each frame, the person's head is facing the robot 110, and the person is moving toward the robot 110. Approaching. This corresponds to the movement in FIG. 8A, and the interest behavior identification unit 316 sets the score of this movement to “1”. The scores in each frame are C1 = 1, C2 = 1, and C3 = 1, and the dialogue intention is evaluated as “1” based on the behavior of the person for three frames on a time average basis.

  FIG. 9 (b) shows three frames of the behavior of the person passing through the robot 110. A route 911, a route 912, and a route 913 indicate a moving route of the person in each frame, and correspond to the movement in FIG. 8C, and the interest behavior identification unit 316 sets the score to 0, for example. At this time, the interest action identification unit 316 sets the scores in each frame to C1 = 0, C2 = 0, and C3 = 0, and evaluates the dialogue intention from the behavior of three frames by time average to “0”.

  FIG. 9C shows three frames of the action of the person approaching the robot 110. The person initially points his head at the robot 110. The subsequent movement route 921, the route 922, and the route 923 approach the robot 110, but the head does not face the robot direction, and corresponds to the movement in FIG. 8B. On the route 921, since the approaching motion is one frame after the head stops moving in the robot direction, the interest action identifying unit 316 sets the score to C1 = 1/1. In the route 922, since the approach operation is performed two frames after the head stops moving in the direction of the robot 110, the score is set to C2 = 1/2. In the route 923, the score is set to C3 = 1/3 since three frames have elapsed since the head of the person no longer turns toward the robot 110. When the dialogue intention is evaluated from the behavior for three frames in FIG. 9C, the interest behavior identification unit 316 gives a score of, for example, 11/18 on a time average.

  FIG. 9D shows three frames of the behavior of the person in front of the robot 110. 8A in which the head is directed toward and approaching the robot 110 on the path 941, and the head is turned away from the robot 110 on the path 942 and moved away from the robot 110 on the path 943 in FIG. 8D. is there. The interest behavior identification unit 316 sets the score C1 of the route 941 to 1. The score C2 = 0 is set on the path 942 because of the stop action, and the score is reset on the path 943 because the head is moving away from the head. Therefore, in FIG. 9D, the intention of the person to interact is reset by the operation of FIG. 8D, and the interest behavior identification unit 316 becomes an interest degree score “0” if evaluated by time average.

  The interest behavior identification unit 316 identifies whether the person has a dialogue intention or interest in the robot 110 based on the features extracted by the time-series feature extraction unit 315, and calculates a score. At this time, the calculated score can be used for ranking for the robot 110 to select a dialogue candidate in an environment where a plurality of people come and go. Here, a threshold value may be provided, and persons not exceeding the threshold value may be excluded from the ranking. For example, in the examples of FIGS. 9 and 10, by setting the threshold value to 0.5, (a) and (c) can be identified as dialog candidates, ranked, and a person (b) passing by or a person moving away (D) can be excluded from the dialogue candidates. Also, by increasing the threshold value to, for example, 0.7, it is possible to exclude (C) approaching while looking away from the dialog candidates. For a person passing in front of the robot 110, the robot 110 can not be regarded as a conversation candidate, so that the calculation process can be simplified and the processing speed of the interest degree score can be increased. When a plurality of persons are candidates, the dialogue candidates can be determined by relative evaluation of the degree of interest of the person, such as the top two persons with the calculated degree of interest.

  Although FIG. 4 illustrates an example of the first estimation process 01, the interest behavior identification unit 316 is not limited to this, and may estimate the likelihood of the interest level using deep learning.

  Specifically, the time-series feature extraction unit 315 inputs a motion of a person who moves around the robot 110 as a moving image. A moving image is a set of continuous images, and a human feature extraction unit 313 extracts a feature amount of a person for each image frame. The feature amounts extracted here are the head-side fixed portion feature amount output by the head-side fixed portion 321, the body-side fixed portion feature amount output by the body-side fixed portion 323, the head-side fixed portion feature amount, and the body-side fixed portion The feature quantity may be a feature quantity obtained by combining the feature quantity with one. After that, the interest behavior identification unit 316 may extract the feature amount of each frame from all the frames of the moving image and output the likelihood of the interest level. At this time, as the input of the teacher data, a moving image of a person traveling in front of the robot may be used as an input, and a discriminator may be created based on an example of a teacher label indicating whether or not the robot has actually come, and used for determination. Note that the above-described method is an example of a means for identifying an intention of a dialogue of an approaching person, and is not limited thereto.

<Specific processing of approach>
FIG. 5 is a flowchart for selecting a method for the robot to work on a dialogue candidate determined to have a degree of interest. Specifically, the output device control unit 303 and the drive control unit 304 are further controlled by a control signal transmitted to the robot 110 when a dialog candidate exists. This processing is performed by the second processor 132 executing a program stored in the second storage device 133. Here, the control target by the control signal may be changed depending on the number of interested persons.

First, in step S501, the interest behavior identification unit 316 determines whether a dialogue candidate exists. If no interaction candidate exists, no control signal is sent.
Next, in step S502, the interest behavior identification unit 316 determines whether there are a plurality of interaction candidates. This is for selecting the output device 104 or the driving device 125 to be controlled depending on whether there are a plurality of dialog candidates or one.

  Next, when there is no plurality of conversation candidates, the process proceeds to step S503, and the person detection unit 313 determines whether there are a plurality of people.

  Next, when there is only one person, the process proceeds to step S504, and the interest behavior identification unit 316 sends a control signal to the output device control unit 303 to control the speaker 121. Specifically, control is performed so as to perform a greeting such as a greeting.

  If it is determined in step S503 that there are a plurality of persons, or if it is determined that there are a plurality of dialogue candidates in step S502, the process proceeds to step S505. In step S505, the interest behavior identification unit 316 sends a control signal for the display device 124 of the first output device 140 to the output device control unit 303. In this way, drawing is performed, such as displaying the robot's face on the display device 124, and the expression is changed, thereby acting on a conversation candidate with a high degree of interest. In addition, if the drive control unit 304 controls the drive device 125, specifically, the drive control unit 304 may express the intention of the robot 110 to talk by heading toward a person of high interest, waving a hand, or bowing. Good.

  Here, the reason why the control target is divided according to the number of people in the vicinity is that there is a fear that the voice call may draw attention of a person with a low degree of interest, so as to avoid affecting the determination result of the interest behavior identification unit 316. It is. If there is only one conversation candidate, a call may be made from a distance.

<Specific processing of the second estimation processing 02>
The reaction confirmation unit 317 observes the reaction of the person acted on by the output device control unit 303 or the drive control unit 304. Accordingly, the reaction confirmation unit 317 implements the second estimation process 02 and extracts a person who can be a conversation target from the conversation candidates.

  FIG. 6 is a flowchart illustrating a second estimation process 02 performed by the output device control unit 303 and the drive control unit 304 when acting on a dialogue candidate.

  First, in step S601, the output device control unit 303 and the drive control unit 304 control the output device 140 to work on a person who is a conversation candidate. In this step, the first processor 127 stores, in the first storage device 128, the time Ta at which the action was performed. This time Ta may be recorded in the second storage device 133.

  Next, in step S602, the reaction confirmation unit 317 determines a change in the response of the dialogue candidate to the action of the robot 110 controlled in step S601. Specifically, the reaction confirmation unit 317 determines that the head orientation detected by the head orientation unit 322 at each time is in the direction of the robot 110 within a very short time, such as within 1 second, with respect to the time Ta. When it changes so as to face, it is determined that it is a reaction of the dialogue candidate to the action of the robot 110, and the dialogue candidate is determined to be a dialogue target. In addition, for example, the reaction confirmation unit 317 determines that the advancing direction of the dialogue candidate is changed to the robot 110 at a close time, for example, within 5 seconds, with respect to the time Ta, by the body shape determination unit 323. Or, if it is determined that no change is made in the direction of the robot 110, it is determined that the object is to be interacted with.

  Next, in step S603, the reaction confirmation unit 317 causes the drive control unit 304 to direct the robot 110 in advance to the person determined to be the conversation target in step S602 or to move toward the person. To the control signal. A control signal may be transmitted to the output device control unit 303 so as to use the speaker 121 to speak to the conversation target.

  In step S602, when the reaction check unit 317 cannot check the reaction of the dialog candidate and cannot determine the reaction as a dialog target, the time-series feature extraction unit 315 determines whether the dialog candidate person approaches the robot 110. It is determined whether or not it is (step S604). The determination as to whether or not the approach is made here is made based on whether or not the conversation candidate person has entered the area around the robot 110. The size of this area may be changed according to the speed at which the target person approaches the robot. In addition, when the target, which may pass for a certain period of time, does not approach, or when the target person is at a certain distance from the robot 110, the processing for the person ends.

  Finally, in step S605, preparations are made for a dialogue with the person determined to be a dialogue target in step S603. For example, at the start of a dialogue with a person, if the driving device 125 has a turning function of the robot 110, the driving device 125 causes the robot 110 to take a facing posture with respect to the dialogue target. When the driving device 125 includes a moving unit, the drive control unit 304 causes the robot 110 to approach the vicinity of the dialogue target, and then the output device control unit 303 uses the speaker 121 to set the robot 110 to the dialogue target. You may call out.

  Specifically, for example, if the robot 110 takes a facing posture, the region is a range of a variable region determined based on the time required to complete the operation and the approach speed of the dialogue candidate. There may be.

  As described above, according to the dialogue system 100 shown in the present embodiment, the first estimation as a dialogue candidate from the characteristics of the approaching motion of a plurality of persons approaching from a distance, and the approach to the dialogue candidate are performed. Then, the second estimation of the dialogue from the reaction behavior of the dialogue candidate in response to this suggests that when the robot is used in an environment where multiple people are It is possible to provide a dialogue robot system and a control method for a dialogue robot, in which a determination is made before approaching and a person to be talked to is narrowed down in advance. In addition, since the robot can actively select and speak with a person, the robot can be effectively targeted for conversation. Further, by performing a process for recognizing a person, for example, by turning the camera before approaching, the external features of the person can be extracted before the start of the dialog, and can be reflected in the contents of the dialog.

100: Dialogue system, 110: Robot, 120: Camera, 121: Speaker, 122: Microphone array, 123: Internal server, 124: Display device, 125: Drive unit, 126: First communication IF, 127: First processor, 128: first storage device, 130: remote server, 131: second communication IF, 132: second processor, 133: second storage device, 140: first output device, 303: output device control unit, 304: drive control Section, 312: person detection section, 313: person feature extraction section, 314: person tracking section, 315: time series feature extraction section, 316: interest behavior identification section, 317: reaction confirmation section, 321: head detection section, 322: Head part fixed part, 323: Body part fixed part.

Claims (14)

An imaging device for imaging the surroundings;
Detecting a person from the image information from the imaging device, tracking the detected person in a plurality of images of the imaging device, the degree of interest of the tracked person, the face of the person in the plurality of images A dialog system comprising: a computer that calculates a change based on the change in the orientation and the orientation of the torso and sets the computer as a dialog candidate based on the calculated degree of interest. A robot having the imaging device and a speaker;
The robot has one of a display device and a driving device,
The computer transmits a control signal for causing the speaker, the display device, or the driving device to act on the interaction candidate, and a person determined to have received a response from the interaction candidate within a predetermined time. 2. The dialogue system according to claim 1, wherein the target is a dialogue target.   When a plurality of persons are captured in the image information from the imaging device, the calculator calculates the degree of interest for each person, and among the plurality of persons, identifies a person whose degree of interest is higher than a threshold as the dialog candidate. The interactive system according to claim 2, wherein: A robot having the imaging device and a display device;
The computer calculates a degree of interest for each person when a plurality of persons are imaged in the image information from the imaging device, and when a plurality of persons whose degree of interest is higher than a threshold value among the plurality of persons, The interactive system according to claim 1, wherein display control of the display device of the robot is performed. The driving device turns the robot,
The interactive system according to claim 2, wherein the imaging device images the person to be interacted with from the front by turning by the driving device.   The computer, based on an image of the person to be interacted with by the imaging device taken from the front, identifies the age and gender of the person before starting conversation with the person. 5. The interactive system according to item 5. A control method for a dialogue system having a robot and a computer,
The surroundings are imaged by an imaging device mounted on the robot, the computer detects a person from image information imaged by the imaged imaging device, tracks the detected person, and tracks the detected person. A control method for a dialogue system, comprising: calculating a degree of interest based on a change in the orientation of the person's face and body in the plurality of images, and using the calculated degree of interest as a dialogue candidate. .   A speaker, a display device, and a driving device mounted on the robot, and a control signal for performing an action is transmitted from the computer to the interaction candidate, and the computer transmits the control signal within a predetermined time to the action. 8. The control method according to claim 7, wherein a person determined to have received a response from the dialogue candidate is set as a dialogue target.   When a plurality of persons are imaged in the image information from the imaging device, the calculator calculates the degree of interest for each person, and among the plurality of persons, a person whose degree of interest is higher than a threshold is a conversation candidate. 9. The control method for a dialogue system according to claim 8, wherein:   When a plurality of persons are imaged in the image information from the imaging device, the calculator calculates the degree of interest for each person, and when there are a plurality of persons whose degree of interest is higher than a threshold from among the plurality of persons. 9. The method according to claim 8, wherein control is performed to switch a display image of the display device.   9. The control method according to claim 8, wherein the computer turns the robot to transmit a control signal for imaging the person to be interacted with from the front.   The dialog system according to claim 11, wherein the computer identifies the age and gender of the person before starting the dialog with the person, based on an image of the person to be talked from the front. Control method. In a dialogue system using a robot installed in an environment where multiple people come and go,
An imaging device for imaging the periphery of the robot,
A person detection unit that detects a person from image information from the imaging device,
A person tracking unit that tracks the same person from a plurality of images from the imaging device,
A time-series feature extraction unit that calculates the degree of interest of the person tracked by the person tracking unit with respect to the robot based on changes in the orientation of the person's face and body in the plurality of images; A dialogue system comprising: an interest behavior identification unit that determines a dialogue candidate based on the degree of interest. The robot has at least one of an output device or a driving device,
By controlling either the output device or the driving device with respect to the dialogue candidate, there is provided a reaction confirming unit for acting on the person and confirming whether or not the person has responded to the act. The interactive system according to claim 13, wherein:

Images