JP6607092B2 - Guide robot control system, program, and guide robot - Google Patents

Description

  The present invention relates to a guide robot control system, a program, and a guide robot.

  Conventionally, a robot that provides a guidance service to visitors has been disclosed (for example, Patent Document 1).

JP 2008-260107 A

  Since the robot described in Patent Document 1 provides a guidance service according to a pattern related to the movement of a user such as a visitor, there is a possibility that the guidance service is forcibly provided regardless of the user's intention. was there.

  Therefore, an object of the present invention is to provide a guidance robot control system, a program, and a guidance robot that can perform a voice call in consideration of a user's intention.

The present invention solves the above problems by the following means.
According to a first aspect of the present invention, there is provided a guide data storage unit that stores guide data for a user, an image acquisition unit that acquires a plurality of frame images continuous in time series, and an analysis of the plurality of frame images acquired by the image acquisition unit And a direction determining unit that determines a direction that the user is viewing for each frame image, and a direction change amount that represents a magnitude of a change in the direction over time based on the direction determined by the direction determining unit. Based on the direction change calculated by the direction change amount calculating means, guidance determining means for determining whether or not to guide based on the direction change calculated by the direction change amount calculating means, and the guidance determining means A guidance robot control system comprising: guidance means for extracting guidance data from the guidance data storage unit and performing guidance by a robot.
According to a second invention, in the guidance robot control system according to the first invention, a distance calculation unit that calculates a distance between the user and the robot based on the position information of the user in the frame image acquired by the image acquisition unit. The guidance determination unit is configured to determine that guidance is performed when the distance calculated by the distance calculation unit is within a specified range.
According to a third aspect of the present invention, in the guidance robot control system according to the first or second aspect, the movement for calculating the moving speed of the user based on the position information of the user in the frame image acquired by the image acquisition means. The guidance robot control system includes a speed calculation unit, and the guidance determination unit determines to perform guidance when the movement speed calculated by the movement speed calculation unit is within a predetermined range.
According to a fourth aspect of the present invention, in the guidance robot control system of the third aspect, the robot can move, and the moving direction of the user based on the position information of the user in the frame image acquired by the image acquiring means. And a movement control means for moving the robot based on the movement direction of the user calculated by the movement direction calculation means and the movement speed calculated by the movement speed calculation means. The guide robot control system characterized by the above.
According to a fifth aspect of the present invention, in the guidance robot control system according to any one of the first to third aspects of the invention, the direction determining means is based on a relative position between the user's eyeball and pupil, and is viewed by the user. A plurality of frames acquired by the image acquisition means, and a movement direction calculation means for calculating a movement direction of the user based on position information of the user in the frame image acquired by the image acquisition means. Analyzing the image and acquiring face data of the user; and face direction determining means for determining the orientation of the user's face for each of the frame images, wherein the guidance determining means includes the movement When the moving direction of the user calculated by the direction calculating unit is different from the direction of the face common to the plurality of frame images determined by the face direction determining unit, It is determined that the to a guidance robot control system according to claim.
According to a sixth aspect of the present invention, in the guidance robot control system according to the fifth aspect of the invention, the direction determining means detects a facial organ based on the face data acquired by the face acquiring means, A guidance robot control system characterized by determining a direction and acquiring a direction that the user is looking at.
According to a seventh invention, in the guided robot control system according to any one of the first invention to the sixth invention, the user's facial expression is analyzed by analyzing at least one frame image acquired by the image acquisition means. The guidance data storage unit stores guidance data corresponding to a facial expression, and the guidance unit refers to the guidance data storage unit and is determined by the facial expression determination unit. A guidance robot control system characterized by extracting guidance data based on a user's facial expression and performing guidance.
According to an eighth aspect of the present invention, in the guidance robot control system according to the seventh aspect of the present invention, after guiding by guidance data, at least the processing by the image acquisition means and the processing by the facial expression discrimination means are executed, and the user before guidance Facial expression comparing means for comparing the facial expression of the user and the facial expression of the user after guidance, the guidance means refers to the guidance data storage unit and extracts guidance data based on the comparison result by the facial expression comparison means This is a guidance robot control system characterized by performing guidance.
According to a ninth invention, in the guided robot control system according to any one of the first to eighth inventions, the attribute of the user is analyzed by analyzing at least one of the frame images acquired by the image acquisition means. Attribute determining means for determining the guide data, the guide data storage unit stores guide data corresponding to a user attribute, and the guide means is determined by the attribute determination unit with reference to the guide data storage unit A guidance robot control system characterized in that guidance is performed by extracting guidance data corresponding to the attribute of the user.
A tenth invention is a program for causing a processor of a robot to function as any of the guidance robot control systems from the first invention to the ninth invention.
An eleventh aspect of the invention is directed to a guide data storage unit that stores guide data for a user, a photographing unit, an image acquisition unit that acquires a plurality of frame images continuous in time series, and the plurality of frames acquired by the image acquisition unit. Based on the direction discriminating means for analyzing the frame image and discriminating the direction in which the user is looking for each frame image, and the magnitude of change in the direction with time based on the direction discriminated by the direction discriminating means. Direction change amount calculating means for calculating a direction change amount to be expressed, guidance determination means for determining whether or not to guide based on a change in direction calculated by the direction change amount calculation means, and guidance by the guidance determination means When it is determined to do, the guidance robot is provided with guidance means for extracting guidance data from the guidance data storage unit and performing guidance.

  ADVANTAGE OF THE INVENTION According to this invention, the guidance robot control system, program, and guidance robot which can call out in consideration of a user's intention can be provided.

It is a figure which shows the example of a scene suitable for operation | use of the guidance robot control system which concerns on 1st Embodiment. It is a functional block diagram of the communication robot which concerns on 1st Embodiment. It is a figure which shows the example of the guidance data storage part of the communication robot which concerns on 1st Embodiment. It is a flowchart which shows the robot control process in the communication robot which concerns on 1st Embodiment. It is a flowchart which shows the image analysis process in the communication robot which concerns on 1st Embodiment. It is a flowchart which shows the user determination process in the communication robot which concerns on 1st Embodiment. It is a flowchart which shows the guidance post-process in the communication robot which concerns on 1st Embodiment. It is a figure which shows the specific example of the guidance robot control system which concerns on 1st Embodiment. It is a figure which shows the example of a scene suitable for operation | use of the guidance robot control system which concerns on 2nd Embodiment. It is a functional block diagram of the guidance robot control system which concerns on 2nd Embodiment. It is a figure which shows the example of the guidance data storage part of the communication robot which concerns on 2nd Embodiment. It is a flowchart which shows the robot control process in the communication robot which concerns on 2nd Embodiment. It is a flowchart which shows the image analysis process in the communication robot which concerns on 2nd Embodiment. It is a flowchart which shows the user determination process in the communication robot which concerns on 2nd Embodiment. It is a figure which shows the specific example of the guidance robot control system which concerns on 2nd Embodiment. It is a figure which shows the specific example of the guidance robot control system which concerns on a deformation | transformation form.

DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. This is merely an example, and the technical scope of the present invention is not limited to this.
(First embodiment)
<Guiding robot control system 100>
FIG. 1 is a diagram illustrating an example of a scene suitable for operation of the guidance robot control system 100 according to the first embodiment.
The guide robot control system 100 is a system using the communication robot 1 (guide robot). The communication robot 1 is disposed in a space S such as an exhibition hall or a store, for example. The communication robot 1 is preferably arranged at a position where the space S can be looked over.

The communication robot 1 is, for example, a humanoid robot having a height of about 1 m.
The communication robot 1 acquires and analyzes a frame image including the user P in the space S (users P1 to P3 in the example of FIG. 1), determines the user P to be called, Make a voice call (guidance). In the example described below, the communication robot 1 estimates that the user P who is currently in trouble is in trouble, and determines that it is a subject to call.

<Communication robot 1>
FIG. 2 is a functional block diagram of the communication robot 1 according to the first embodiment.
FIG. 3 is a diagram illustrating an example of the guidance data storage unit 22 of the communication robot 1 according to the first embodiment.
As shown in FIG. 2, the communication robot 1 includes a control unit 10, a storage unit 20, a camera 31 (imaging unit), a speaker 32, and a motor unit 33.
The control unit 10 is a central processing unit (CPU) that controls the entire communication robot 1. The controller 10 executes various functions in cooperation with the hardware described above by appropriately reading and executing an operating system (OS) and application programs stored in the storage unit 20.

The control unit 10 includes an image acquisition unit 11 (image acquisition unit), an image analysis unit 12, a user determination processing unit 13, a guidance data extraction unit 15, a movement control unit 16 (movement control unit), and a guidance unit 17. (Guide means) and a facial expression comparison unit 18 (facial expression comparison means).
The image acquisition unit 11 acquires a plurality of frame images via the camera 31. The acquired plurality of frame images are continuous in time series.
The image analysis unit 12 analyzes the frame image acquired by the image acquisition unit 11. The image analyzing unit 12 includes a direction determining unit 12a (direction determining unit), a distance calculating unit 12b (distance calculating unit), an expression determining unit 12c (expression determining unit), and an attribute determining unit 12d (attribute determining unit). Prepare.
The direction determination unit 12a determines, for each user P included in the frame image, the direction in which the user P is viewing.
The distance calculation unit 12b calculates the distance between the communication robot 1 and the user P for each user P included in the frame image.
The facial expression determination unit 12c determines the facial expression of the user P for each user P included in the frame image.
The attribute determination unit 12d determines the attribute of the user P for each user P included in the frame image. Here, the attribute of the user P refers to, for example, age, sex, and the like.

The user determination processing unit 13 determines a user P to speak on the basis of the frame image analyzed by the image analysis unit 12. The user determination processing unit 13 includes a direction change amount calculation unit 13a (direction change amount calculation unit), a movement speed calculation unit 13b (movement speed calculation unit), a movement direction calculation unit 13c (movement direction calculation unit), and guidance determination. 13d (guidance judgment means).
The direction change amount calculation unit 13a calculates a direction change amount that is the magnitude of a change with time in the direction that the user P is viewing determined by the direction determination unit 12a.
The movement speed calculation unit 13b calculates the movement speed based on the change of the distance of the user P calculated by the distance calculation unit 12b with time.
The movement direction calculation unit 13c calculates the movement direction based on the change of the distance of the user P calculated by the distance calculation unit 12b over time.
The guidance determination unit 13d determines the user P to call out based on the direction change amount and the like.

The guidance data extraction unit 15 extracts guidance data for guiding the determined user P from the guidance data storage unit 22.
The movement control unit 16 moves the communication robot 1 to approach the determined user P based on the movement direction calculated by the movement direction calculation unit 13c.
The guide unit 17 outputs the guide data extracted by the guide data extraction unit 15 via the speaker 32.
The facial expression comparison unit 18 compares facial expressions determined by the facial expression determination unit 12c before and after guidance.
Details of each process will be described later.

The storage unit 20 is a storage area such as a semiconductor memory element for storing programs, data, and the like necessary for the control unit 10 to execute various processes.
The storage unit 20 includes a program storage unit 21, a guidance data storage unit 22, and an image data storage unit 23.
The program storage unit 21 is a storage area for storing a program. The program storage unit 21 stores a control program 21a (program) for executing the various functions of the control unit 10 described above.
The guide data storage unit 22 is a storage area for storing guide data issued by the communication robot 1. As shown in FIG. 3, the guidance data storage unit 22 stores guidance data for calling in association with facial expressions and attributes of the user P. Further, the next guidance data is stored in association with the facial expression of the user P after calling.
Returning to FIG. 2, the image data storage unit 23 is a storage area for storing the frame image acquired by the image acquisition unit 11.

The camera 31 is a photographing device. The camera 31 is provided, for example, at a position corresponding to the eye part of the communication robot 1. Then, the camera 31 captures a subject in the direction that the communication robot 1 faces.
The speaker 32 is an audio output device. The speaker 32 is provided, for example, at a position corresponding to the mouth portion of the communication robot 1. The speaker 32 outputs sound based on an instruction from the control unit 10.

The motor unit 33 is a motor for causing the communication robot 1 to operate. The motor unit 33 includes a head motor 33a and a foot motor 33b.
The head motor 33 a is a motor that moves the head of the communication robot 1. The head motor 33a is controlled by the control unit 10 and moves the head up, down, left, and right in order to photograph various users P in the space S as subjects.
The foot motor 33 b is a motor for moving the communication robot 1 on the foot of the communication robot 1.

<Processing of communication robot 1>
Next, processing of the communication robot 1 will be described.
FIG. 4 is a flowchart showing a robot control process in the communication robot 1 according to the first embodiment.
FIG. 5 is a flowchart showing image analysis processing in the communication robot 1 according to the first embodiment.
FIG. 6 is a flowchart showing a user determination process in the communication robot 1 according to the first embodiment.
FIG. 7 is a flowchart showing post-guidance processing in the communication robot 1 according to the first embodiment.

In step S (hereinafter referred to as “S”) 10 in FIG. 4, the control unit 10 (image acquisition unit 11) of the communication robot 1 performs an image acquisition process. In the image acquisition process, the control unit 10 always acquires a frame image via the camera 31, and determines whether or not the acquired frame image includes one or more users P. When it is determined that one or more users P are included in the frame image, the control unit 10 stores the frame image acquired at an appropriate timing (for example, every 0.1 second) in the image data storage unit 23. Remember.
In S11, the control unit 10 determines whether or not a processing condition has been met. The processing condition refers to, for example, a case where a plurality of frame images (for example, 30 images) are stored in the image data storage unit 23, or a case where a predetermined time (for example, 3 seconds) has elapsed. If it is determined that the processing condition is satisfied (S11: YES), the control unit 10 moves the process to S12. On the other hand, when it is determined that the processing condition is not satisfied (S11: NO), the control unit 10 moves the process to S10.

In S <b> 12, the control unit 10 (image analysis unit 12) performs an image analysis process for analyzing the frame image stored in the image data storage unit 23.
Here, the image analysis processing will be described with reference to FIG.
In S <b> 30 of FIG. 5, the control unit 10 extracts one frame image from the image data storage unit 23.
In S31, the control unit 10 (direction determination unit 12a) determines the direction in which the user P is viewing for each user P included in the frame image. For example, when two users P1 and P2 are included in the frame image, the control unit 10 determines the direction that the user P1 is viewing and the direction that the user P2 is viewing. For example, the control unit 10 can detect a face image from the frame image and determine the orientation of the face based on the arrangement position of each part (organ) of the face. Therefore, the control unit 10 can change the orientation of the determined face to the direction in which the user P is looking.
There are various known methods for obtaining the position of each organ (face, nose, mouth, etc.) of the face. For example, a facial organ detection method using a regression model can be used. A facial organ detection method using a regression model is described in, for example, US Patent Application Publication No. 2014/0185924.

In S32, the control unit 10 (distance calculation unit 12b) calculates the distance between the user P and the communication robot 1 for each user P included in the frame image. For example, the control unit 10 can calculate a rough distance between the user P and the communication robot 1 based on the size of the user P included in the frame image, the background of the frame image, and the like.
In S33, the control unit 10 (expression determination unit 12c) determines the facial expression for each user P included in the frame image. As the facial expression, the control unit 10 can determine, for example, whether it is a smile (positive expression) or a sad face (negative expression) from the shape of eyes, nose, mouth, and the like.

In S <b> 34, the control unit 10 determines whether or not all of the processing target frame images included in the image data storage unit 23 have been processed. When processing has been performed on all the frame images to be processed (S34: YES), the control unit 10 ends this processing and shifts the processing to FIG. On the other hand, when the processing has not been performed for all the frame images to be processed (S34: NO), the control unit 10 moves the process to S35.
In S <b> 35, the control unit 10 extracts one unprocessed frame image from the processing target frame images included in the image data storage unit 23. Thereafter, the control unit 10 moves the process to S31.

Returning to FIG. 4, in S <b> 13, the control unit 10 (user determination processing unit 13) performs user determination processing.
Here, the user determination process will be described with reference to FIG.
In S40 of FIG. 6, the control unit 10 arranges the frame images subjected to the image analysis process in time series.
In S41, the control unit 10 (direction change amount calculation unit 13a) calculates the direction change amount for each user P based on the plurality of rearranged frame images. Here, the amount of change in direction refers to the amount of change in the direction in which the user P is looking as time passes. Specifically, when the user P is drenching violently, the degree of choking is high, and the control unit 10 calculates the direction change amount to be high.

In S42, the control unit 10 (movement speed calculation unit 13b) calculates the movement speed for each user P based on the plurality of rearranged frame images. The control unit 10 can calculate the moving speed based on the change in the position of the user P in the plurality of frame images. Note that the control unit 10 may calculate whether the moving speed is faster than a specified range.
In S43, the control unit 10 (movement direction calculation unit 13c) calculates the movement direction for each user P based on the rearranged frame images. The control unit 10 can calculate the moving direction based on the change in the position of the user P in the plurality of frame images.
In S <b> 44, the control unit 10 (user determination processing unit 13) determines the facial expression that appears most frequently for each user P based on the rearranged frame images.

In S45, the control unit 10 (guidance determination unit 13d) calculates a voice call priority based on each index.
Here, the voice call priority is a degree calculated by weighting each of the direction change amount, the position of the user P, the moving speed, and the facial expression. In this example, since the user P who is in trouble is given priority, the degree of weighting is set highest for the direction change amount. Then, when the direction change amount is large, the moving speed is small, and the facial expression is negative, the control unit 10 performs weighting so that the calling priority is the highest. In addition, when the distance between the user P and the communication robot 1 is within a specified range (for example, 10 m), the control unit 10 sets the user P as a calling target.

In S46, the control unit 10 (guidance determination unit 13d) determines the user P to call out based on the calling priority.
Note that when the user P has a small direction change amount, the moving speed is slow, and the priority is a small value even if the facial expression is negative. In such a case, that is, if the priority is less than the predetermined value, the control unit 10 may determine that there is no user P to call.
Then, the control part 10 complete | finishes this process, and moves a process to FIG.

Returning to FIG. 4, in S <b> 14, the control unit 10 (user determination processing unit 13) determines whether or not the user P is determined by the user determination processing. When the user P is determined (S14: YES), the control unit 10 moves the process to S15. On the other hand, when the user P is not determined (S14: NO), the control unit 10 ends this process.
In S15, the control unit 10 (attribute determination unit 12d) determines the attribute of the determined user P. Specifically, the control unit 10 determines an approximate age or the like from the image of the user P extracted from the frame image. Control part 10 may discriminate whether it is a child or an adult, without distinguishing age. In that case, the control unit 10 may estimate the height from the image of the user P and determine whether or not the child is a child.

In S16, the control unit 10 (guidance data extraction unit 15) refers to the guidance data storage unit 22 (FIG. 3) based on the facial expression of the user P determined in S44 of FIG. 6 and the attribute determined in S15. Extract data.
In S17, the control unit 10 (movement control unit 16) performs a movement process. As the moving process, the control unit 10 controls the motor unit 33 (foot motor 33b) based on the moving speed and moving direction of the user P calculated in S42 and S43 in FIG. The robot 1 is moved.
In S18, the control unit 10 (guide unit 17) performs a guide process. As the guidance process, the control unit 10 outputs the guidance data extracted in S <b> 16 via the speaker 32 so as to talk (speak) to the user P after moving.

In S19, the control unit 10 performs post-guidance processing.
Here, the post-guidance process will be described with reference to FIG.
In S50 of FIG. 7, the control unit 10 (image acquisition unit 11) acquires a frame image including the user P after outputting the guidance data via the camera 31.
In S51, the control unit 10 (expression determination unit 12c) determines the facial expression by analyzing the acquired frame image.
In S52, the control unit 10 (facial expression comparison unit 18) performs facial expression comparison processing. As the facial expression comparison process, the control unit 10 compares the facial expression before the guidance data is output (the process in S44 of FIG. 6) with the facial expression acquired in S51.

In S53, the control unit 10 (guidance data extraction unit 15) extracts the guidance data corresponding to the comparison result of facial expressions from the guidance data storage unit 22 (FIG. 3).
In S <b> 54, the control unit 10 (guidance unit 17) outputs the guidance data extracted in S <b> 53 via the speaker 32 as a guidance process. Thereafter, the control unit 10 shifts the processing to FIG.
Returning to FIG. 4, the control unit 10 ends this process.

<Description of specific examples>
Here, a specific example using the guidance robot control system 100 will be described.
FIG. 8 is a diagram illustrating a specific example of the guidance robot control system 100 according to the first embodiment.
First, the control unit 10 of the communication robot 1 obtains a frame image of the child C1 who is lost and crawls as shown in FIG. 8A (S10 in FIG. 4). It is determined that there is (S14 in FIG. 4 is YES).
Next, the control unit 10 extracts guidance data at the time of calling from the guidance data storage unit 22 (see FIG. 3) when the attribute is a child and the facial expression is negative (S16 in FIG. 4).
Then, as shown in FIGS. 8A and 8B, the control unit 10 moves toward the position of the child C1 (S17 in FIG. 4), and outputs the extracted guidance data via the speaker 32. (S18 in FIG. 4).

Thereby, the communication robot 1 can guess the state of the child C1 based on the movement and expression of the child C1, and can make a voice call that matches the intention of the child C1.
Thereafter, the control unit 10 acquires a frame image of the child C1. If the facial expression of the child C1 remains negative as shown in FIG. 8C, the control unit 10 reads the voice data from the guidance data storage unit 22. Guidance data of the corresponding facial expression after being hung is extracted and output through the speaker 32 (S19 in FIG. 4).

Thus, according to the guidance robot control system 100 of the first embodiment, the following effects are obtained.
(1) The amount of change in the direction in which the user P is looking is calculated from a plurality of frame images. Since the direction change amount becomes a large value when the user P is joking, the communication robot 1 determines that the user P is in trouble and calls out. Therefore, the state of the user P as to whether or not the user P is in trouble is inferred based on the magnitude of the amount of change in the direction in which the user P is looking. It is possible to preferentially make a voice call that suits the intention.

(2) When the position of the user P and the position of the communication robot 1 are within the specified range, the communication robot 1 speaks to the user P. Therefore, the communication robot 1 can preferentially talk to the user P nearby.
(3) When the moving speed of the user P is within the specified range, the communication robot 1 calls out to the user P. Therefore, when the user P is walking slowly or stops, the communication robot 1 can speak with priority.
(4) When the movement position of the user P is estimated and the communication robot 1 approaches, the conversation by the communication robot 1 with respect to the user P can be performed more naturally.

(5) The guidance robot is selected based on the facial expression of the user P, and the communication robot 1 speaks to the user P. Therefore, the communication robot 1 can talk by guiding the content that matches the facial expression of the user P.
(6) Based on the change in the facial expression of the user P before and after calling, the communication robot 1 further provides the next guidance. Therefore, the communication robot 1 can talk about the content with which communication was more possible.

(7) The attribute such as the age of the user P is discriminated from the frame image, the guidance data corresponding to the attribute of the user P is selected, and the communication robot 1 speaks to the user P. Therefore, it is possible to talk by guiding the content that matches the attribute of the user P.
(8) The communication robot 1 can perform all processing from acquisition of frame images to output of guidance data. Therefore, only the communication robot 1 needs to be prepared, and installation can be performed easily.

(Second Embodiment)
In the second embodiment, a description will be given of what causes a communication robot to call a user. In the following description, parts that perform the same functions as those in the first embodiment described above are given the same reference numerals or the same reference numerals at the end, and redundant descriptions are omitted as appropriate.

<Guiding robot control system 200>
FIG. 9 is a diagram illustrating an example of a scene suitable for operation of the guidance robot control system 200 according to the second embodiment.
FIG. 10 is a functional block diagram of the guidance robot control system 200 according to the second embodiment.
FIG. 11 is a diagram illustrating an example of the guide data storage unit 222 of the communication robot 201 according to the second embodiment.

The guidance robot control system 200 arranges the communication robot 201 at the entrance of a space 200S such as an exhibition booth or a store entrance. The guidance robot control system 200 is a system used by the communication robot 201 to call the user P (in this example, the user P4) to enter the space 200S.
The guidance robot control system 200 includes a communication robot 201 and a camera 205. The communication robot 201 and the camera 205 are installed near the entrance of the space 200S.

In this example, the camera 205 is a surveillance camera. The camera 205 acquires a frame image of the user P passing through the space 200S and transmits the frame image to the communication robot 201.
The communication robot 201 analyzes the frame image of the user P received from the camera 205, determines the user P to be called, and provides guidance to the user P. The communication robot 201 makes a call to the user P passing in front of the camera 205 by determining whether he / she is interested in the space 200S.

<Communication robot 201>
As shown in FIG. 10, the communication robot 201 includes a control unit 210, a storage unit 220, a speaker 32, and a motor unit 33.
The control unit 210 includes an image reception unit 211 (image acquisition unit), an image analysis unit 212, a user determination processing unit 213, a face data verification unit (face verification unit) 214, a guidance data extraction unit 215, and movement control. A unit 16, a guide unit 17, and a facial expression comparison unit 18 are provided.
The image receiving unit 211 receives a frame image by receiving the frame image transmitted by the camera 205.

The image analysis unit 212 analyzes the frame image received by the image reception unit 211. The image analysis unit 212 includes a direction acquisition unit 212a (user direction acquisition unit), a distance calculation unit 12b, a facial expression determination unit 212c, and a face acquisition unit 212e (face acquisition unit).
The direction acquisition unit 212a analyzes the frame image to identify the user P, determines the direction that the user P is viewing for each user P, and acquires the direction that the user P is viewing.
The facial expression determination unit 212c determines the facial expression of the user P based on the face data of the user P acquired by the face acquisition unit 212e.
The face acquisition unit 212e acquires the face data of the user P for each user P included in the frame image.

Based on the frame image analyzed by the image analysis unit 212, the user determination processing unit 213 determines a user P who is to speak to by a change with time. The user determination processing unit 213 includes a movement speed calculation unit 13b, a movement direction calculation unit 13c, and a guidance determination unit 213d.
The guidance determination unit 213d determines the user P to call on, based on the viewing direction of the user P acquired by the direction acquisition unit 212a, the movement direction calculated by the movement direction calculation unit 13c, and the like.
The face data collation unit 214 collates the face data stored in the face data storage unit 224 and the face data acquired by the face acquisition unit 212e.
The guidance data extraction unit 215 extracts guidance data for guiding the user P determined by the user determination processing unit 213 from the guidance data storage unit 222. At that time, the guidance data extraction unit 215 extracts guidance data different from the previous one when the face data matching unit 214 can collate.

The storage unit 220 includes a program storage unit 21, a guidance data storage unit 222, an image data storage unit 23, and a face data storage unit 224.
The program storage unit 21 stores a control program 221a for executing various functions of the control unit 210 described above.
As shown in FIG. 11, the guidance data storage unit 222 stores guidance data for calling in association with facial expressions of the user P. Further, the next guidance data is stored in association with the facial expression of the user P after calling.
The face data storage unit 224 is a storage area for storing face data. When the guide data is output, the face data storage unit 224 stores the face data acquired by the face acquisition unit 212e in association with the output guide data.

<Camera 205>
The camera 205 is a live camera represented by a web camera, for example. The camera 205 may be a stereo camera. A stereo camera refers to a camera that can capture information about the depth of an object by simultaneously capturing images from slightly shifted positions with a plurality of lenses and generating parallax.
The camera 205 includes an image acquisition unit 251 and an image transmission unit 252.
The image acquisition unit 251 is a control unit that acquires a frame image.
The image transmission unit 252 is a control unit that transmits the acquired frame image via the communication network N.
The communication robot 201 and the camera 205 are communicably connected via a communication network N. The communication network N is, for example, an Internet line or a mobile terminal communication network. Further, the communication network N may use, for example, short-range wireless communication based on the Bluetooth (registered trademark) standard.

<Processing of communication robot 201>
Next, processing of the communication robot 201 will be described.
FIG. 12 is a flowchart showing robot control processing in the communication robot 201 according to the second embodiment.
FIG. 13 is a flowchart showing image analysis processing in the communication robot 201 according to the second embodiment.
FIG. 14 is a flowchart illustrating user determination processing in the communication robot 201 according to the second embodiment.

In S210 of FIG. 12, the control unit 210 (image reception unit 211) of the communication robot 201 receives the frame image transmitted by the camera 205. Then, the control unit 210 stores the received frame image in the image data storage unit 23.
In S211, the control unit 210 (image analysis unit 212) performs an image analysis process for analyzing the frame image stored in the image data storage unit 23.
Here, the image analysis processing will be described with reference to FIG.

In S230 of FIG. 13, the control unit 210 (direction acquisition unit 212a) detects the user P included in the frame image, and acquires the direction in which the user P is looking. For example, when two users P1 and P2 are included in the frame image, the control unit 210 detects the users P1 and P2, and acquires the direction in which each user P is looking.
There are various known methods for detecting the user P. For example, a moving object detection method can be used. In the moving object detection method, a frame image in which the user P is not captured is stored as background image data in advance, and a difference between the background image data and the frame image to be processed is obtained, so that the user P who is a moving object. Is detected. Further, human detection using the HOG feature amount can also be used. Human detection using HOG features is described in “N. Dalal and B. Triggs. Histograms of Oriented Gradients for Human Detection. In CVPR, pages 886-893, 2005”.

Further, as the direction in which the user P is viewing, the control unit 210 acquires, for example, the direction of the face of the user P. The face direction of the user P is assumed to be the front direction with respect to the camera 205 when the face can be detected by the control unit 210 (face acquisition unit 212e) using a face detection method. There are various known methods for detecting a face. For example, face detection using a Haar-like feature can be used. For face detection using Haar-like features, see “P. Viola and M. J. Jones:“ Rapid Object Detection Using the Coordinates of the Cofesto et Ecée et al. ”Proceedings of the Effort. , Pp. 511-518, (2001).
Note that the control unit 210 may detect a face image from the frame image, extract an eye from the face image, and acquire the direction in which the user P is looking based on the relative position between the eyeball and the pupil.

  The processing of S231 and S232 is the same as the processing of S32 and S33 of the first embodiment (FIG. 5). In S232, the control unit 210 (face acquisition unit 212e) acquires face data, and the control unit 210 (expression determination unit 212c) determines the expression based on the acquired face data. Thereafter, the control unit 210 shifts the processing to FIG.

Returning to FIG. 12, in S212, the control unit 210 (user determination processing unit 213) performs user determination processing.
Here, the user determination process will be described with reference to FIG.
In S240 of FIG. 14, the control unit 210 arranges the frame images subjected to the image analysis process in time series.
In S241, the control unit 210 (direction acquisition unit 212a) acquires the viewing direction for each user P based on the plurality of rearranged frame images. Here, in a plurality of frame images, when the direction in which the user P is viewing is the same direction, the control unit 210 acquires the direction as the viewing direction. Therefore, for example, when the user P is jogging, the control unit 210 does not acquire the viewing direction.

  The processing from S242 to S244 is the same as the processing from S42 to S44 in the first embodiment (FIG. 6). In the calculation of the moving direction for each user P, for example, human body tracking using the Mean-shift feature can be used. Regarding human body tracking using the Mean-shift feature, see “Collins R .: 'Mean-Shift Blob Tracking through Scale Scale,” Proc. IEEE Conf. ."It is described in.

In S245, the control unit 210 (guidance determination unit 213d) calculates the calling priority based on each index.
Here, the voice call priority is a degree calculated by weighting each of the direction that the user P is viewing, the moving direction, the position of the user P, and the moving speed. In this example, the user 200 who is interested in the space 200S is preferentially spoken to the user P who is about to pass, so the degree of weighting is different in the direction in which the user P is viewing and the direction of movement. Yes, when the user P is facing the front direction (the direction of the camera 205), the highest is set. Then, when the above condition is satisfied and the moving speed is slow, the control unit 210 performs weighting so that the calling priority is the highest. In addition, when the distance between the user P and the communication robot 201 is within a specified range (for example, 5 m), the control unit 210 sets the user P as a calling target.
In S246, the control unit 210 (guidance determination unit 213d) determines the user P to call on the basis of the calling priority.

If the user P is not facing the front direction, the direction of movement of the user P is different from the direction of movement, and the distance between the user P and the communication robot 201 is within the specified range. Alternatively, the control unit 210 may determine that there is no user P to call out.
Thereafter, the control unit 210 ends this processing and shifts the processing to FIG.

Returning to FIG. 12, in S213, the control unit 210 (user determination processing unit 213) determines whether or not the user P is determined by the user determination processing. When the user P is determined (S213: YES), the control unit 210 moves the process to S214. On the other hand, when the user P is not determined (S213: NO), the control unit 210 ends this process.
In S214, the control unit 210 (face data collation unit 214) collates the face data acquired in S232 of FIG. 13 with the face data in the face data storage unit 224.
In S215, the control unit 210 (face data collation unit 214) determines whether or not collation has been completed. If the verification is successful (S215: YES), the control unit 210 moves the process to S216. On the other hand, when collation cannot be performed (S215: NO), the control unit 210 moves the process to S216a. In addition, the case where it can collate is a case where the user P was called in the past.

In S <b> 216, the control unit 210 refers to the face data storage unit 224 and the guidance data storage unit 222. The control data is guidance data that is different from the output guidance data and that corresponds to the facial expression of the user P. Extracted from the storage unit 222.
On the other hand, in S216a, the control unit 210 refers to the guidance data storage unit 222 and extracts guidance data corresponding to the facial expression of the user P.
The processing of S217 and S218 is the same as the processing of S17 and S18 of the first embodiment (FIG. 4).
In S <b> 219, the control unit 210 associates the face data of the user P with the output guidance data and causes the face data storage unit 224 to store them.
In S220, control unit 210 performs post-guidance processing. The post-guidance process is the same as in the first embodiment (FIG. 7). Thereafter, the control unit 210 ends this process.

<Description of specific examples>
Next, a specific example using this guidance robot control system 200 will be described.
FIG. 15 is a diagram illustrating a specific example of the guidance robot control system 200 according to the second embodiment.
The communication robot 201 and the camera 205 are installed near the entrance of the space 200S. The image acquisition unit 251 of the camera 205 acquires frame images as needed, and the image transmission unit 252 transmits the acquired frame images to the communication robot 201 as needed.

As shown in FIG. 15 (A), the controller 210 walks in front of the space 200S and targets the user P4 whose face is facing the communication robot 201 (S213 in FIG. YES).
Next, the control unit 210 determines whether or not the acquired face data is stored in the face data storage unit 224. If the control unit 210 determines that the acquired face data is not stored, the control unit 210 determines from the guidance data storage unit 222 (see FIG. 11). Then, guidance data at the time of calling corresponding to the facial expression is extracted (S216a in FIG. 12). Then, the communication robot 201 outputs the extracted guidance data via the speaker 32 (S218 in FIG. 12). In this example, the communication robot 201 calls the user P4 “Please drop in at the store”.
Then, when the user P4 stops and looks at the communication robot 201, he further calls (S220 in FIG. 12).

After a while, when the same user P4 is walking in front of the space 200S and the face is facing the communication robot 201 as shown in FIG. 15B, the control unit 210 moves the user P4 to the user P4. , A voice call target (S213 in FIG. 12 is YES).
Next, the control unit 210 determines whether or not the acquired face data is stored in the face data storage unit 224. If the control unit 210 determines that the user P4 has previously guided, the control data storage unit 222 (FIG. 11), the guidance data at the time of calling different from the previous one corresponding to the facial expression is extracted (S216 in FIG. 12). Then, the communication robot 201 outputs the extracted guidance data via the speaker 32 (S218 in FIG. 12). In this example, the communication robot 201 calls out to the user P4, “You have already passed?”.

Thus, according to the second embodiment, the guidance robot control system 200 has the following effects.
(1) The communication robot 201 determines whether or not to speak to the user P based on the relationship between the direction in which the user P is viewing and the moving direction of the user P. Therefore, for example, the user P who is looking in the direction of the space 200S while passing in front of the space 200S can be inferred that he / she is interested in the space 200S and can speak to the user P.
(2) Since the communication robot 201 can acquire the direction in which the user P is looking based on whether or not the user P's face has been detected, the processing can be easily performed.

(3) Since the communication robot 201 stores the face data of the user P who has been guided, when the next guidance is given to the same user P, different guidance data from the previous one is provided. Can be output. Therefore, since the guidance can be changed by using the face authentication function, different voice calls can be made.
(4) Since the communication robot 201 performs processing using the frame image acquired by the camera 205, even if the communication robot 201 moves, the camera 205 is fixed and the position for acquiring the frame image does not change. For example, it is possible to easily process a plurality of frame images arranged in time series.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to embodiment mentioned above. In addition, the effects described in the embodiments are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments. In addition, although embodiment mentioned above and the deformation | transformation form mentioned later can also be used combining suitably, detailed description is abbreviate | omitted.

(Deformation)
(1) In the first embodiment, the communication robot includes a camera. In the second embodiment, the communication robot receives a frame image from the camera. However, the present invention is not limited to this.
For example, in the first embodiment, the communication robot may receive a frame image from a camera. In that case, the position of the camera is not limited to the vicinity of the communication robot, and may be installed on the ceiling of the space. In addition, frame images may be acquired from a plurality of cameras.
In the second embodiment, a communication robot including a camera may be used. In that case, in order to easily perform image processing, it is desirable that the communication robot does not move and does not change the acquisition position of the frame image.

  (2) In each embodiment, the communication robot performs various processes as an example, but the present invention is not limited to this. A server that is communicably connected to the communication robot may be provided, and the server may perform various processes. In that case, the communication robot only needs to include at least a speaker that outputs guidance data transmitted from the server and a motor unit for operating the communication robot. Various processes may be performed by both the communication robot and the server.

(3) In each embodiment, the example using the frame image acquired at an appropriate timing has been described as an example, but the present invention is not limited to this. For example, moving image data may be used. In that case, a plurality of frame images may be acquired from the moving image data, and image analysis processing may be performed for each frame image.
(4) In each embodiment, the example in which image analysis processing is performed on all users included in the acquired frame image has been described as an example, but the present invention is not limited to this. One user may be selected from the acquired frame images, and an image of the selected user may be analyzed to determine whether or not to make a call target.
(5) In each embodiment, the guide data is output and then the user's facial expressions before and after the guide data is output are compared to output the guide data. However, the present invention is not limited to this. For example, the guidance data may continue to be output until the user leaves the communication robot.

  (6) In the first embodiment, the direction in which the user is viewing is set as the face direction, and when the neck is moved, the direction change amount is increased as if it is staggered. However, the present invention is not limited to this. Not. For example, the direction that the user is viewing may be the direction of the line of sight. For example, the control unit can acquire the orientation of the line of sight based on the relative position between the eyeball and the pupil by detecting a face image from the frame image and extracting the eye from the face image. In that case, when the user's moving direction is different from the face direction, and the user's viewing direction is different, the user can make a voice call.

(7) In the second embodiment, an example using a Web camera or a stereo camera has been described, but the present invention is not limited to this. For example, a camera having a sensor that acquires depth information represented by Kinect (registered trademark) may be used. By using this kinect, the process of detecting the user's face from the frame image and calculating the moving direction of the user can be performed more easily.
(8) In the second embodiment, the processing related to the user attribute is not performed. However, as in the first embodiment, the user attribute is determined, and the guidance data corresponding to the determined user attribute is extracted. It may be.

(9) In the second embodiment, although it has been described that the communication robot outputs the guidance data different from the previous when it is determined that the communication robot is the same as the user who spoke before, the present invention is not limited to this. For example, as shown in FIG. 16, different guidance data may be output according to the moving direction of the user.
FIG. 16 is a diagram illustrating a specific example of a guidance robot control system 300 according to a modification.
FIG. 16A shows a case where the user P5 who is walking toward the space 300S and whose face is facing the communication robot 301 (camera 305) is the target of voice call. In this case, the communication robot 301 outputs the guidance data corresponding to the moving direction, “Welcome. Would you like to guide you?” And speaks to the user P5.

Next, FIG. 16B shows a case where the user P6 who comes out of the space 300S and whose face is facing the communication robot 301 is targeted for speaking. In this case, the communication robot 301 outputs “Thank you” which is guidance data corresponding to the moving direction, and speaks to the user P6.
Thus, the communication robot 301 extracts and outputs guidance data corresponding to the moving direction of the user P. Therefore, the communication robot 301 can make different calls to the user P5 entering the space 300S and the user P6 coming out of the space 300S, and different appropriate calls according to the situation of the user P. The communication robot 301 can be used.

1,201,301 Communication robot 10,210 Control unit 11 Image acquisition unit 12,212 Image analysis unit 12a Direction determination unit 12b Distance calculation unit 12c, 212c Expression determination unit 12d Attribute determination unit 13,213 User determination processing unit 13a Direction change Quantity calculation unit 13b Movement speed calculation unit 13c Movement direction calculation unit 13d, 213d Guidance determination unit 16 Movement control unit 17 Guide unit 18 Expression comparison unit 20,220 Storage unit 21a, 221a Control program 22,222 Guide data storage unit 31,205 , 305 Camera 32 Speaker 33 Motor unit 100, 200, 300 Guide robot control system 211 Image receiving unit 212a Direction acquisition unit 212e Face acquisition unit 214 Face data collation unit 224 Face data storage unit 251 Image acquisition unit 252 Image transmission unit C1 child P, P1 to P6 User S, 200S, 300S Space

Claims (10)

A guidance data storage unit for storing guidance data for the user;
Image acquisition means for acquiring a plurality of frame images continuous in time series;
Analyzing the plurality of frame images acquired by the image acquisition unit, and for each frame image , based on a relative position between the user's eyeball and pupil, a direction determination unit that determines a direction in which the user is viewing;
A moving direction calculating means for calculating the moving direction of the user based on the position information of the user in the frame image acquired by the image acquiring means;
Analyzing the plurality of frame images acquired by the image acquisition means, and acquiring face data of the user;
Face direction determining means for determining the orientation of the user's face for each frame image;
Direction change amount calculating means for calculating a direction change amount representing the magnitude of change in the direction with time based on the direction determined by the direction determining means;
Guidance determining means for determining whether or not to guide based on a change in direction calculated by the direction change amount calculating means;
Guidance means for extracting guidance data from the guidance data storage unit and performing guidance by a robot when the guidance judgment means judges to guide;
Equipped with a,
The guidance determination unit is further configured in a case where the movement direction of the user calculated by the movement direction calculation unit is different from the direction of the face common to the plurality of frame images determined by the face direction determination unit. To decide to give guidance,
Guide robot control systems that characterized the. In the guidance robot control system according to claim 1 ,
The direction determining means detects facial organs based on the face data acquired by the face acquiring means, determines the orientation of the face from their positional relationship, and acquires the direction that the user is viewing;
Guide robot control system characterized by In the guidance robot control system according to claim 1 or 2 ,
A distance calculating unit that calculates a distance between the user and the robot based on the position information of the user in the frame image acquired by the image acquiring unit;
The guidance determination means determines to provide guidance when the distance calculated by the distance calculation means is within a specified range;
Guide robot control system characterized by In the guidance robot control system according to any one of claims 1 to 3 ,
A moving speed calculating means for calculating the moving speed of the user based on the position information of the user in the frame image acquired by the image acquiring means;
The guidance determination means determines to provide guidance when the movement speed calculated by the movement speed calculation means is within a specified range;
Guide robot control system characterized by In the guidance robot control system according to claim 4 ,
The robot can move,
A moving direction calculating means for calculating the moving direction of the user based on the position information of the user in the frame image acquired by the image acquiring means;
A movement control means for moving the robot based on the movement direction of the user calculated by the movement direction calculation means and the movement speed calculated by the movement speed calculation means;
Preparing,
Guide robot control system characterized by In the guidance robot control system according to any one of claims 1 to 5 ,
Analyzing at least one or more of the frame images acquired by the image acquisition means, comprising facial expression determination means for determining the facial expression of the user;
The guidance data storage unit stores guidance data corresponding to facial expressions,
The guidance means refers to the guidance data storage unit and performs guidance by extracting guidance data based on the facial expression of the user determined by the facial expression determination means;
Guide robot control system characterized by In the guidance robot control system according to claim 6 ,
After performing the guidance by the guidance data, at least the processing by the image acquisition unit and the processing by the facial expression discrimination unit are executed,
A facial expression comparison means for comparing the facial expression of the user before guidance with the facial expression of the user after guidance;
The guidance means refers to the guidance data storage unit and performs guidance by extracting guidance data based on a comparison result by the facial expression comparison means;
Guide robot control system characterized by In the guidance robot control system according to any one of claims 1 to 7 ,
Analyzing at least one or more of the frame images acquired by the image acquisition unit, and determining attribute of the user,
The guidance data storage unit stores guidance data corresponding to user attributes,
The guidance means refers to the guidance data storage unit and performs guidance by extracting guidance data corresponding to the attribute of the user determined by the attribute determination means;
Guide robot control system characterized by The program for functioning the processor of a robot as a guidance robot control system in any one of Claim 1-8 . A guidance data storage unit for storing guidance data for the user;
A shooting section;
Image acquisition means for acquiring a plurality of frame images continuous in time series via the imaging unit;
Analyzing the plurality of frame images acquired by the image acquisition unit, and for each frame image , based on a relative position between the user's eyeball and pupil, a direction determination unit that determines a direction in which the user is viewing;
A moving direction calculating means for calculating the moving direction of the user based on the position information of the user in the frame image acquired by the image acquiring means;
Analyzing the plurality of frame images acquired by the image acquisition means, and acquiring face data of the user;
Face direction determining means for determining the orientation of the user's face for each frame image;
Direction change amount calculating means for calculating a direction change amount representing the magnitude of change in the direction with time based on the direction determined by the direction determining means;
Guidance determining means for determining whether or not to guide based on a change in direction calculated by the direction change amount calculating means;
Guidance means for performing guidance by extracting guidance data from the guidance data storage unit when it is judged by the guidance judgment means to be guided;
Equipped with a,
The guidance determination unit is further configured in a case where the movement direction of the user calculated by the movement direction calculation unit is different from the direction of the face common to the plurality of frame images determined by the face direction determination unit. To decide to give guidance,
Guide robot said.

Images