JP5482412B2 - Robot, position estimation method and program - Google Patents

Description

  The present invention relates to a robot that estimates the position of a person using a panable and tiltable camera, a position estimating method, and a program.

  In a human-type or pet-type communication robot (hereinafter simply referred to as a robot), it is desirable to make eye contact with the user in order to give a close feeling to the user. In order for the robot to make eye contact with the user, it is necessary to provide a function of moving the neck of the robot so that the robot meets the user's eyes. This function can be realized by, for example, a method in which the user's face is imaged with a camera installed at the position of the eyes or nose of the robot, and the neck is controlled so that the user's face is positioned at the center of the captured image.

  However, while the robot is communicating with the user, there may be a case where the robot wants to make a gesture, shifts the line of sight to something other than the user, or uses the neck or camera for other tasks. Even in such a case, it is desirable for the robot to make eye contact with the user at an appropriate timing in order for the robot to communicate naturally with the user. However, depending on the task being performed by the robot, the user may It may be out of sight. Even in such a case, in order for the robot to make eye contact with the user smoothly, the user can be observed by observing the user even when the user is out of view or when only a part of the user is in view. It is necessary to estimate the position of.

  Conventionally, a method has been proposed in which a user's position is estimated by sound source detection using an array microphone provided in the robot, and the robot's neck is moved according to the estimated position to make eye contact with the user. However, with this method, the robot cannot make eye contact because the user's position cannot be estimated unless the user speaks. Also, depending on the environment in which the robot is used, it is difficult to accurately estimate the user's position from noise such as sound reflection.

  On the other hand, a method has been proposed in which a particle filter whose likelihood is the degree of coincidence between a silhouette image of a person and a simulation image simulating a silhouette image is configured for a user to be observed (Patent Document 1). However, in the method of estimating the position of the user based on the captured image captured by the camera installed at the position of the eyes or nose of the robot, the field of view changes depending on the posture of the neck of the robot (that is, the posture of the camera). Change. For this reason, a background difference obtained by comparing the background image prepared by the camera with a specific neck posture and the observation image captured by the camera with a different neck posture is obtained to track the user in the observed image. Therefore, when the background changes, it is difficult to accurately estimate the position of the user from the background difference. In addition, in order to obtain a background difference that makes it possible to estimate the position of the user regardless of the posture of the robot's neck, it is necessary to prepare an enormous amount of background images in advance, and to obtain the background difference It is necessary to perform an amount comparison operation, which increases the load on the processor that performs the comparison operation.

  There is a method of estimating the user's position by recognizing the user's face from the observed images captured by the camera, but the face recognition requires a complicated operation, and the load on the processor that performs the operation also increases. large. In addition, if the user's face is not facing the camera within a certain angle range, for example, the facial features such as eyes, nose and mouth are not included in the captured image of the camera, so that the face is not recognized. It is difficult to accurately estimate the user's position based on the result.

  For example, techniques for tracking or detecting an object using a color histogram have also been proposed (Patent Document 2, Patent Document 3 and Patent Document 4).

Japanese Patent No. 3664784 Japanese Patent Laid-Open No. 11-136664 JP 2002-247440 A JP 2006-508461 A

  In the conventional position estimation method, there is a problem that it is relatively easy and difficult to accurately estimate the position of the user to be observed.

  Accordingly, an object of the present invention is to provide a robot, a position estimation method, and a program capable of estimating the position of a user who is an observation target relatively easily and accurately.

According to an aspect of the present invention, a camera that can rotate about at least one axis, an image generation unit that calculates an observation value of an image that appears to be an observation object from an image captured by the camera, and an observation object An image prediction unit that calculates a predicted value of an image that seems to be an observation target based on a past estimation result of the position of the current position and the current posture of the camera, and compares the observed value with the predicted value to compare the observed value with the predicted value An image comparison unit that calculates the degree of coincidence as a likelihood, a position estimation unit that estimates the position of the observation target based on the likelihood, and the uncertainty of the position of the observation target estimated by the position estimation unit There is provided a robot including an observation policy determination unit that determines an observation policy based on the evaluation and a control unit that controls a rotational position of the camera relative to the at least one axis based on the observation policy .

According to an aspect of the present invention, an image generation step of calculating an observation value of an image that appears to be an observation target estimated from an image captured by a camera that can rotate about at least one axis by an image generation unit; An image prediction step for calculating a predicted value of an image that seems to be an observation target based on a past estimation result of the position of the current position and the current posture of the camera, and an image comparison unit for comparing the observed value and the predicted value. An image comparison step for calculating the degree of coincidence between the observed value and the predicted value as a likelihood, a position estimating step for estimating the position of the observation target based on the likelihood, and a position estimating step. An observation policy determining step for determining an observation policy by an observation policy determination unit based on the estimated uncertainty of the position of the observation target; and Look including a control step of controlling by the control unit the rotational position also with respect to one axis, the image generation process, a face area of a person is an observation target from a captured image from the camera is extracted by the face region detection unit, the camera A position estimation method is provided in which a foreground region is extracted from a captured image from a foreground region extraction unit, and an image that looks like a person is generated by the image generation unit based on the extracted face region and foreground region .

According to one aspect of the present invention, there is provided a program for causing a computer to perform a position estimation process for estimating the position of an observation target, the observation being estimated as an observation target from a captured image of a camera that can rotate about at least one axis. Calculate the observed value of the image that looks like the object and store it in the storage unit, calculate the predicted value of the image that looks like the observation object based on the past estimation result of the position of the observation object and the current posture of the camera An image prediction procedure to be stored in the storage unit, an image comparison procedure in which the observed value and the predicted value are compared and a degree of coincidence between the observed value and the predicted value is calculated as a likelihood and stored in the storage unit; a position estimation procedure for estimating the position of the observed object on the basis of the likelihood, observation policy for determining an observation policy based on evaluation of uncertainty of estimated position of the observed object by the position estimation procedure A constant procedure, the control procedure for controlling the rotational position relative to said at least one axis of the camera based on the observation policy is run on the computer, the image generation procedures, the extracted face region and the person on the basis of the foreground area A program for generating an image that looks like is provided.

  According to the disclosed robot, position estimation method, and program, it is possible to estimate the position of a user who is an observation target relatively easily and accurately.

It is a block diagram which shows an example of the robot in one Example of this invention. It is a figure explaining calculation of the prediction picture using a particle filter. It is a figure explaining the example which produces | generates an estimated image with the template which expressed the face in the circular shape and the body in the ellipse. It is a figure which shows an example of a face area | region and a template area | region. It is a flowchart explaining the process which produces | generates the image which looks like a person from an observation image. It is a figure which shows an example of the color histogram with respect to a 20 * 20 pixel block. It is a figure which shows the result of having calculated the multi-valued image which seems to be a foreground using the batacharia distance in extraction of the foreground area | region using a color histogram. It is a flowchart explaining operation | movement of a robot. It is a flowchart explaining the process of FIG.8 S13-S15 in detail. It is a flowchart explaining the process of FIG.8 S24 in detail. It is a block diagram which shows an example of a computer system.

  In the disclosed robot, the position estimation method, and the program, an observation value of an image that appears to be an observation object (or an image that is estimated to be a person) extracted from a captured image of a camera that can rotate around at least one axis is calculated and observed. Calculate the predicted value of the image that looks like the observation target based on the past estimation result of the target position and the camera posture, and the observation target based on the likelihood calculated from the comparison result of the observation value of the target target image and the predicted value Estimate the position.

  The estimation result of the position of the observation target can be used to control the posture of the camera based on an evaluation such as uncertainty.

  Hereinafter, embodiments of the disclosed robot, position estimation method, and program will be described with reference to the drawings.

  FIG. 1 is a block diagram showing an example of a robot in one embodiment of the present invention. In this embodiment, the present invention is applied to a humanoid communication robot.

  The communication robot 1 includes a robot body 11, a face area detection unit 12, a foreground area extraction unit 13, an image generation unit 14 that generates a person-like image, and an image prediction unit 15 that predicts a person-like image, as shown in FIG. An image comparison unit 16 that compares images that look like a person, a position estimation unit 17 that estimates the position of the person, an observation policy determination unit 18, and a neck control unit 19 that controls the rotation of the neck of the robot 1. Needless to say, the robot body 11 may include a known moving mechanism (not shown) that allows the robot 1 to walk or run.

  The robot body 11 includes a camera 21 capable of panning and tilting, a neck angle (or posture) acquisition unit 22, and a neck rotation driving unit 23. The camera 21 has a known structure that can be swung, and is provided at, for example, the position of the eyes or nose of the robot body 11 and captures an image that can be seen from the robot 1. Panning around the pan axis of the camera 21 and tilting around the tilt axis (hereinafter simply referred to as pan / tilt) are controlled by the neck rotation drive unit 23 in a known manner. The neck angle acquisition unit 22 acquires the neck angle of the camera 21, that is, the posture of the camera 21 by a known method. The neck angle of the camera 21 includes a pan angle and a tilt angle.

  The face area detection unit 12 extracts a face area of a user (that is, an observation target) that the robot 1 should communicate with from the captured image from the camera 21. As will be described later, the face area detection unit 12 detects a user-like area (that is, a face area) and does not perform personal authentication for recognizing the user's face. The foreground area extraction unit 13 extracts a foreground area from the captured image from the camera 21. The image generation unit 14 generates a person-like image, that is, a user-like image based on the extracted face area and foreground area. On the other hand, the image prediction unit 15 predicts a person-like image, that is, a user-like image, based on the neck angle acquired by the neck angle acquisition unit 22 and the past estimation result of the position of the person from the position estimation unit 17. The image comparison unit 16 compares the image (or observation value) generated by the image generation unit 14 with the predicted image (or prediction value) predicted by the image prediction unit 15, and compares the comparison result with the position estimation unit 17. Output to. For example, the image comparison unit 16 includes a known particle filter whose likelihood is the degree of coincidence between the observed image to be compared and the predicted image, and outputs a comparison result including the likelihood to the position estimation unit 17.

  The particle filter discretizes the continuous posterior probability density distribution of the observation target state, and sequentially matches the consistency with the observation results for the state of each member of the sample group called particle set (Particle Set). Evaluate and perform resampling. As a result, the posterior probability density distribution simulated with particles is converged to a true probability density distribution.

  The position estimation unit 17 estimates the position of the person, that is, the position of the user based on the comparison result from the image comparison unit 16. The observation policy determination unit 18 determines the user's observation policy, that is, what rule (or rule) to observe the user based on the evaluation of the estimated uncertainty of the position of the user. To do. For example, if the observation policy relates to the frequency of confirming the position of the user, the frequency of confirming the position of the user is set if the distance between the estimated user position and the previously estimated user position is within a threshold. It is set every 1 time, and when the threshold value is exceeded, the confirmation frequency is set every 2nd time shorter than the 1st time. The neck control unit 19 controls the posture of the camera 21 so that the user enters the field of view of the camera 21 by controlling the neck rotation driving unit 23 based on the observation policy determined by the observation policy determination unit 18.

  At least some of the face area detection unit 12, the foreground area extraction unit 13, the image generation unit 14, the image prediction unit 15, the image comparison unit 16, the position estimation unit 17, the observation policy determination unit 18, and the neck control unit 19 are robots. 1 may be externally attached. In this case, a portion externally attached to the robot 1 is connected to the robot 1 via an appropriate interface (not shown) provided on the robot 1. What is necessary is just to enable communication with the robot 1 and the part attached externally by a well-known method, for example via a wireless network (not shown). That is, the neck rotation driving unit 23 and the like of the robot 1 may be remotely controlled.

  Based on the neck angle acquired by the neck angle acquisition unit 22, the image prediction unit 15 calculates a predicted image that predicts an image that looks like a user using, for example, a known particle filter. FIG. 2 is a diagram for explaining prediction image calculation using a particle filter. In FIG. 2, the position of the user 30 is represented by parameters d, h, and θ shown in FIG. The parameter d indicates the shortest distance from the pan axis of the camera 21 (or neck) of the robot 1 to the user 30, and in this example is the distance to the position of the body 32 of the user 30. The parameter h indicates the height from the reference position of the camera 21 of the robot 1 to the center of the face 31. The parameter θ represents an angle formed between the line of sight FY at the fixed position of the camera 21 and the direction of the shortest distance d.

  In the example of FIG. 2, the robot 1 is a pet-type communication robot in the shape of a teddy bear. For example, the camera 21 is installed on the bear's eyes (or nose), the neck rotation drive unit 23 is provided on the bear's neck, and the neck angle acquisition unit 22 is provided on the bear's trunk. .

The predicted image can be generated from the estimated position of the face 31 of the user 30 and a human template representing the silhouette of a person. In this case, if the number of pixels per degree in the horizontal direction and the vertical direction is α and β, the coordinates (uf, vf) of the face 31 in the image are the coordinates of the image center (uc, vc), pan The tilt angles are pan and tilt, respectively, and the distance between the pan axis and tilt axis and the optical center (or optical axis) of the camera 21 is a,
b, where the distance in the optical axis direction between the pan axis and the tilt axis is denoted by c, it can be calculated based on the following equation. In this example, the pan axis is perpendicular to the horizontal plane on which the robot 1 is installed, and the tilt axis is parallel to the horizontal plane and orthogonal to the pan axis.

uf = uc + α × atan (d × sin (θ-pan) / (d × cos (θ-pan) -a))
vf = vc + β × atan (d × (atan (h / d) -tilt) / ((dc) × cos (atan (h / d) -tilt) -b))

  In particular, when the parameters a, b, and c are small enough to be ignored, the calculation can be simplified as in the following equation.

uf = uc + α × (θ-pan)
vf = vc + β × (atan (h / d) -tilt)

  FIG. 3 is a diagram illustrating an example in which a predicted image is generated using a template in which a face is represented by a circle and a body is represented by an ellipse. Strictly speaking, the shape of the person changes depending on the tilt angle. However, in this example, only a rough estimation is required for convenience of explanation, and it is assumed that errors caused by changes in the shape of the person can be ignored. A regular human template 30A shown in FIG. 3 has (vf, uf) as a base point, and has a circular face template portion 31A and an elliptical body template portion 32A. Here, the size of the humanoid template 30 </ b> A is changed according to the distance d from the camera 21 of the user 30.

  When generating an image representing the degree of humanity from an image captured and observed by the camera 21, a combination of several different generation methods is employed. This is because, due to the pan / tilt of the camera 21, notable features of the user such as the face 31 may not enter the field of view of the camera 21. For example, an image that looks like a person may be generated from the observed image based on at least one of the following indices or determination criteria JR1 to JR3.

  (JR1) When the face area 31B is detected, the face area 31B has a very high degree of personality (or probability).

  (JR2) The region of the human template 30A similar to the above with the face region 31B as a base point has a high degree of humanity (or probability).

  (JR3) The area extracted as the foreground by the foreground area extraction unit 13 (ie, the foreground area) has a higher degree of personality (or probability) than the area set as the background (ie, the background area).

  FIG. 4 is a diagram illustrating an example of a face area and a template area. In FIG. 4, when the face area 31 is detected, it can be seen that the humanoid area 30B including the body area 32B based on the image of the face area 31B has a very high human character. The size of the body region 32B (or the humanoid region 30B) is proportional to the size of the detected face region 31B. Therefore, for example, according to Table 1, a value indicating the character of a person can be determined from the observed image.

  FIG. 5 is a flowchart for explaining processing for generating an image that looks like a person from an observed image. In FIG. 5, step S <b> 1 is executed by the face area extraction unit 12 of FIG. 1, detects the user's face from the captured image from the camera 21, and extracts the face area 31 </ b> B and the body area 32 </ b> B. Step S <b> 2 is executed by the foreground area extraction unit 13 of FIG. 1, and extracts the foreground area from the captured image from the camera 21. Step S3 is executed by the image generation unit 14 of FIG. 1, and determines the value of each pixel according to Table 1 based on the extracted face region 31B and foreground region, and generates a person-like image, that is, a user-like image. .

  When the foreground region is extracted in step S2, the feature amount of each block divided by dividing the captured image into a plurality of appropriate blocks (or areas) in the entire movable range of the neck of the robot 1 is, for example, the foreground. The data is stored in a storage unit (not shown) in the region extraction unit 13 and stored in the storage unit and the feature amount of each block of the region currently captured by the camera 21 (ie, currently visible from the camera 21). The feature quantities of the blocks in the corresponding area are compared, and the difference is treated as the foreground area. It is desirable that the feature amount used in this case represents the feature of the region and is not easily affected by changes in illumination or pan / tilt errors. In this example, a color histogram is used as a feature amount. A color histogram is an image that is converted into an HSV image (Hue, Saturation, Value), for example, and the appearance frequency is measured for each range of the hue value of each pixel. It is the result.

  FIG. 6 is a diagram illustrating an example of a color histogram for a rectangular block formed of 20 pixels × 20 pixels in a captured image. 6A shows a captured image from the camera 21, and FIG. 6B shows a color histogram calculated for the rectangular block in FIG. In the color histogram, C1 to C5 indicate different color components.

  When storing the background area in the storage unit, the camera 21 captures an image within the movable range of the neck of the robot 1 and stores the feature amount in the storage unit at a certain angle both right and left and up and down with respect to the entire movable range of the neck. The association between the angle at the time of storing in the storage unit and the feature amount of the captured image can be determined from the current pan angle and tilt angle (hereinafter referred to as pan / tilt angle) and the position of each block in the image. Rather than storing the captured image for the movable range of the neck, the feature amount is stored in the storage unit, so that the storage capacity required for the storage unit can be greatly reduced as compared to storing the captured image.

  On the other hand, when extracting the foreground area, the foreground area is extracted based on the comparison between the feature quantity of the background area stored in the storage unit and the feature quantity of the image currently captured by the camera 21. At this time, as a comparison method, a Bhattacharyya distance, a normalized correlation, or the like can be used. When a person-like image is generated, the normalized correlation value and the batcharia distance value can be used. That is, for example, in the comparison of color histograms, a value obtained by appropriately converting the scale used for the comparison (normalized correlation or batcharia distance) can be used as the measure of humanity.

  For example, in order to obtain a multi-valued image of 0 to 1, the batcharia distance takes a value of 0 to 1, and the feature value to be compared is similar as the value of the batcharia is closer to 0. Can be directly used as the pixel value of the multi-valued image. In order to obtain a multi-valued image of 0 to 1, the normalized correlation value takes a value of −1 to 1, and the closer to 1, the more similar the feature quantity to be compared. A value obtained by multiplying the value of −0.5 by −1 and adding 1 can be used as the pixel value of the multilevel image.

  FIG. 7 is a diagram illustrating a result of calculating a multi-valued image that looks like the foreground using the batacharia distance in the extraction of the foreground region using the color histogram as shown in FIG. 7A shows a captured image from the camera 21, and FIG. 7B shows a comparison between the color histogram calculated for the captured image in FIG. 7A and the color histogram of the background portion stored in the storage unit. The multi-valued image calculated using the Batacharia distance is shown. Here, for convenience, the multi-valued image is indicated by black pixels.

  Table 2 shows an example of a method for actually determining a person-like image from multi-valued foreground information (hereinafter referred to as foreground level). The processing in this case may be performed in the same procedure as the processing in FIG. For convenience of explanation, it is assumed in Table 2 that the foreground level is normalized to 0 to 1.

  Note that the color histogram of the human area is stored in the storage unit, and how often the representative color in each block (or each area) of the captured image from the camera 21 is in the human area (that is, the color frequency). It is also possible to use information indicating whether or not. In this case, for example, as shown in Table 3, the values of person-likeness, that is, the values of the face area 31B and the body area 32B may be determined. In this case, the same procedure as the process of FIG. 5 may be performed. Table 3 shows an example of a method for determining the person-likeness from the captured image from the camera 21. For convenience of explanation, it is assumed in Table 3 that the color frequency in the human area (that is, the face area 31B and the body area 32B) is normalized to 0 to 1.

  If the face area extraction unit 12 of FIG. 1 has a known face identification function, the human template may be changed for each person whose face is identified.

Also, when face identification can be used when calculating the above likelihood, the probability that there is another person in front of the identified face is low, so the likelihood of particles that meet that condition is reduced. When performing likelihood calculation in the image comparison unit 16 in FIG. 1, when the hand is held in front of the camera 21, the entire region of the captured image is determined as the foreground. For this reason, when the face area 31B is not detected and many areas in the captured image are foreground areas, the observation result may be invalid and nothing may be observed.

  If the required estimation accuracy cannot be obtained with only the captured image, information indicating in which direction a person may be present with respect to the robot 1 is also used for likelihood calculation by an ultrasonic sensor, an array microphone, or the like. May be used. For example, particles in a direction in which the ultrasonic sensor reacts have a high likelihood. Ultrasonic sensors, array microphones, and the like may receive reflections, but erroneous detection due to reflections can be prevented by using the current estimation result and the previous estimation result together.

  In addition, when the lighting condition changes greatly or the posture of the camera is not fixed like a communication robot, the installation position (or attachment position) of the camera may deviate from the design position. In such a case, it is effective to periodically store the background feature amount in the storage unit so that the position estimation accuracy does not deteriorate. For this reason, the background feature amount may be stored again in the storage unit for a region having a very low probability of being a human region.

  If a person's existence probability varies to some extent, a function may be provided in which a camera is pointed at the person and checked. In this case, the robot's neck is moved toward the average value of the particles (ie, the predicted value of the face). If the estimated value changes due to a person entering the field of view of the camera while moving the neck, the manner of moving the neck may be changed as appropriate. This makes it possible to periodically estimate the position of a person who does not enter the field of view of the camera and maintain the position estimation accuracy at a certain level.

  In the case of tracking a plurality of persons, a plurality of person estimation programs may be operated to create a predicted image by adding a plurality of images.

  In the case of a communication robot, it is not necessary to detect a passing person, and it is only necessary to estimate the position of only the person that the communication robot faces. For this reason, when a face is detected in a place where no person can exist in the captured image of the camera, the estimation of a new person may be started.

  Also, if the distribution of particles spreads and the accuracy falls, after taking measures to improve the accuracy, such as checking the state of the corresponding direction, if the accuracy does not improve, it is determined that the person has left Just do it.

  In the case of a camera in which the neck can rotate in the roll direction around the roll axis, such as a neck movement by a communication robot, an image with appropriate coordinate transformation is input. For example, by obtaining the rotation center in the captured image in advance and generating the predicted image in the captured image rotated around the rotation center, the person can be tracked even when there is a roll angle.

  FIG. 8 is a flowchart for explaining the operation of the robot 1. 8, in step S11, for example, the neck control unit 19 controls the neck rotation driving unit 23 according to the default setting to move the neck of the robot 1, and the background region feature amount ( The background feature amount is also calculated and stored in the storage unit.

  In step S <b> 12, angular position information indicating the captured image from the camera 21 and the pan / tilt angle from the neck angle acquisition unit 22 is acquired. In step S <b> 12, the neck angle acquisition unit 22, for example, a motor (not shown) in a neck rotation drive unit 23 that controls the pan angle of the neck (or camera 21) of the robot 1 and a neck rotation drive that controls the tilt angle. The neck angle can be acquired from the angle position signal output from the angle sensor that detects the rotational angle position of each motor in the unit 23. When the motor itself that controls the pan angle and the tilt angle has a function of outputting an angle position signal indicating the rotation angle position, the neck angle may be acquired from the angle position signal output by each motor. When omitting the process of querying the rotational angle position of the motor, approximate to the instruction (target rotational angle θdist and arrival time duration) for the motor, the rotational angle θ0 at the time of the motor instruction, the instruction time t0, and the current time t by linear interpolation. The current rotation angle can also be obtained. In this case, the current angle θ can be calculated from, for example, an equation of θ = θ0 + (θdist−θ0) × (t−t0) / duration.

  In step S13, the background feature amount for each block in the field of view of the current camera 21 is acquired from the pan / tilt angle and stored in the storage unit. The process of step S13 will be described in detail later. In step S14, the captured image is divided into a plurality of blocks, a feature amount for each block is calculated and stored in the storage unit. In step S15, the foreground region is extracted by comparing the background feature quantity in block units acquired in step S13 with the feature quantity in block units calculated in step S14, and is stored in the storage unit. On the other hand, in step S16, the face area 31B is extracted from the captured image from the camera 21 and stored in the storage unit. In step S17, a person-like image is generated based on the extracted face area 31B and the extracted foreground area, and stored in the storage unit.

  In step S18, one hypothesis (or candidate) of an image that looks like a person is selected, and in step S19, the neck is driven by the neck rotation driving unit 23 and moved by a predetermined amount to generate a state transition. In step S20, a predicted image is generated as described above based on the angular position information indicating the pan / tilt angle and the past estimation result of the position of the person from the position estimation unit 17, and is stored in the storage unit. In step S21, the likelihood is calculated based on at least the generated predicted image and the person-like image, and is stored in the storage unit. In step S22, it is determined whether or not the likelihood calculation has been completed for all hypotheses. If the determination result is NO, the process returns to step S18.

  On the other hand, if the decision result in the step S22 is YES, a step S23 performs a resampling process by the particle filter. In step S24, a control instruction to move the neck if necessary is output to the neck rotation driving unit 23. In step S25, the background area is updated for a place where the probability that a person is present is lower than the threshold, that is, the probability is extremely low, and the process returns to step S12.

  In FIG. 8, if step S16 is omitted, step S17 may generate a person-like image based on the extracted foreground region. Further, step S25 may be omitted. In this case, after step S24, the process returns to step S12.

  By the way, in step S13, assuming that the number of pixels per 1 ° in the horizontal direction and the vertical direction is α and β, the coordinates (uf, vf) of the face 31 in the image are the coordinates of the image center (uc, vc), When the pan / tilt angle is represented by pan and tilt, and the distance between the pan axis and tilt axis and the optical center of the camera 21 is represented by a and b, respectively, it can be calculated in the same manner as when the predicted image is generated. At the time of generating the predicted image, it is necessary to obtain a point in the image from the predicted (θ, d, h). In step S13, the distance d is sufficiently long when the background feature is stored. Assuming that the angle in the tilt direction from the pan / tilt reference position is φ, the image feature quantity for the periphery of (θ, φ) in the background is stored. For example, when storing feature values for each of dθ and dφ, pan / tilt is performed over the entire movable range of the neck by dθ and dφ, and image features near the center of the captured image at each rotational position (pan / tilt position). The amount is calculated and stored in the storage unit. When the limit of the movable range of the neck is reached, the image feature amount is calculated for each of dθ and dφ outside the movable range and stored in the storage unit. Panning and tilting may be performed larger than dθ and dφ, and the image feature values of a plurality of blocks may be stored in the storage unit at a time. However, storing the image feature values near the center of the captured image is more effective when comparing feature values. The possibility of errors is reduced.

  In order to obtain the image feature amount of each block in the captured image from the pan / tilt angle (pan, tilt), the feature amount for each (θ, φ) stored in the storage unit and the current captured image It is necessary to find the correspondence with the coordinates (u, v). In this example, it is assumed that d is long enough to ignore a and b, and is obtained by the following equation.

u = uc + α × (θ-pan)
v = vc + β × (φ-tilt)

  When the above equation is converted into an equation for obtaining (θ, φ), the following equation is obtained.

θ = (u-uc) / α + pan
φ = (v-vc) / β + tilt
Thereby, the image feature amount (θ, φ) stored in the storage unit at the coordinates (u, v) in the captured image can be acquired. An image feature amount is calculated in the vicinity of the coordinates (u, v) corresponding to (θ, φ), and compared with the acquired image feature amount. The feature quantity for (θ, φ) may be calculated from blocks that overlap with the adjacent feature quantity (θ + dθ, φ + dφ).

  FIG. 9 is a flowchart for explaining in more detail the processing in steps S13 to S15 in FIG. 8 for acquiring the background feature amount of each block in the captured image and extracting the foreground. The block division method may be determined dynamically around each coordinate (u, v), or may be divided into blocks in advance to obtain the feature of the block including each coordinate (u, v). good.

  In FIG. 9, step S <b> 31 obtains a range of (θ, φ) within the field of view of the camera 21. In step S32, coordinates (u, v) corresponding to the discrete values (θ, φ) within the range stored in the storage unit are obtained. In step S33, one coordinate is selected from the obtained coordinates (u, v). In step S34, the background feature quantity for the selected coordinates (u, v) is acquired from the corresponding (θ, φ). Step S35 calculates the feature-value with respect to the peripheral area | region of the coordinate (u, v) in a captured image. A step S36 compares the background feature quantity with the feature quantity in the captured image to determine whether or not the block including the coordinates (u, v) is the foreground area, or foreground indicating the probability of being the foreground area. Find the degree. In step S37, it is determined whether or not the processing for all (u, v) extracted in step S32 has been completed. If the determination result is NO, the processing returns to step S33. On the other hand, the process ends if the decision result in the step S37 is YES.

  FIG. 10 is a flowchart for explaining the process of step S24 of FIG. 8 in more detail. In FIG. 10, step S41 calculates the variance of the estimated value θ for all persons currently estimated. In step S42, the person having the largest variance of the estimated value θ among all the persons and its variance are obtained. Step S43 determines whether or not the variance of the estimated value θ is larger than the threshold value. If the decision result in the step S43 is YES, a step S44 obtains (θ, d, h) of the estimated value (average estimated value) of the person to be observed. In step S45, a control instruction is output to the neck rotation drive unit 23 so as to control the pan / tilt angle to θ and atan (h / d), respectively. After step S45 or when the determination result in step S43 is NO, the process ends.

  As for the method of determining the user who is the observation target for making eye contact with the eyes of the robot 1, it is not always determined that the variance of the estimated value θ is the maximum, but priority is given to another user of interest, etc. What is necessary is just to determine suitably according to the application and use environment of the robot 1. FIG. In the above example, it is determined whether or not the robot 1 performs pan / tilt depending on whether or not the variance of the estimated value θ is larger than the threshold value. For example, since the face area 31B is detected last. The pan / tilt of the robot 1 can be controlled in accordance with various conditions such as the elapsed time of.

  FIG. 11 is a block diagram illustrating an example of a computer system. A computer system 100 illustrated in FIG. 11 has a configuration in which a CPU 101, a storage unit 102, an interface (I / F) 103, an input device 104, and a display unit 105 are connected by a bus 106. The CPU 101 controls the entire computer system 100 by executing a program stored in the storage unit 102. The storage unit 102 can be formed of a semiconductor storage device, a magnetic recording medium, an optical recording medium, a magneto-optical recording medium, and the like. The storage unit 102 stores the above programs and various data, and also performs intermediate results and calculation results of calculations executed by the CPU 101. It also functions as a temporary memory for temporarily storing etc. The I / F 103 can receive a captured image from the camera and can receive data stored in the storage unit 102 from a network (not shown). The input device 104 can be formed by a keyboard or the like. The display unit 105 can be formed by a display or the like. The input device 104 and the display unit 105 may be formed of an input / output device having functions of both the input device and the display unit, such as a touch panel. The input device 104 can be omitted when the user does not need to input an instruction or the like to the robot 1, and the display unit 105 is omitted when the robot 1 does not need to display a message or the like to the user. Is possible. Needless to say, a voice output unit (not shown) may be provided instead of the display unit 105 to output a message to the user by voice.

  The CPU 101 causes the computer system 100 to function as a robot by executing a program stored in the storage unit 102. That is, the program causes the CPU 101 to realize the function of each unit of the robot. In other words, the program causes the CPU 101 to execute at least the procedure of the robot position estimation process, and may be stored in an appropriate computer-readable storage medium including the storage unit 102. Therefore, the CPU 101 can execute the processes shown in FIGS.

  According to the above embodiment, since the communication robot can make eye contact with an arbitrary user at an arbitrary timing, communication between the communication robot and the user can be performed smoothly, and a more natural interactive operation is possible. Here, communication is not limited to dialogue in language. In order to maintain the accuracy of the estimated position of the user, the movement of the communication robot sometimes turns the line of sight to the user, for example, an action similar to that of an infant who occasionally confirms the presence of a mother while playing, making the movement of the robot very natural. Become. In addition, the sensing performance necessary for communication is improved, such as the user's facial expression with respect to the operation of the communication robot can be immediately confirmed, leading to a significant improvement in the basic performance of the communication robot.

  In the above embodiment, since the camera can pan and tilt, it can rotate around two orthogonal axes (for example, the x axis and the y axis of the xy coordinate system), or pan, tilt, and roll. Therefore, it is possible to rotate around three axes orthogonal to each other (for example, the x axis, the y axis, and the z axis of the xyz coordinate system). However, as long as the position of the user can be estimated, it is sufficient that the camera can rotate about at least one axis.

The following additional notes are further disclosed with respect to the embodiment including the above examples.
(Appendix 1)
A camera rotatable about at least one axis;
An image generation unit that calculates an observation value of an image that seems to be an observation target estimated from an image captured by the camera;
An image prediction unit that calculates a predicted value of an image that seems to be an observation target based on the past estimation result of the position of the observation target and the current posture of the camera;
An image comparison unit that compares the observed value with the predicted value and calculates the degree of coincidence between the observed value and the predicted value as a likelihood;
A robot comprising a position estimation unit that estimates the position of the observation target based on the likelihood.
(Appendix 2)
A face area detection unit that extracts a face area of a person to be observed from a captured image from the camera;
Further comprising a foreground region extraction unit for extracting a foreground region from a captured image from the camera;
The robot according to appendix 1, wherein the image generation unit generates an image that looks like a person based on the extracted face region and foreground region.
(Appendix 3)
When extracting the foreground area, the foreground area extraction unit stores in advance a feature amount of each block obtained by dividing the captured image in the entire movable range of the camera in a storage unit, and stores the current captured image of the camera. The robot according to supplementary note 2, wherein a feature amount of each block is compared with a feature amount of a corresponding block stored in advance in the storage unit, and a portion having a difference is set as a foreground region.
(Appendix 4)
The foreground area extracting unit extracts a butteraria distance (Bhattacharyya Distance) when extracting the foreground area based on a comparison between a feature quantity of a background area stored in the storage unit and a feature quantity of a current captured image of the camera. The robot according to appendix 3, wherein when the normalized correlation is used to generate an image that looks like a person, the normalized correlation value or the batcharia distance value is used as it is as the pixel value of the multi-valued image.
(Appendix 5)
The robot according to any one of appendices 1 to 4, further comprising an observation policy determination unit that determines an observation policy based on an evaluation of the uncertainty of the position of the observation target estimated by the position estimation unit.
(Appendix 6)
The robot according to claim 5, further comprising a control unit that controls a rotational position of the camera with respect to the at least one axis based on the observation policy.
(Appendix 7)
The robot according to appendix 6, wherein when the presence probability of a person varies to some extent, the camera is moved toward a predicted face value by controlling the control unit.
(Appendix 8)
An image generation step of calculating an observation value of an image that seems to be an observation object estimated from an image captured by a camera rotatable around at least one axis by an image generation unit;
An image prediction step of calculating a predicted value of an image that seems to be an observation target by an image prediction unit based on a past estimation result of the position of the observation target and the current posture of the camera;
An image comparison step of comparing the observed value and the predicted value by an image comparing unit and calculating the degree of coincidence of the observed value and the predicted value as a likelihood;
Including a position estimation step of estimating the position of the observation target by a position estimation unit based on the likelihood,
In the image generation step, a face area of a person to be observed is extracted from a captured image from the camera by a face area detection unit, and a foreground area is extracted from a captured image from the camera by a foreground area extraction unit. A position estimation method in which an image that looks like a person is generated by the image generation unit based on a face area and a foreground area.
(Appendix 9)
When extracting the foreground area, the feature amount of each block obtained by dividing the captured image in the entire movable range of the camera is stored in advance in a storage unit, and the feature amount of each block of the current captured image of the camera and the block The position estimation method according to appendix 8, wherein the feature amounts of the corresponding blocks stored in advance in the storage unit are compared, and a portion having a difference is set as a foreground region.
(Appendix 10)
When extracting the foreground region based on the comparison between the feature amount of the background region stored in the storage unit and the feature amount of the current captured image of the camera, a Bachattacharyya Distance or a normalized correlation is used. The position estimation method according to appendix 9, wherein when the image that looks like a person is generated, the value of the normalized correlation or the value of the virtual distance is used as it is as a pixel value of the multilevel image.
(Appendix 11)
The appendix 8 to any one of the appendixes 8 to 10, further including an observation policy determination step of determining an observation policy by an observation policy determination unit based on an evaluation of the uncertainty of the position of the observation target estimated by the position estimation step. Location estimation method.
(Appendix 12)
The position estimation method according to claim 11, further comprising a control step of controlling a rotational position of the camera relative to the at least one axis by a control unit based on the observation policy.
(Appendix 13)
A program that executes a position estimation process that causes a computer to estimate the position of an observation target,
An image generation procedure for calculating an observation value of an image that seems to be an observation object from a captured image of a camera that can rotate about at least one axis and storing the observation value in a storage unit;
An image prediction procedure for calculating a predicted value of an image that seems to be an observation target based on the past estimation result of the position of the observation target and the current posture of the camera, and storing it in the storage unit,
An image comparison procedure for comparing the observed value with the predicted value and calculating the degree of coincidence between the observed value and the predicted value as a likelihood and storing it in the storage unit;
A position estimation procedure for estimating the position of the observation target based on the likelihood;
A face area extraction procedure for extracting a face area of a person to be observed from a captured image from the camera and storing it in the storage unit;
Causing the computer to execute a foreground area extraction procedure for extracting a foreground area from a captured image from the camera and storing the foreground area in the storage unit;
The image generation procedure is a program for generating a person-like image based on the extracted face area and foreground area.
(Appendix 14)
In the foreground area extraction procedure, when extracting the foreground area, the feature amount of each block obtained by dividing the captured image over the entire movable range of the camera is stored in a storage unit in advance, and the current captured image of the camera is stored. 14. The program according to appendix 13, wherein the feature amount of each block is compared with the feature amount of the corresponding block stored in advance in the storage unit, and a portion having a difference is set as a foreground region.
(Appendix 15)
The foreground area extraction procedure includes a batch area distance (Bhattacharyya Distance) when extracting the foreground area based on a comparison between a feature quantity of a background area stored in the storage unit and a feature quantity of a current captured image of the camera. The program according to appendix 14, wherein the normalized correlation value or the virtual distance value is used as it is as the pixel value of the multi-valued image when generating an image that looks like a person using normalized correlation.
(Appendix 16)
The appendix 13 to 15, wherein the computer further executes an observation policy determination procedure for determining an observation policy based on an evaluation of the uncertainty of the position of the observation target estimated by the position estimation procedure. program.
(Appendix 17)
The program according to claim 16, further causing the computer to execute a control procedure for controlling a rotational position of the camera relative to the at least one axis based on the observation policy.

  As described above, the disclosed position estimation method, robot, and program have been described by way of examples. However, the present invention is not limited to the above examples, and various modifications and improvements can be made within the scope of the present invention. Needless to say.

DESCRIPTION OF SYMBOLS 1 Communication robot 11 Robot main body 12 Face area detection part 13 Foreground area extraction part 14 Image generation part 15 Image prediction part 16 Image comparison part 17 Position estimation part 18 Observation policy determination part 19 Neck control part 21 Camera 22 Neck angle acquisition part 23 Neck Rotation drive unit 101 CPU
102 storage unit

Claims (5)

A camera rotatable about at least one axis;
An image generation unit that calculates an observation value of an image that seems to be an observation target estimated from an image captured by the camera;
An image prediction unit that calculates a predicted value of an image that seems to be an observation target based on the past estimation result of the position of the observation target and the current posture of the camera;
An image comparison unit that compares the observed value with the predicted value and calculates the degree of coincidence between the observed value and the predicted value as a likelihood;
A position estimation unit that estimates the position of the observation target based on the likelihood ;
An observation policy determination unit that determines an observation policy based on an evaluation of the uncertainty of the position of the observation target estimated by the position estimation unit;
A robot comprising a control unit that controls a rotational position of the camera with respect to the at least one axis based on the observation policy . A face area detection unit that extracts a face area of a person to be observed from a captured image from the camera;
Further comprising a foreground region extraction unit for extracting a foreground region from a captured image from the camera
The robot according to claim 1, wherein the image generation unit generates a person-like image based on the extracted face area and foreground area.   When extracting the foreground area, the foreground area extraction unit stores in advance a feature amount of each block obtained by dividing the captured image in the entire movable range of the camera in a storage unit, and stores the current captured image of the camera. The robot according to claim 2, wherein a feature amount of each block is compared with a feature amount of a corresponding block stored in advance in the storage unit, and a portion having a difference is set as a foreground region. An image generation step of calculating an observation value of an image that seems to be an observation object estimated from an image captured by a camera rotatable around at least one axis by an image generation unit;
An image prediction step of calculating a predicted value of an image that seems to be an observation target by an image prediction unit based on a past estimation result of the position of the observation target and the current posture of the camera;
An image comparison step of comparing the observed value and the predicted value by an image comparing unit and calculating the degree of coincidence of the observed value and the predicted value as a likelihood;
A position estimation step of estimating the position of the observation target based on the likelihood by a position estimation unit ;
An observation policy determination step for determining an observation policy by an observation policy determination unit based on an evaluation of the uncertainty of the position of the observation target estimated by the position estimation step;
A control step of controlling a rotational position of the camera relative to the at least one axis based on the observation policy by a control unit ;
In the image generation step, a face area of a person to be observed is extracted from a captured image from the camera by a face area detection unit, and a foreground area is extracted from a captured image from the camera by a foreground area extraction unit. A position estimation method in which an image that looks like a person is generated by the image generation unit based on a face area and a foreground area. A program that executes a position estimation process that causes a computer to estimate the position of an observation target,
An image generation procedure for calculating an observation value of an image that seems to be an observation object from a captured image of a camera that can rotate about at least one axis and storing the observation value in a storage unit;
An image prediction procedure for calculating a predicted value of an image that seems to be an observation target based on the past estimation result of the position of the observation target and the current posture of the camera, and storing it in the storage unit,
An image comparison procedure for comparing the observed value with the predicted value and calculating the degree of coincidence between the observed value and the predicted value as a likelihood and storing it in the storage unit;
A position estimation procedure for estimating the position of the observation target based on the likelihood;
A face area extraction procedure for extracting a face area of a person to be observed from a captured image from the camera and storing it in the storage unit;
A foreground area extraction procedure for extracting a foreground area from a captured image from the camera and storing it in the storage unit ;
An observation policy determination procedure for determining an observation policy based on an evaluation of the uncertainty of the position of the observation target estimated by the position estimation procedure;
Causing the computer to execute a control procedure for controlling a rotational position of the camera relative to the at least one axis based on the observation policy ;
The image generation procedure is a program for generating a person-like image based on the extracted face area and foreground area.

Images