JPWO2018084170A1 - Autonomous robot that identifies people - Google Patents

Abstract

The robot 100 images the user with a built-in camera, and extracts a feature vector that quantitatively represents the user's physical characteristics from the captured image. Further, the robot 100 extracts a feature vector (master vector) from a captured image (master image) when held by the user. In the robot system, the user is identified by comparing the master vector with a feature vector extracted from an unknown user. The robot 100 may extract a master vector from a master image that is touched by a user instead of being held.

Description

  The present invention relates to a robot that autonomously selects an action according to an internal state or an external environment.

  Humans acquire various information from the external environment through sensory organs and select behavior. Some choose consciously, others choose unconscious behavior. Repeated behavior eventually becomes unconscious behavior, and new behavior remains in the conscious area.

  Humans believe that they have the will to choose their actions freely, that is, free will. Humans feel emotions such as love and hate for others because they believe that others have free will. Those who have free will, at least those who can be assumed to have free will, will also heal human loneliness.

  The reason humans keep pets is that they give healing rather than whether they are useful or not. Pets can become good human companions because they are more or less free-minded.

  On the other hand, many people give up pets for various reasons, such as not having enough time to care for pets, lack of living environment to keep pets, allergies, and difficulty in bereavement. If there is a robot that plays the role of a pet, it may be possible to give healing to a person who cannot keep a pet (see Patent Document 1).

JP 2000-323219 A

In recent years, robot technology has been progressing rapidly, but has not yet realized its presence as a companion like a pet. This is because the robot does not seem to have free will. By observing behaviors that can only be thought of as a pet's free will, humans feel the presence of free will, sympathize with the pet, and be healed by the pet.
Therefore, if the robot can express human / biological behavior, it is considered that the sympathy with the robot can be greatly enhanced, especially if the robot changes its behavior according to the opponent.

  In order to realize the behavioral characteristics described above, the robot must have the ability to identify humans. In the face authentication technology, by comparing a captured image (hereinafter referred to as “master image”) to be a reference of a known person A and a captured image of an unconfirmed person X (hereinafter referred to as “inspection image”). It is determined whether the person A and the person X are the same person. When acquiring a master image, the system often instructs the person who is the subject about the posture and facial expression at the time of imaging.

  A high-quality master image is required to improve the person identification accuracy, but it is not preferable to place an excessive burden on the user in order to acquire the master image. In particular, forcing a user to act on a robot that should realize biological behavior characteristics may cause the user to feel the abiotic nature of the robot.

  The present invention has been completed on the basis of the above recognition, and its main object is to provide a technique for increasing the identification ability of a robot while suppressing the burden on the user.

An autonomous behavior robot according to an aspect of the present invention includes an imaging control unit that controls a camera, a recognition unit that determines a moving object based on a feature vector extracted from a captured image of the moving object, and a determination result, An operation selection unit that selects a motion of the robot, a drive mechanism that executes the motion selected by the operation selection unit, and an operation detection unit that detects lifting of the robot by a moving object.
The recognition unit sets a captured image when the robot is held on the moving object as a master image, and sets a moving object discrimination criterion based on a feature vector extracted from the master image.

An autonomous behavior robot according to another aspect of the present invention includes an imaging control unit that controls a camera, a recognition unit that discriminates a moving object based on a feature vector extracted from a captured image of the moving object, and a determination result. A motion selection unit that selects a motion of the robot, a drive mechanism that executes the motion selected by the motion selection unit, and a motion detection unit that detects a touch by a moving object.
The recognition unit sets a captured image when a touch is detected as a master image, and sets a moving object discrimination criterion based on a feature vector extracted from the master image.

An autonomous behavior robot according to another aspect of the present invention includes an imaging control unit that controls a camera, a recognition unit that discriminates a moving object based on a feature vector extracted from a captured image of the moving object, and a determination result. A motion selection unit that selects a motion of the robot, and a drive mechanism that executes the motion selected by the motion selection unit.
The recognition unit sets, as a master image, an image captured when the moving object is located at a predetermined relative point with respect to the robot, and sets a moving object discrimination criterion based on a feature vector extracted from the master image To do.

The behavior control program according to an aspect of the present invention is a computer program for object recognition by a robot.
This program has a function of setting a captured image of a moving object when the robot is held on the moving object as a master image, a function of setting a discrimination criterion for the moving object based on a feature vector extracted from the master image, and The robot is caused to exhibit the function of discriminating the moving object based on the feature vector extracted from the captured image of the moving object.

A behavior control program according to another aspect of the present invention is a computer program for object recognition by a robot.
A function for setting a captured image of a moving object when the robot touches the moving object as a master image, a function for setting a discrimination criterion for the moving object based on a feature vector extracted from the master image, and imaging of the moving object The robot is allowed to demonstrate a function of discriminating a moving object based on a feature vector extracted from an image.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes easy to improve the identification capability of a robot, suppressing the burden on a user.

It is a front external view of a robot. It is a side external view of a robot. It is sectional drawing which represents the structure of a robot roughly. It is a block diagram of a robot system. It is a conceptual diagram of an emotion map. It is a hardware block diagram of a robot. It is a functional block diagram of a robot system. It is an image figure when holding a robot. It is a data structure figure of master information. It is a 1st schematic diagram for demonstrating a user identification method. It is a 2nd schematic diagram for demonstrating a user identification method. It is a flowchart which shows the extraction process process of a master vector. It is a schematic diagram which shows the image tracking method of a user. It is a schematic diagram for demonstrating the method to extract a master vector from a distance.

FIG. 1A is a front external view of the robot 100. FIG. 1B is a side external view of the robot 100.
The robot 100 according to the present embodiment is an autonomous behavior type robot that determines an action and a gesture (gesture) based on an external environment and an internal state. The external environment is recognized by various sensors such as a camera and a thermo sensor. The internal state is quantified as various parameters expressing the emotion of the robot 100. These will be described later.

  As a general rule, the robot 100 uses the interior of the owner's family as the range of action. Hereinafter, a person related to the robot 100 is referred to as a “user”, and a user who is a member of the home to which the robot 100 belongs is referred to as an “owner”. The “moving object” to be identified by the robot 100 includes both humans and pets, but in the present embodiment, description will be made on a human (user).

  The body 104 of the robot 100 has a rounded shape as a whole and includes an outer skin formed of a soft and elastic material such as urethane, rubber, resin, or fiber. The robot 100 may be dressed. By making the body 104 round, soft, and comfortable to touch, the robot 100 provides the user with a sense of security and a comfortable touch.

  The robot 100 has a total weight of 15 kg or less, preferably 10 kg or less, more preferably 5 kg or less. By 13 months after birth, the majority of babies start walking alone. The average weight of a 13-month-old baby is over 9 kilograms for boys and less than 9 kilograms for girls. Therefore, if the total weight of the robot 100 is 10 kg or less, the user can hold the robot 100 with almost the same effort as holding a baby who cannot walk alone. The average weight of babies under 2 months of age is less than 5 kilograms for both men and women. Therefore, if the total weight of the robot 100 is 5 kilograms or less, the user can hold the robot 100 with the same effort as holding an infant.

  Various attributes such as moderate weight, roundness, softness, and good touch feel that the user can easily hold the robot 100 and that the user wants to hold it. For the same reason, the height of the robot 100 is 1.2 meters or less, preferably 0.7 meters or less. It is an important concept that the robot 100 in this embodiment can be held.

  The robot 100 includes three wheels for traveling on three wheels. As shown, a pair of front wheels 102 (left wheel 102a and right wheel 102b) and one rear wheel 103 are included. The front wheel 102 is a driving wheel, and the rear wheel 103 is a driven wheel. The front wheel 102 does not have a steering mechanism, but the rotation speed and the rotation direction can be individually controlled. The rear wheel 103 is a so-called omni wheel, and is rotatable in order to move the robot 100 forward, backward, left and right. By making the rotation speed of the right wheel 102b larger than that of the left wheel 102a, the robot 100 can turn left or rotate counterclockwise. By making the rotation speed of the left wheel 102a larger than that of the right wheel 102b, the robot 100 can turn right or rotate clockwise.

  The front wheel 102 and the rear wheel 103 can be completely accommodated in the body 104 by a drive mechanism (rotation mechanism, link mechanism). Although most of the wheels are hidden by the body 104 even during traveling, the robot 100 is incapable of moving when the wheels are completely stored in the body 104. That is, the body 104 descends and sits on the floor F as the wheels are retracted. In this seated state, a flat seating surface 108 (grounding bottom surface) formed on the bottom of the body 104 abuts against the floor surface F.

  The robot 100 has two hands 106. The hand 106 does not have a function of gripping an object. The hand 106 can perform simple operations such as raising, shaking, and vibrating. The two hands 106 can also be individually controlled.

  The eye 110 can display an image using a liquid crystal element or an organic EL element. The robot 100 includes various sensors such as a microphone array, an ultrasonic sensor, an odor sensor, a distance measuring sensor, and an acceleration sensor that can specify the sound source direction. The robot 100 also has a built-in speaker and can emit simple voices. A capacitive touch sensor is installed on the body 104 of the robot 100. With the touch sensor, the robot 100 can detect a user's touch.

  A horn 112 is attached to the head of the robot 100. Since the robot 100 is lightweight as described above, the user can lift the robot 100 by holding the horn 112. An omnidirectional camera is attached to the horn 112, and the entire upper area of the robot 100 can be imaged at once.

FIG. 2 is a cross-sectional view schematically showing the structure of the robot 100.
As shown in FIG. 2, the body 104 of the robot 100 includes a base frame 308, a main body frame 310, a pair of resin wheel covers 312, and an outer skin 314. The base frame 308 is made of metal and constitutes an axis of the body 104 and supports an internal mechanism. The base frame 308 is configured by vertically connecting an upper plate 332 and a lower plate 334 by a plurality of side plates 336. A sufficient space is provided between the plurality of side plates 336 so as to allow ventilation. A battery 118, a control circuit 342, and various actuators are housed inside the base frame 308.

  The main body frame 310 is made of a resin material and includes a head frame 316 and a body frame 318. The head frame 316 has a hollow hemispherical shape and forms the head skeleton of the robot 100. The torso frame 318 has a stepped cylinder shape and forms a torso skeleton of the robot 100. The body frame 318 is fixed integrally with the base frame 308. The head frame 316 is assembled to the upper end portion of the trunk frame 318 so as to be relatively displaceable.

  The head frame 316 is provided with three axes of a yaw axis 320, a pitch axis 322, and a roll axis 324, and an actuator 326 for rotationally driving each axis. Actuator 326 includes a plurality of servo motors for individually driving each axis. The yaw shaft 320 is driven for the swinging motion, the pitch shaft 322 is driven for the rolling motion, and the roll shaft 324 is driven for the tilting motion.

  A plate 325 that supports the yaw shaft 320 is fixed to the upper portion of the head frame 316. The plate 325 is formed with a plurality of ventilation holes 327 for ensuring ventilation between the upper and lower sides.

  A metal base plate 328 is provided to support the head frame 316 and its internal mechanism from below. The base plate 328 is connected to the plate 325 via a cross link mechanism 329 (pantograph mechanism), and is connected to the upper plate 332 (base frame 308) via a joint 330.

  The body frame 318 houses the base frame 308 and the wheel drive mechanism 370. Wheel drive mechanism 370 includes a rotation shaft 378 and an actuator 379. The lower half of the body frame 318 has a small width so as to form a storage space S for the front wheel 102 between the body frame 318 and the wheel cover 312.

  The outer skin 314 is made of urethane rubber and covers the main body frame 310 and the wheel cover 312 from the outside. The hand 106 is integrally formed with the outer skin 314. An opening 390 for introducing outside air is provided at the upper end of the outer skin 314.

FIG. 3 is a configuration diagram of the robot system 300.
The robot system 300 includes a robot 100, a server 200, and a plurality of external sensors 114. A plurality of external sensors 114 (external sensors 114a, 114b,..., 114n) are installed in advance in the house. The external sensor 114 may be fixed to the wall surface of the house or may be placed on the floor. In the server 200, the position coordinates of the external sensor 114 are registered. The position coordinates are defined as x, y coordinates in the house assumed as the action range of the robot 100.

Server 200 is installed in a house. The server 200 and the robot 100 in this embodiment usually correspond one-on-one. Based on information obtained from the sensors built in the robot 100 and the plurality of external sensors 114, the server 200 determines the basic behavior of the robot 100.
The external sensor 114 is for reinforcing the sensory organ of the robot 100, and the server 200 is for reinforcing the brain of the robot 100.

The external sensor 114 periodically transmits a radio signal (hereinafter referred to as “robot search signal”) including the ID of the external sensor 114 (hereinafter referred to as “beacon ID”). When the robot 100 receives the robot search signal, it returns a radio signal including a beacon ID (hereinafter referred to as a “robot response signal”). The server 200 measures the time from when the external sensor 114 transmits the robot search signal to when it receives the robot response signal, and measures the distance from the external sensor 114 to the robot 100. By measuring the distances between the plurality of external sensors 114 and the robot 100, the position coordinates of the robot 100 are specified.
Of course, the robot 100 may periodically transmit its position coordinates to the server 200.

FIG. 4 is a conceptual diagram of the emotion map 116.
The emotion map 116 is a data table stored in the server 200. The robot 100 selects an action according to the emotion map 116. The emotion map 116 shown in FIG. 4 shows the magnitude of the affectionate feeling with respect to the location of the robot 100. The x-axis and y-axis of the emotion map 116 indicate two-dimensional space coordinates. The z-axis indicates the size of feelings of good and bad. When the z value is positive, the place is highly likable. When the z value is negative, the place is disliked.

In the emotion map 116 of FIG. 4, the coordinate P <b> 1 is a point where the favorable feeling is high (hereinafter referred to as “favorite point”) in the indoor space managed by the server 200 as the action range of the robot 100. The favor point may be a “safe place” such as the shade of a sofa or under a table, a place where people can easily gather, such as a living room, or a lively place. Further, it may be a place that has been gently stroked or touched in the past.
The definition of what kind of place the robot 100 prefers is arbitrary, but in general, it is desirable to set a place favored by small children, small animals such as dogs and cats, as a favorable point.

The coordinate P2 is a point where the bad feeling is high (hereinafter referred to as “disgusting point”). Disgusting points include places that make loud noises such as near TVs, places that get wet easily like baths and toilets, closed spaces and dark places, and places that lead to unpleasant memories that have been treated wildly by users. There may be.
Although the definition of what place the robot 100 dislikes is arbitrary, it is generally desirable to set a place where a small child or a small animal such as a dog or cat is afraid as an aversion point.

  The coordinate Q indicates the current position of the robot 100. The server 200 specifies the position coordinates of the robot 100 based on the robot search signal periodically transmitted by the plurality of external sensors 114 and the robot response signal corresponding thereto. For example, when the external sensor 114 with the beacon ID = 1 and the external sensor 114 with the beacon ID = 2 each detect the robot 100, the distance of the robot 100 is obtained from the two external sensors 114, and the position coordinates of the robot 100 are obtained therefrom. .

Alternatively, the external sensor 114 with the beacon ID = 1 transmits a robot search signal in a plurality of directions, and the robot 100 returns a robot response signal when receiving the robot search signal. Thereby, the server 200 may grasp how far the robot 100 is from which external sensor 114 in which direction. In another embodiment, the movement distance of the robot 100 may be calculated from the number of rotations of the front wheel 102 or the rear wheel 103 to identify the current position, or the current position may be determined based on an image obtained from a camera. You may specify.
When the emotion map 116 shown in FIG. 4 is given, the robot 100 moves in a direction attracted to the favorable point (coordinate P1) and away from the disgusting point (coordinate P2).

  The emotion map 116 changes dynamically. When the robot 100 reaches the coordinate P1, the z value (favorable feeling) at the coordinate P1 decreases with time. Thereby, the robot 100 can emulate the biological behavior of reaching the favored point (coordinate P1), “satisfied with emotion”, and eventually “getting bored” at that place. Similarly, the bad feeling at the coordinate P2 is also alleviated with time. As time passes, new favor points and dislike points are born, and the robot 100 makes a new action selection. The robot 100 has an “interest” at a new favorable point and continuously selects an action.

  The emotion map 116 represents the undulation of emotion as the internal state of the robot 100. The robot 100 aims at the favor point, avoids the dislike point, stays at the favor point for a while, and eventually takes the next action. By such control, the action selection of the robot 100 can be made human and biological.

  The map that affects the behavior of the robot 100 (hereinafter collectively referred to as “behavior map”) is not limited to the emotion map 116 of the type shown in FIG. For example, it is possible to define various behavior maps such as curiosity, feelings of avoiding fear, feelings of peace, feelings of calm and dimness, feelings of physical comfort such as coolness and warmth. Then, the destination point of the robot 100 may be determined by weighted averaging the z values of the plurality of behavior maps.

  In addition to the behavior map, the robot 100 has parameters indicating the sizes of various emotions and sensations. For example, when the value of the emotion parameter of loneliness is increasing, the weighting coefficient of the behavior map that evaluates a safe place is set large, and the value of the emotion parameter is decreased by reaching the target point. Similarly, when the value of a parameter indicating a sense of boring is increasing, a weighting coefficient of an action map for evaluating a place satisfying curiosity may be set large.

FIG. 5 is a hardware configuration diagram of the robot 100.
The robot 100 includes an internal sensor 128, a communication device 126, a storage device 124, a processor 122, a drive mechanism 120, and a battery 118. The drive mechanism 120 includes the wheel drive mechanism 370 described above. The processor 122 and the storage device 124 are included in the control circuit 342. Each unit is connected to each other by a power line 130 and a signal line 132. The battery 118 supplies power to each unit via the power line 130. Each unit transmits and receives control signals via a signal line 132. The battery 118 is a lithium ion secondary battery and is a power source for the robot 100.

  The internal sensor 128 is a collection of various sensors built in the robot 100. Specifically, a camera (omnidirectional camera), a microphone array, a distance measuring sensor (infrared sensor), a thermo sensor, a touch sensor, an acceleration sensor, an odor sensor, a touch sensor, and the like. The touch sensor is installed between the outer skin 314 and the main body frame 310 to detect a user's touch. The odor sensor is a known sensor that applies the principle that the electrical resistance changes due to the adsorption of molecules that cause odors. The odor sensor classifies various odors into a plurality of categories.

  The communication device 126 is a communication module that performs wireless communication for various external devices such as the server 200, the external sensor 114, and a mobile device possessed by the user. The storage device 124 includes a nonvolatile memory and a volatile memory, and stores a computer program and various setting information. The processor 122 is a computer program execution means. The drive mechanism 120 is an actuator that controls the internal mechanism. In addition to this, displays and speakers are also installed.

  The processor 122 selects an action of the robot 100 while communicating with the server 200 and the external sensor 114 via the communication device 126. Various external information obtained by the internal sensor 128 also affects action selection. The drive mechanism 120 mainly controls the wheel (front wheel 102) and the head (head frame 316). The drive mechanism 120 changes the movement direction and movement speed of the robot 100 by changing the rotation speed and rotation direction of each of the two front wheels 102. The drive mechanism 120 can also raise and lower the wheels (the front wheel 102 and the rear wheel 103). When the wheel rises, the wheel is completely stored in the body 104, and the robot 100 comes into contact with the floor surface F at the seating surface 108 and enters the seating state.

FIG. 6 is a functional block diagram of the robot system 300.
As described above, the robot system 300 includes the robot 100, the server 200, and the plurality of external sensors 114. Each component of the robot 100 and the server 200 includes an arithmetic unit such as a CPU (Central Processing Unit) and various coprocessors, a storage device such as a memory and a storage, hardware including a wired or wireless communication line connecting them, and a storage It is realized by software stored in the apparatus and supplying processing instructions to the arithmetic unit. The computer program may be constituted by a device driver, an operating system, various application programs located in an upper layer thereof, and a library that provides a common function to these programs. Each block described below is not a hardware unit configuration but a functional unit block.
Some of the functions of the robot 100 may be realized by the server 200, and some or all of the functions of the server 200 may be realized by the robot 100.

(Server 200)
Server 200 includes a communication unit 204, a data processing unit 202, and a data storage unit 206.
The communication unit 204 is in charge of communication processing with the external sensor 114 and the robot 100. The data storage unit 206 stores various data. The data processing unit 202 executes various processes based on the data acquired by the communication unit 204 and the data stored in the data storage unit 206. The data processing unit 202 also functions as an interface between the communication unit 204 and the data storage unit 206.

  In the present embodiment, the communication unit 204 of the server 200 is connected to the communication unit 142 of the robot 100 through two types of communication lines, a first communication line and a second communication line. The first communication line is a 920 MHz ISM frequency (Industrial, Scientific and Medical Band) communication line. The second communication line is a 2.4 GHz communication line. The first communication line has a frequency lower than that of the second communication line, so that radio waves are easily circulated, but the communication speed is slow.

The data storage unit 206 includes a motion storage unit 232, a map storage unit 216, and a personal data storage unit 218.
The robot 100 has a plurality of motion patterns (motions). Various motions are defined, such as shaking the hand 106, approaching the owner while meandering, and staring at the owner with the neck raised.

  The motion storage unit 232 stores a “motion file” that defines the control content of the motion. Each motion is identified by a motion ID. The motion file is also downloaded to the motion storage unit 160 of the robot 100. Which motion is to be executed may be determined by the server 200 or may be determined by the robot 100.

  Many of the motions of the robot 100 are configured as compound motions including a plurality of unit motions. For example, when the robot 100 approaches the owner, the unit motion may be expressed as a combination of a unit motion that turns toward the owner, a unit motion that approaches while raising the hand, a unit motion that approaches while shaking the body, and a unit motion that sits while raising both hands. . The combination of these four motions realizes a motion of “sitting close to the owner, raising hands halfway, and finally shaking the body”. In the motion file, the rotation angle and angular velocity of an actuator provided in the robot 100 are defined in association with the time axis. Various motions are expressed by controlling each actuator over time according to a motion file (actuator control information).

The transition time when changing from the previous unit motion to the next unit motion is called “interval”. The interval may be defined according to the time required for changing the unit motion and the content of the motion. The length of the interval is adjustable.
Hereinafter, settings related to behavior control of the robot 100 such as when and which motion to select and output adjustment of each actuator for realizing the motion are collectively referred to as “behavior characteristics”. The behavior characteristics of the robot 100 are defined by a motion selection algorithm, a motion selection probability, a motion file, and the like.

  In addition to the motion file, the motion storage unit 232 stores a motion selection table that defines a motion to be executed when various events occur. In the motion selection table, one or more motions and their selection probabilities are associated with events.

  The map storage unit 216 stores a map indicating an arrangement state of obstacles such as a chair and a table in addition to a plurality of behavior maps. The personal data storage unit 218 stores information on users, particularly owners. Specifically, master information indicating the intimacy for the user and the physical characteristics / behavioral characteristics of the user is stored. Other attribute information such as age and gender may be stored. Details of the master information will be described later with reference to FIG.

The robot system 300 (the robot 100 and the server 200) identifies the user based on the user's physical characteristics and behavioral characteristics. The robot 100 images the periphery with an omnidirectional camera. Then, the physical characteristics and behavioral characteristics of the person appearing in the image are extracted. Physical features include eye-to-eye spacing, eye-mouth and nose balance, height, preferred clothes, glasses, skin color, hair color, ear size, etc. It may be a visual feature associated with the body, or may include other features such as average body temperature, smell, voice quality, and the like. Specifically, behavioral features are features associated with behavior such as a location that the user likes, activeness of movement, and the presence or absence of smoking. For example, an owner identified as a father often does not stay at home, and often does not move on the sofa when at home, but a behavioral characteristic such as the mother is often in the kitchen and has a wide action range is extracted.
The robot system 300 according to the present embodiment extracts a plurality of parameters indicating physical characteristics from a master image described later, and identifies a user based on the master image. Hereinafter, the process of identifying the user based on the master image is referred to as “user identification process”. Details of the user identification process will be described later.

  The robot 100 has an internal parameter called familiarity for each user. When the robot 100 recognizes a behavior that favors oneself, such as picking up oneself or singing a voice, the familiarity with the user increases. The intimacy with respect to a user who is not involved in the robot 100, a user who works wildly, or a user who encounters less frequently becomes low.

The data processing unit 202 includes a position management unit 208, a map management unit 210, a recognition unit 212, an operation control unit 222, a closeness management unit 220, and an emotion management unit 244.
The position management unit 208 specifies the position coordinates of the robot 100 by the method described with reference to FIG. The position management unit 208 may track the position coordinates of the user in real time.

  The emotion management unit 244 manages various emotion parameters indicating emotions (solitude, fun, fear, etc.) of the robot 100. These emotion parameters are constantly fluctuating. The importance of the plurality of behavior maps changes according to the emotion parameter, the movement target point of the robot 100 changes according to the behavior map, and the emotion parameter changes with the movement of the robot 100 and the passage of time.

  For example, when the emotion parameter indicating loneliness is high, the emotion management unit 244 sets a large weighting coefficient for the behavior map that evaluates a safe place. When the robot 100 reaches a point where the loneliness can be eliminated in the action map, the emotion management unit 244 reduces the emotion parameter indicating loneliness. In addition, various emotion parameters are changed by a response action described later. For example, the emotion parameter indicating loneliness decreases when the owner “holds”, and the emotion parameter indicating loneliness gradually increases when the owner is not visually recognized for a long time.

The map management unit 210 changes the parameters of each coordinate by the method described with reference to FIG. 4 for a plurality of behavior maps. The map management unit 210 may select one of a plurality of behavior maps, or may perform a weighted average of z values of the plurality of behavior maps. For example, in the behavior map A, the z values at the coordinates R1 and R2 are 4 and 3, and in the behavior map B, the z values at the coordinates R1 and R2 are −1 and 3. In the case of the simple average, since the total z value of the coordinates R1 is 4-1 = 3 and the total z value of the coordinates R2 is 3 + 3 = 6, the robot 100 moves in the direction of the coordinate R2 instead of the coordinate R1.
When the action map A is emphasized five times as much as the action map B, the total z value of the coordinates R1 is 4 × 5-1 = 19 and the total z value of the coordinates R2 is 3 × 5 + 3 = 18. Head in the direction of

  The recognition unit 212 recognizes the external environment. The recognition of the external environment includes various recognitions such as recognition of weather and season based on temperature and humidity, and recognition of shade (safe area) based on light quantity and temperature. The recognition unit 156 of the robot 100 acquires various environment information by the internal sensor 128, performs primary processing on the acquired environment information, and then transfers the information to the recognition unit 212 of the server 200. The recognition unit 156 of the robot 100 extracts a moving object, in particular, an image area corresponding to a person or animal from the image, and extracts a “feature vector” indicating the physical characteristics or behavioral characteristics of the moving object from the extracted image area. To do. The robot 100 transmits the feature vector to the server 200.

The recognition unit 212 of the server 200 further includes a person recognition unit 214 and a reception recognition unit 228.
The person recognizing unit 214 compares the feature vector extracted from the image captured by the built-in camera of the robot 100 with the feature vector of the user registered in advance in the personal data storage unit 218, so (User identification process). The person recognition unit 214 includes a facial expression recognition unit 230. The facial expression recognition unit 230 estimates the user's emotion by recognizing the user's facial expression as an image.
Note that the person recognition unit 214 also performs user identification processing for moving objects other than people, such as cats and dogs that are pets.

  As described above, in the present embodiment, the recognition unit 156 of the robot 100 extracts an image area corresponding to a moving object (a person and an animal) from a captured image, and extracts a feature vector from the extracted captured image. A plurality of user feature vectors (hereinafter referred to as “master vectors”) are registered in the personal data storage unit 218 of the server 200 in advance. The master vector is a feature vector extracted based on the user's master image. The person recognition unit 214 of the server 200 identifies the user by comparing the feature vector sent from the robot 100 with the master vector.

  Hereinafter, a user whose master vector is registered in the personal data storage unit 218 is referred to as a “registered user”, and an unconfirmed user who is a target of user identification processing recognized by the camera is referred to as an “unknown user”. If the registered user A's master vector and the unknown user X's feature vector (hereinafter also referred to as “test vector”) match or are similar, it is determined that the unknown user X is the same person as the registered user A.

The response recognition unit 228 recognizes various response actions performed on the robot 100 and classifies them as pleasant / unpleasant actions. The response recognition unit 228 also classifies the response into an affirmative / negative response by recognizing the owner's response to the behavior of the robot 100.
Pleasant / unpleasant behavior is determined based on whether the user's response is pleasant or uncomfortable as a living thing. For example, being held is a pleasant action for the robot 100, and being kicked is an unpleasant action for the robot 100. An affirmative / negative reaction is discriminated based on whether the user's response acts indicate a user's pleasant feeling or an unpleasant feeling. For example, being held is an affirmative reaction indicating a user's pleasant feeling, and being kicked is a negative reaction indicating a user's unpleasant feeling.

  The operation control unit 222 of the server 200 determines the motion of the robot 100 in cooperation with the operation control unit 150 of the robot 100. The operation control unit 222 of the server 200 creates a movement target point of the robot 100 and a movement route therefor based on the action map selection by the map management unit 210. The operation control unit 222 may create a plurality of travel routes, and then select any travel route.

The motion control unit 222 selects the motion of the robot 100 from the plurality of motions in the motion storage unit 232. Each motion is associated with a selection probability for each situation. For example, a selection method is defined such that when a pleasant action is made by the owner, motion A is executed with a probability of 20%, and when the temperature becomes 30 ° C. or higher, motion B is executed with a probability of 5%. .
A movement target point and a movement route are determined in the action map, and a motion is selected by various events described later.

  The intimacy manager 220 manages intimacy for each user. As described above, the familiarity is registered in the personal data storage unit 218 as a part of personal data. When a pleasant act is detected, the intimacy manager 220 increases the intimacy for the owner. Intimacy goes down when unpleasant behavior is detected. In addition, the intimacy of the owner who has not been visible for a long time gradually decreases.

(Robot 100)
The robot 100 includes a communication unit 142, a data processing unit 136, a data storage unit 148, an internal sensor 128, and a drive mechanism 120.
The communication unit 142 corresponds to the communication device 126 (see FIG. 5), and is responsible for communication processing with the external sensor 114, the server 200, and the other robot 100. The data storage unit 148 stores various data. The data storage unit 148 corresponds to the storage device 124 (see FIG. 5). The data processing unit 136 performs various processes based on the data acquired by the communication unit 142 and the data stored in the data storage unit 148. The data processing unit 136 corresponds to the processor 122 and a computer program executed by the processor 122. The data processing unit 136 also functions as an interface for the communication unit 142, the internal sensor 128, the drive mechanism 120, and the data storage unit 148.

The data storage unit 148 includes a motion storage unit 160 that defines various motions of the robot 100.
Various motion files are downloaded from the motion storage unit 232 of the server 200 to the motion storage unit 160 of the robot 100. The motion is identified by a motion ID. The front wheel 102 is housed and seated, the hand 106 is lifted, the two front wheels 102 are reversely rotated, or the robot 100 is rotated by rotating only one of the front wheels 102. In order to express various motions, such as shaking by rotating the front wheel 102, stopping and looking back when leaving the user, the operation timing, operation time, operation direction, etc. of various actuators (drive mechanism 120) Time series are defined in the motion file.

  Various data may also be downloaded to the data storage unit 148 from the map storage unit 216 and the personal data storage unit 218.

  The internal sensor 128 includes a camera 134. The camera 134 in this embodiment is an omnidirectional camera attached to the horn 112.

The data processing unit 136 includes a recognition unit 156, an operation control unit 150, an operation detection unit 152, an imaging control unit 154, and a distance measurement unit 158.
The operation control unit 150 of the robot 100 determines the motion of the robot 100 in cooperation with the operation control unit 222 of the server 200. Some motions may be determined by the server 200, and other motions may be determined by the robot 100. Further, although the robot 100 determines the motion, the server 200 may determine the motion when the processing load on the robot 100 is high. The server 200 may determine a base motion and the robot 100 may determine an additional motion. How the server 200 and the robot 100 share the motion determination process may be designed according to the specifications of the robot system 300.

  The operation control unit 150 of the robot 100 determines the moving direction of the robot 100 together with the operation control unit 222 of the server 200. The movement based on the behavior map may be determined by the server 200, and an immediate movement such as avoiding an obstacle may be determined by the motion control unit 150 of the robot 100. The drive mechanism 120 drives the front wheel 102 in accordance with an instruction from the operation control unit 150 to direct the robot 100 toward the movement target point.

  The operation control unit 150 of the robot 100 instructs the drive mechanism 120 to execute the selected motion. The drive mechanism 120 controls each actuator according to the motion file.

  The motion control unit 150 can execute a motion of lifting both hands 106 as a gesture for holding a “cug” when a close user is nearby, and the left and right front wheels 102 when the user is tired of the “cug”. It is also possible to express a motion that hesitates to hold a child by alternately repeating reverse rotation and stopping while being accommodated. The drive mechanism 120 causes the robot 100 to express various motions by driving the front wheels 102, the hands 106, and the neck (head frame 316) according to instructions from the operation control unit 150.

  The motion detection unit 152 detects “lifting” and “holding” of the robot 100 in addition to the touch by the user. “Holding” is typically an action in which the user lifts the robot 100 with both hands on the body 104 of the robot 100. “Holding down” is typically an action in which the user puts both hands on the body 104 of the robot 100 and lowers the robot 100 onto the floor surface F. The motion detection unit 152 detects a user's touch with a touch sensor installed under the outer skin 314 of the robot 100. On the condition that the acceleration sensor detects an increase in the touched state, the motion detection unit 152 determines that “holding” has been performed. Similarly, when the descent is detected by the acceleration sensor in the touched state, or when the load on the seating surface 108 or the front wheel 102 is detected, the motion detection unit 152 determines that “holding down” has been performed. The camera 134 may capture a moving image of the outside world, and may recognize “lifting” and “holding” by recognizing the rising and falling of the robot 100 from the change in the image.

  The imaging control unit 154 controls the camera 134. The imaging control unit 154 captures an image of a subject when a hand is lifted or held, when a touch is detected, or at various timings described below.

  The distance measuring unit 158 detects a distance from a moving object (a person and a pet) as a subject by a distance measuring sensor (infrared sensor) included in the internal sensor 128. The recognition unit 156 also detects the relative angle between the robot 100 and the subject by recognizing the subject image. A captured image obtained when the robot 100 is positioned at a predetermined relative position with respect to the subject can also be used as a master image candidate (hereinafter referred to as “master candidate image”). A method for acquiring a master candidate image based on distance measurement will be described later with reference to FIG.

The recognition unit 156 of the robot 100 interprets external information obtained from the internal sensor 128. The recognition unit 156 can perform visual recognition (visual part), odor recognition (olfactory part), sound recognition (auditory part), and tactile recognition (tactile part).
The recognition unit 156 periodically images the outside world with a built-in omnidirectional camera to detect moving objects such as people and pets. The feature vector extracted from the captured image of the moving object by the recognition unit 156 is transmitted to the server 200, and the person recognition unit 214 of the server 200 identifies the user. The recognition unit 156 of the robot 100 also detects the user's smell and the user's voice. Smell and sound (voice) are classified into a plurality of types by a known method.

When a strong impact is applied to the robot 100, the recognizing unit 156 recognizes this with the built-in acceleration sensor, and the response recognizing unit 228 of the server 200 recognizes that the “violent act” has been performed by a nearby user. When the user grabs the horn 112 and lifts the robot 100, it may be recognized as a violent act. When a user who is facing the robot 100 speaks in a specific volume range and a specific frequency band, the reception recognition unit 228 of the server 200 may recognize that a “speaking action” has been performed on the server 200. Further, when a temperature of about body temperature is detected, it may be recognized that the “contact act” by the user has been performed.
In summary, the robot 100 acquires the user's action as physical information by the internal sensor 128, the motion detection unit 152 determines an action such as “lifting” or “holding”, and the response recognition unit 228 of the server 200 is pleasant. -Discomfort is determined, and the recognition unit 212 of the server 200 executes user identification processing based on the feature vector.

  The response recognition unit 228 of the server 200 recognizes various user responses to the robot 100. Some typical response actions among various response actions are associated with pleasure or discomfort, affirmation or denial. In general, most of the response actions that become pleasant acts are positive responses, and most of the response actions that become unpleasant acts are negative responses. Pleasant / unpleasant behavior is related to intimacy, and positive / negative reactions affect the behavior selection of the robot 100.

  Among a series of recognition processes including detection, analysis, and determination, the recognition unit 156 of the robot 100 performs selection and extraction of information necessary for recognition, and interpretation processing such as determination is executed by the recognition unit 212 of the server 200. . The recognition process may be performed only by the recognition unit 212 of the server 200, or may be performed only by the recognition unit 156 of the robot 100, or the above-described recognition process may be performed while both roles are shared as described above. Also good.

  The closeness management unit 220 of the server 200 changes the closeness to the user according to the response action recognized by the recognition unit 156. In principle, the intimacy for a user who has performed a pleasant act is increased, and the intimacy for a user who has performed an unpleasant action is decreased.

  The recognition unit 212 of the server 200 determines pleasure / discomfort according to the response, and the map management unit 210 changes the z value of the point where the pleasant / discomfort act is performed in the action map expressing “attachment to a place”. May be. For example, when a pleasant act is performed in the living room, the map management unit 210 may set a favorable point in the living room with a high probability. In this case, a positive feedback effect that the robot 100 likes the living room and receives a pleasant act in the living room more and more likes the living room is realized.

  The intimacy with respect to the user varies depending on what action is performed from the moving object (user).

  The robot 100 sets a high familiarity with people who often meet, people who often touch, and people who often speak. On the other hand, the intimacy of people who rarely see, people who do not touch much, violent people, and people who speak loudly is low. The robot 100 changes the familiarity for each user based on various external information detected by sensors (visual sense, tactile sense, auditory sense).

  The actual robot 100 autonomously performs complex action selection according to the action map. The robot 100 behaves while being influenced by a plurality of behavior maps based on various parameters such as loneliness, boredom, and curiosity. If the influence of the action map is excluded, or the robot 100 is in an internal state where the influence of the action map is small, in principle, the robot 100 tries to approach a person with high intimacy, and away from a person with low intimacy. And

The behavior of the robot 100 is categorized as follows according to intimacy.
(1) User with very high intimacy The robot 100 approaches the user (hereinafter referred to as “proximity action”) and performs a affection gesture that is defined in advance as a gesture that favors the person. Express strongly.
(2) User whose degree of closeness is relatively high The robot 100 performs only the proximity action.
(3) User with relatively low familiarity The robot 100 does not perform any special action.
(4) User whose degree of closeness is particularly low The robot 100 performs a withdrawal action.

  According to the above control method, the robot 100 approaches the user when a user with a high familiarity is found, and conversely moves away from the user when a user with a low familiarity is found. By such a control method, it is possible to express so-called “shrinking”. Moreover, when a visitor (user A with a low intimacy) appears, the robot 100 may move away from the visitor toward the family (user B with a high intimacy). In this case, the user B can feel that the robot 100 feels anxiety because of shyness and relies on himself / herself. By such behavioral expression, the user B is aroused by the joy of being selected and relied upon and the feeling of attachment accompanying it.

  On the other hand, when the user A who is a visitor frequently visits, speaks, and touches, the closeness of the robot 100 to the user A gradually increases, and the robot 100 does not perform a shy behavior (withdrawal behavior) to the user A . The user A can also be attached to the robot 100 by feeling that the robot 100 has become familiar with him.

  Note that the above action selection is not always executed. For example, when an internal parameter indicating the curiosity of the robot 100 is high, an action map for finding a place satisfying the curiosity is emphasized, and thus the robot 100 may not select an action influenced by intimacy. . Further, when the external sensor 114 installed at the entrance detects the return of the user, the user's welcome action may be executed with the highest priority.

FIG. 7 is an image diagram when the robot is held.
Since the robot 100 has a round, soft and comfortable body 104 and an appropriate weight, and recognizes the touch as a pleasant act, it is easy for the user to hold the robot 100. The robot 100 applies the feeling of wanting to be involved in the user identification process.

In order for the robot 100 to identify the user, information serving as a clue is necessary. For example, the thickness of eyebrows, eye size, eye shape, skin color, skin brightness, wrinkle shape, hair brightness, bangs length, the size of the eyes and nose in the entire face Physical features such as proportion, eye-to-eye spacing, etc. are clues. In the present embodiment, first, the robot 100 acquires a master image. The recognition unit 156 of the robot 100 extracts a feature vector (master vector) from the master image. The feature vector has a plurality of vector components. The feature vector component is a numerical value obtained by quantifying the various physical features described above. For example, the lateral width of the eyes is digitized in a range of 0 to 1, and these form a feature vector component. A technique for extracting a feature vector from a captured image of a person is an application of a known face recognition technique. The master vector of user A is stored as master information 224 in the personal data storage unit 218.
Hereinafter, the process of extracting a feature vector from a captured image is referred to as “vector extraction process”.

  When the robot 100 images the unknown user X, the recognition unit 156 extracts a feature vector (inspection vector) from the captured image (inspection image) of the unknown user X. The person recognition unit 214 of the server 200 determines that the unknown user X and the registered user A are the same person if the inspection vector of the unknown user X is similar to the master vector of the registered user A.

  In order to improve the identification accuracy, it is necessary to capture a high-quality master image from which a master vector can be easily extracted, more specifically, a user at a short distance. The recognition unit 156 in the present embodiment sets a captured image when the motion detection unit 152 detects the holding of the robot 100 as a master image. When the robot 100 is held, the robot 100 can capture images with high accuracy by the built-in camera 134. This is because when the robot 100 is held up, the distance between the user's face and the camera 134 built in the robot 100 is within a certain range. Instead of giving a “behavior instruction” to the user in order to capture the master image, it is possible to obtain a good master image without imposing a burden on the user by estimating the timing when the user holds the robot 100 by his / her own intention.

FIG. 8 is a data structure diagram of the master information.
The master information 224 is stored in the reception recognition unit 228. In FIG. 8, three master vectors are associated with a user with user ID = 01 (hereinafter referred to as “user (01)”). The master image is acquired not only from the front of the user (01), but also from the side faces such as the right side and the left side. For this reason, by imaging a user from a plurality of angles and a plurality of distances, a plurality of master vectors are associated with one registered user. The master vector is identified by the master ID. The master vector (01) is extracted from the master image when the face of the user (01) is imaged from the front, and the master vector (02) is extracted from the master image when the face of the user (01) is imaged from the right side. .

  In order to simplify the description, the master vector shown in FIG. 8 will be described as a five-dimensional vector having five vector components. The five vector components a to e correspond to arbitrary feature amounts such as eye-to-eye spacing and skin color. The master vector (01) includes feature amounts a1, b1, and c1 corresponding to the three vector components a to c. On the other hand, no feature quantity is set for the vector components d and e. For example, when the vector component d is a feature quantity indicating the size of the ear, the component d may not be extracted from the front master image.

  The master vector (02) includes feature amounts a2, c2, and e2 corresponding to the three vector components a, c, and e, but does not include feature amounts corresponding to the vector components b and d. The master vector (03) includes feature amounts a3, b3, d3, and e3 corresponding to the four vector components a, b, d, and e but does not include a feature amount corresponding to the vector component c. If a plurality of master images are acquired by imaging the user (01) from a plurality of directions, the physical characteristics of the user (01) can be grasped three-dimensionally.

  The person recognizing unit 214 calculates the centroid vector MB by arithmetically averaging the three master vectors of the user (01). The vector component a of the centroid vector MB is an average value of the a components (a1, a2, a3) of the three master vectors. Since only the master vector (03) has the vector component d, the vector component d of the centroid vector MB becomes the feature quantity d3 of the master vector (03). The person recognition unit 214 executes user identification processing based on the master vector or the center of gravity vector MB (described later).

Assume a situation where there is no registered user.
The motion detection unit 152 acquires a master image when being held by an unknown user A. The person recognizing unit 214 records the user ID = 01 in the master information 224 in association with the master vector (01) extracted from the master image of the unknown user A. At this time, the person recognition unit 214 also records the acquisition date and time of the master vector (01). Through the above processing, the unknown user A is registered in the master information 224 as a registered user (01). In FIG. 8, the master vector (01) is acquired on June 7, 2016.

  The motion detection unit 152 also acquires a master image when a new unknown user X holds the robot 100 after the user (01) is registered. The person recognizing unit 214 compares the master vector MX extracted from the master image of the unknown user X with the master vector (01) of the registered user (01).

(1) When Not Registered The person recognizing unit 214 determines that the unknown user X is different from the user (01) if the vector distance between the master vector MX and the master vector (01) is equal to or greater than a predetermined distance. The distance of the feature vector may be calculated as an Euclidean distance, or may be a distance calculation based on another definition such as a Chebyshev distance. The person recognizing unit 214 registers the unknown user X as a new registered user (02) in the master information 224, assigns a user ID = 02 to the user X, and assigns a master ID = 04 to the master vector MX. Through the above processing, two users (01) and (02) are registered in the master information 224.

(2) In the case of registration If the distance between the master vector MX and the master vector (01) is less than the predetermined distance, the person recognition unit 214 determines that the unknown user X and the registered user (01) are the same person. The person recognizing unit 214 sets master ID = 02 in the master vector MX and associates it with the user (01). The user (01) has two master vectors, and information for identifying the user (01) is enriched.

  Since a high-quality master vector is obtained from the master image, it is possible to easily determine whether the user holding the robot 100 is a registered user or a person by comparing the master vectors.

  When there are a plurality of registered users, the master vector of each registered user is a comparison target. When one registered user has two or more master vectors, the registered user's center of gravity vector and the unknown user's master vector are compared.

FIG. 9 is a first schematic diagram for explaining the user identification method.
9 and 10, in order to illustrate the principle of the user identification process, two vector components a and b among the vector components included in the feature vector will be described. The processing method is the same when there are three or more vector components.
In FIG. 9, one master vector MA and one master vector MB are extracted for each of registered user A and registered user B. Master vector MA = (a1, b1), master vector MB = (a2, b2). In such a situation, it is assumed that the robot 100 has acquired a captured image (inspection image) of an unknown user X walking from the front. The recognizing unit 156 determines whether the unknown user X shown in the inspection image is the registered user A or B. Since it is not held, the feature vector (inspection vector) obtained from the inspection image of the unknown user X usually does not have the accuracy as the master vector.

  The recognition unit 156 first extracts an inspection vector DX = (ax, bx) from the inspection image of the unknown user X. The communication unit 142 of the robot 100 transmits the inspection vector DX to the communication unit 204 of the server 200. The person recognition unit 214 of the server 200 calculates ra, which is the distance between the inspection vector DX and the master vector MA, and rb, which is the distance between the inspection vector DX and the master vector MB.

  When an arbitrary threshold value rm is set, if rb <ra and rb <rm, the person recognition unit 214 determines that the unknown user X is the registered user B. On the other hand, if ra <rb and ra <rm, the person recognition unit 214 determines that the unknown user X is the registered user A. On the other hand, when ra> rm and rb> rm, the unknown user X does not correspond to any of the registered users A and B. When it is determined that the unknown user X is the registered user A having a high degree of closeness, the operation control unit 150 may select an intimate action such as rushing to the unknown user X. On the other hand, when it is determined that the unknown user X is the registered user B having a low familiarity, the operation control unit 150 may select a repelling action such as escaping from the unknown user X.

  When the unknown user X cannot be identified, the person recognizing unit 214 may notify the robot 100 that it has not been confirmed, and the motion control unit 150 of the robot 100 may select a motion that holds the unknown user X. Specifically, motions such as approaching the unknown user X, raising the hand 106, and sitting in front of the unknown user X can be considered.

  When the unknown user X holds the robot 100 and the motion detection unit 152 detects “pickup”, the imaging control unit 154 controls the camera 134 to image the unknown user X from a short distance. The recognition unit 156 extracts the master vector MX from the master image of the unknown user obtained at the time of carrying. The person recognition unit 214 of the server 200 may execute the user identification process again by comparing the master vector MX of the unknown user X with the existing master vectors MA and MB. Since it is a comparison between master vectors, it is possible to identify with higher accuracy. When the unknown user X is determined to be a different person from the registered users A and B by comparing the master vectors, the person recognizing unit 214 sets the unknown user X as the third registered user in the master information 224 together with the master vector MX. sign up.

  When it is determined that the unknown user X is the registered user A, the person recognizing unit 214 may register the inspection vector obtained from the inspection image of the unknown user X as a new master vector for the registered user A.

FIG. 10 is a second schematic diagram for explaining the user identification method.
In FIG. 10, a plurality of master vectors are extracted for each of registered user A and registered user B. The person recognition unit 214 calculates the centroid vector MB (A) of the registered user A and the centroid vector MB (B) of the registered user B. In such a situation, it is assumed that the robot 100 acquires a captured image (inspection image) of an unknown user X walking from the front. The recognizing unit 156 determines whether the unknown user X shown in the inspection image is the registered user A or B.

  The recognition unit 156 first extracts an inspection vector DX = (ax, bx) from the inspection image of the unknown user X. The communication unit 142 of the robot 100 transmits the inspection vector DX to the communication unit 204 of the server 200. The person recognition unit 214 of the server 200 calculates ra, which is the distance between the inspection vector DX and the centroid vector MB (A), and rb, which is the distance between the inspection vector DX and the centroid vector MB (B).

  When an arbitrary threshold value rm is set, if rb <ra and rb <rm, the person recognition unit 214 determines that the unknown user X is the registered user B. On the other hand, if ra <rb and ra <rm, the person recognition unit 214 determines that the unknown user X is the registered user A. On the other hand, when ra> rm and rb> rm, the unknown user X does not correspond to any of the registered users A and B.

FIG. 11 is a flowchart showing a master vector extraction process.
When the motion detection unit 152 of the robot 100 detects the holding of the robot 100, the vector extraction process of FIG. 11 is executed. The motion control unit 150 executes a predetermined guidance motion when the holding is detected (S10). The guided motion is a motion that is defined in advance to attract the user's attention. Specifically, non-verbal motions such as shaking the hand 106, shaking the body 104, turning the head frame 316 toward the user, shaking the head frame 316 up and down or left and right are assumed. Guided motion is not limited to mechanical motion. The operation control unit 150 displays an image of “pupil” on the eye 110 using an organic EL element. The operation control unit 150 may instruct image control such as widening the pupil image to open the pupil, shaking the pupil, or winking.

  The user's face is directed to the robot 100 by attracting the user's attention with the guided motion. Also, by preparing various induced motions, various master vectors corresponding to various facial expressions can be extracted by extracting various facial expressions of the user. For example, a feature quantity peculiar to a smile such as a laughing candy or a dimple can be included as a vector component of the master vector.

  After executing the guidance motion, the imaging control unit 154 controls the camera 134 to image the user (S12). The captured image at this time is a “master candidate image”. By capturing the user at the timing when the user looks at the robot 100 by the guided motion, a high-quality master candidate image that easily recognizes the user's face can be acquired.

  The recognition unit 156 determines the quality of the master candidate image (S14). Hereinafter, the quality determination of the master candidate image is referred to as “quality inspection”. A master candidate image that passes the quality inspection is set as a master image. If the quality inspection fails (N in S14), the process returns to S10, and the master candidate image is reacquired. At this time, another type of guided motion may be executed. For quality inspection, a plurality of evaluation items are set in advance for the size, light quantity, facial expression, etc. of the user's face. For example, the quality inspection fails when the user is closed, when the master candidate image is too dark or too bright, or when the master candidate image is not in focus. What evaluation items are set for quality inspection is arbitrary.

  The recognition unit 156 employs the master candidate image that has passed the quality inspection as a formal master image (Y in S14). The recognition unit 156 extracts a master vector from the master image (S16). The communication unit 142 transmits the master vector to the server 200 (S18).

  The person recognizing unit 214 compares the newly obtained master vector with the master vector already registered in the master information 224 (S20). When the newly obtained master vector is close to the already registered master vector (Y in S20), the master vector is additionally registered (S22). For example, when a master vector similar to the master vector (01) of the user (01) is obtained, the new master vector is also associated with the user (01). When it is not close to any of the registered master vectors (N in S20), a new user ID and master ID are assigned and a new master vector is registered (S24).

  In S20, the registered master vector and the newly extracted master vector may be compared, or the registered centroid vector and the newly extracted master vector may be compared as described with reference to FIG.

  The master vector extraction process is not limited to a cuddle, and may be executed when the user touches the robot 100. When the user touches the robot 100, since the user is near the robot 100, there is a possibility that a high-quality master image can be obtained.

  When the motion detection unit 152 detects that the robot 100 is held down, the recognition unit 156 also executes a master vector extraction process. The motion detection unit 152 continuously captures images of the user when the holding down is detected. The recognition unit 156 sequentially inspects the plurality of master candidate images obtained at this time, and extracts a plurality of master vectors. When held down, physical features such as chin, waist and legs can be imaged at close range.

  A master vector obtained when the user holds the robot 100 is referred to as a “first master vector”, and a master vector obtained when the user lowers or after the robot 100 is lowered is referred to as a “second master vector”. . After obtaining the first master vector of the user (01), the recognizing unit 156 also extracts one or more second master vectors from the master image when held. When the first master vector with high accuracy is obtained in this manner, the master vector of the user (01) can be enriched by acquiring the second master vector even when the first master vector is held. The “second master vector” referred to here includes master vectors obtained from various distances and angles including the user's rear view even after the robot 100 is held down. As shown in the master information 224, the first master vector and the second master vector are associated with each other for one user.

FIG. 12 is a schematic diagram showing a user image tracking method.
Even after the robot 100 is lowered to the floor surface F, the imaging control unit 154 further tracks the user by the camera 134 (global camera). The celestial sphere imaging range 418 shown in FIG. 12 is an imaging range by the omnidirectional camera. The omnidirectional camera can image the entire upper hemisphere of the robot 100 at one time. The recognition unit 156 of the robot 100 tracks the user in the celestial sphere imaging range 418 for a predetermined period, for example, about 10 seconds after extracting the first master vector. The imaging control unit 154 captures a user's master image from various angles and various distances during tracking. For example, the length of the hair, the thinness of the waist, etc. are information that cannot be obtained without leaving the user. The recognition unit 156 enhances the master vector by extracting various second master vectors from the master image obtained during tracking. These second master vectors are managed in association with the first master vector. In addition to tracking the user on the image in the celestial sphere imaging range 418, the operation control unit 150 may execute tracking behavior such as following the user or moving around the user. The imaging control unit 154 may enrich the master vector by imaging the user even during the tracking action. The tracking control may be instructed by the operation control unit 150, or the operation control unit 222 of the server 200 may instruct the operation control unit 150.

FIG. 13 is a schematic diagram for explaining a method of remotely extracting a master vector.
The imaging control unit 154 captures a master candidate image when the user is positioned at a predetermined relative position with respect to the robot 100 as well as a cuddle and a touch. The relative point here includes both the distance and the relative angle between the user and the robot 100. The distance measuring unit 158 periodically measures one or more users recognized in the celestial sphere imaging range 418. In FIG. 13, the robot 100 is located at a relative point of a horizontal angle a with respect to the front direction of the user, an elevation angle b with respect to the position of the user's face, and a distance r from the user. The recognition unit 156 determines the orientation of the user's body through image recognition. The imaging control unit 154 captures a master candidate image when the distance, the horizontal angle, and the elevation angle are within a predetermined range (hereinafter referred to as “master shot range”). The recognizing unit 156 inspects the quality of the master candidate image, and extracts the master vector if it is acceptable.

  The distance measuring unit 158 has a plurality of master shot ranges set in advance. The ranging unit 158 notifies the imaging control unit 154 every time the user enters the master shot range, and the imaging control unit 154 acquires a master candidate image. For example, when the master shot ranges R1 to R3 are defined and the new user C enters the master shot range R1, a master candidate image corresponding to the master shot range R1 is acquired. In this way, master vectors corresponding to the master shot ranges R1 to R3 are extracted. By capturing the user C from a plurality of master shot ranges, in other words, from a plurality of relative points in a multifaceted manner and acquiring master vectors from multiple directions, the physical characteristics of the user can be grasped three-dimensionally.

  Since the user can be photographed from a close range when touching, such as a hug or a touch, it is easy to obtain good quality information about the user's face. On the other hand, even when the baby is not held or touched, when the distance measuring unit 158 detects a user at a close distance, the imaging control unit 154 may acquire the master candidate image. For example, even if a small child has resistance to a cuddle or touch, a master vector can be extracted when approaching with interest. Also, in order to obtain information on the user's hair length and body shape, the robot 100 must be separated from the user to some extent. By setting various master shot ranges, various master vectors including not only the user's face but also the body shape can be acquired.

  The first master vector is a feature vector useful for identifying the user because it is based on a master image obtained by capturing the user from a close range. On the other hand, the second master vector often does not show the user's physical characteristics as clearly as the first master vector. Therefore, the imaging control unit 154 enters the tracking mode in response to the extraction of the first master vector (A) from the master image A at the time of holding or touching. The tracking mode may be continued for a predetermined time. For example, the imaging control unit 154 acquires the master image B1 when detecting the holding down. A second master vector (B1) is extracted from the master image B1, and is associated with the first master vector (A) extracted earlier. The tracking mode continues even after holding the camera, and when the user enters the master shot range, the master image B2 is further acquired. The second master vector (B2) obtained from the master image B2 is also associated with the first master vector (A) that triggered the tracking mode. In this manner, various second master vectors obtained thereafter are associated with the first master vector (A) that can easily identify the user. Even if it is the 2nd master vector whose feature is hard to appear like "back view", the master vector group corresponding to one user can be enriched by matching with the 1st master vector used as the opportunity. .

The robot system 300 including the robot 100 and the robot 100 has been described above based on the embodiment.
In the face recognition technology, a master image is often acquired after giving a language instruction such as “please look to the front” or “look at the camera” to the user. Such language instructions such as voice and characters tend to be a burden on the user. In addition, the language instruction for acquiring the master image is not desirable in that it makes the user aware of the abiotic nature of the robot 100. The robot 100 in the present embodiment can casually acquire a master image at the timing when the user holds the robot 100. The robot 100 has a shape that a human wants to touch, such as small, soft, light, and round. Instead of forcing the user to take any action, it is possible to acquire a high-quality master image by capturing the timing when the user naturally “holds”. The master image can be casually acquired by utilizing the characteristic of the robot 100 that stimulates the feeling of wanting to hold and touch.

  The robot 100 further alerts the user by a non-language guidance motion such as flapping the hand 106. Since it is a method that draws attention to the user through non-verbal communication, the user is less likely to have a forced sense.

  When the robot 100 is held, the robot 100 images the user and obtains a first master vector. Furthermore, the robot 100 can also acquire a plurality of second master vectors by capturing a master image when the robot 100 is held down or after being held down. By accumulating the second master vector at the timing when the first master vector is extracted, it becomes easier to grasp the physical characteristics of the user from various aspects.

  According to this embodiment, it becomes easy to establish a precise discrimination criterion for user identification processing based on a high quality and a large number of master images. The user identification process is a premise for recognition of response actions and calculation of intimacy. By identifying the user with high accuracy based on the master vector, the robot 100 can change the behavior characteristics according to the user.

  In addition, this invention is not limited to the said embodiment and modification, A component can be deform | transformed and embodied in the range which does not deviate from a summary. Various inventions may be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments and modifications. Moreover, you may delete some components from all the components shown by the said embodiment and modification.

  Although it has been described that the robot system 300 is configured by one robot 100, one server 200, and a plurality of external sensors 114, some of the functions of the robot 100 may be realized by the server 200. May be assigned to the robot 100. One server 200 may control a plurality of robots 100, or a plurality of servers 200 may cooperate to control one or more robots 100.

  A third device other than the robot 100 and the server 200 may bear a part of the function. A set of each function of the robot 100 described in FIG. 6 and each function of the server 200 can be grasped as one “robot” in a global manner. How to allocate a plurality of functions necessary for realizing the present invention to one or a plurality of hardware depends on the processing capability of each hardware, the specifications required for the robot system 300, and the like. It only has to be decided.

  As described above, “robot in the narrow sense” refers to the robot 100 that does not include the server 200, but “robot in the broad sense” refers to the robot system 300. Many of the functions of the server 200 may be integrated into the robot 100 in the future.

  The master vector may include a feature quantity other than the feature quantity extracted from the master image as a vector component. For example, the odor detected by the odor sensor, the voice quality detected by the microphone, and the body temperature detected by the temperature sensor may be included as vector components. In particular, when holding, it is easy to detect the user's odor or body temperature with high accuracy. The master image may not be a still image but a moving image (hereinafter referred to as “master moving image”). The recognizing unit 156 may extract wrinkles such as how the user walks and poor yusu from the master moving image, and may include such feature information in the master vector component.

  The camera 134 in the present embodiment is an omnidirectional camera, but the camera 134 may be a normal camera. The camera 134 may be built in the horn 112 or may be built in the eye 110. Moreover, both an omnidirectional camera and a normal camera may be incorporated.

  In FIG. 8, the centroid vector is formed by arithmetic averaging of a plurality of master vectors. However, as a modification, the median value of the plurality of master vectors may be used as the centroid vector component. For example, if a1 <a2 <a3 in FIG. 8, the a component of the centroid vector may be a2.

  In the quality inspection of the master candidate image, a plurality of evaluation items may be weighted. Possible evaluation items include (E1) facing the front, (E2) whether the amount of light is appropriate, (E3) whether the eyes are open, and the like. The master candidate image may be scored for each evaluation item, and the quality of the master candidate image may be determined by weighted averaging of the item points. For example, if the coefficients of p1, p2, and p3 are set in E1 to E3 (p1 + p2 + p3 = 1) and the item values of E1 to E3 are s1, s2, and s3, the total point is p1 · s1 + p2 · s2 + p3 · s3. . If the total point is equal to or greater than a predetermined threshold, the master image is adopted and the master vector is registered in the master information 224.

  The guided motion may be executed other than when it is held. When the distance between the robot 100 and the user is within a predetermined range, the operation control unit 150 may execute a guidance motion. For example, the master candidate image may be captured after performing the guidance motion when the user is in the master shot range shown in FIG. In summary, while the robot 100 is interacting with the user in various ways, the robot 100 extracts a master vector without missing a “shutter opportunity” effective in grasping the physical and behavioral characteristics of the user. A variety of high-quality master vectors can be collected without making the user conscious. It is also possible to actively create “shutter opportunities” through non-linguistic actions using guided motion.

  The guided motion in this embodiment is a kind of non-language communication. The non-verbal motion here may include a voice that does not make sense as a language, such as an animal cry. As a modification, the robot 100 may call the user in a simple language.

  When the robot 100 is embraced by the user, the robot 100 may acquire three face images of a front face, a right face, and a left face as a master image. The recognition unit 156 may determine from which direction the user is looking by recognizing the user's ear or nose. As one of the internal sensors 128, the robot 100 may be equipped with a gyroscope. The recognizing unit 156 may detect the tilt direction of the robot 100 when held by the user by the gyroscope, and thereby determine from which direction the user is looking.

  In addition to the method shown in FIG. 9 and FIG. 10 (hereinafter referred to as “distance determination method”), user identification may be executed by Mahalanobis distance. In FIG. 10, when a plurality of master vectors are obtained, the person recognizing unit 214 considers the variance value and obtains a Mahalanobis' distance between the inspection vector DX and the master vector group of the user A. . Similarly, the person recognizing unit 214 obtains a Mahalanobis distance between the inspection vector DX and the master vector group of the user B. Then, based on the Mahalanobis distance for each group, it may be determined whether the unknown user X is the user A or the user B by a known discriminant analysis method (hereinafter referred to as “Mahalanobis determination method”). Called).

  The person recognizing unit 214 may form a neural network in which each user's master vector group is used as teacher data, and execute user identification based on the fit between the inspection vector of the unknown user X and the master vector. (Hereafter referred to as “Neural Network Judgment Method”).

  The person recognition unit 214 may identify the user by combining a plurality of distance determination methods, Mahalanobis determination methods, and neural network determination methods. Further, not only the comparison between the inspection vector and the master vector, but also when comparing the master vector of the registered user and the master vector of the unknown user, the similarity determination may be performed by the various methods described above.

  In the present embodiment, the imaging control unit 154 has been described as acquiring a master image at a timing such as holding or touching. As a modification, the imaging control unit 154 may periodically image the user, and the recognition unit 156 may select a large number of captured images as master candidate images. For example, the user is imaged at a timing of once every 10 seconds, and the recognition unit 156 performs quality inspection using this as a master candidate image. The recognition unit 156 extracts a master vector from the passed master image. According to such a method, a master vector can be extracted from a high-quality captured image obtained by chance.

  The person recognizing unit 214 may delete the old master vector from the personal data storage unit 218 when the number of master vectors reaches a predetermined number or more. Or you may delete the old master vector, for example, the master vector acquired 3 years or more ago. According to such a control method, not only can the amount of data stored in the personal data storage unit 218 be suppressed, but also a change in physical characteristics associated with the user's aging and growth can be accommodated.

  In this embodiment, the feature vector is extracted by the robot 100 and the user 200 is identified by comparing the feature vector in the server 200. As a modified example, the robot 100 may send a captured image to the server 200 and the person recognition unit 214 of the server 200 may execute both feature vector extraction and user identification. Alternatively, the robot 100 may execute the user identification process in the recognition unit 156 without depending on the processing capability of the server 200. In this case, the robot 100 may manage the master vector of each user in the data storage unit 148 of the robot 100.

  The physical / behavioral characteristics of the user may be extracted by a sensor incorporated in the external sensor 114 without being limited to the camera and various sensors incorporated in the robot 100. The external sensor 114 captures the user when the user is nearby and transmits the captured image to the robot 100. The recognition unit 156 of the robot 100 may perform quality inspection and component extraction of the captured image.

  In the present embodiment, the personal data storage unit 218 has been described as storing a master vector instead of a master image. However, both the master image and the master vector may be stored.

  The robot system 300 does not need to have a user identification function based on a master vector from the time of factory shipment. For example, the robot system 300 may perform user identification by a clustering technique using deep learning. After the robot system 300 is shipped, the function enhancement of the robot system 300 may be realized by downloading an action control program that realizes a user identification function based on a master vector via a communication network.

  As described above, the recognition unit 156 selects a captured image when the robot 100 is held up as a master candidate image. The recognition unit 156 may detect the position and orientation of the user's face using a temperature sensor such as a thermal camera, or may detect the distance between the user and the robot 100 using a distance measuring sensor. The recognizing unit 156 may select, as a master candidate image, a captured image when a predetermined specific condition is satisfied for both or one of temperature information from the thermal camera and distance information from the distance measuring sensor. For example, the recognition unit 156 can confirm that the user is facing the robot 100 using a thermal camera, and use a captured image when the distance between the user and the robot 100 is within a predetermined range using a distance measuring sensor as a master candidate image. You may choose. According to such a control method, it becomes easy to carefully select an appropriate master candidate image based on a plurality of types of sensors.

  The camera mounted on the robot 100 may be an omnidirectional camera. When the robot 100 is held by the user from the back side, in other words, even when the robot 100 and the user are not facing each other, the robot 100 can photograph the rear user with the omnidirectional camera. Therefore, even when the robot 100 is held from the back side, the recognition unit 156 can acquire an appropriate master candidate image, so that the opportunity for acquiring the master candidate image can be expanded. Alternatively, the recognition unit 156 may specify a master candidate image on the condition that the robot 100 and the user are facing each other.

  When a plurality of users are shown in the captured image when held, the recognition unit 156 may not select the captured image as a master candidate image.

  When the registered user P1 is detected in the captured image including a plurality of users, the recognition unit 156 extracts the feature vector of the registered user P1 from the captured image, and additionally registers it as a new master vector of the registered user P1. It is good. In other words, when a registered user is not detected in a captured image including a plurality of users, in other words, when a captured image including only a plurality of unknown users is obtained, the recognition unit 156 has a predetermined value such as facing the front. A feature vector may be extracted for the unknown user P2 that satisfies the condition, and this may be newly registered as a master vector of the unknown user P2.

  The recognizing unit 156 may also acquire a user's voice (voice information) using a microphone during shooting. The master vector is not limited to image information, and may include a feature vector based on audio information. Similarly, the recognition unit 156 may acquire a user's odor (olfactory information) using a odor sensor when photographing. Thus, as information for specifying a registered user, various sensor information such as audio information and olfactory information may be included in addition to image information.

  The robot 100 may include a plurality of microphones. At the time of voice registration, the voice may be detected only from a microphone corresponding to the direction in which the user exists, for example, a microphone attached in front of the robot 100. The recognition unit 156 may disable other microphones. According to such a control method, environmental sounds other than those from the user are hardly captured in the master vector. It is desirable that the microphone, particularly the microphone attached in front, has directivity.

  The recognition unit 156 may acquire the user's voice information as a part of the master vector on the condition that the voice information is obtained when motion is detected on the user's lip reflected in the captured image. When the unknown user is detected, the recognition unit 156 may execute a motion that approaches the unknown user and holds the user.

Claims (11)

An imaging control unit for controlling the camera;
A recognition unit that discriminates a moving object based on a feature vector extracted from a captured image of the moving object;
According to the determination result, an operation selection unit that selects the motion of the robot,
A drive mechanism for executing the motion selected by the operation selection unit;
An operation detection unit for detecting lifting of the robot by a moving object,
The recognition unit sets a captured image when a robot is held on the moving object as a master image, and sets a discrimination criterion for the moving object based on a feature vector extracted from the master image. An autonomous behavioral robot.   The autonomous behavior type robot according to claim 1, wherein the recognition unit sets a plurality of captured images obtained by capturing the moving object from a plurality of angles as a master image. The operation selection unit causes the drive mechanism to execute a predetermined guidance motion,
The autonomous behavior type robot according to claim 1, wherein the imaging control unit captures a master image of the moving object in response to execution of the guidance motion.   The autonomous behavior robot according to claim 3, wherein the guided motion is a non-language motion.   The autonomous behavior type robot according to claim 1, wherein the recognition unit further sets a captured image when the robot is held on the moving object as a master image. The operation selection unit selects a motion for tracking the moving object after the robot is held by the moving object,
The autonomous behavior robot according to claim 1, wherein the imaging control unit captures a master image of the moving object during tracking. The imaging control unit tracks the moving object even after the robot is held by the moving object, captures a second master image of the moving object at a predetermined timing,
The recognition unit extracts a plurality of feature vectors related to the moving object by associating the first master image acquired when the robot is held with the moving object and the second master image. The autonomous behavior type robot according to claim 1, wherein the robot is an autonomous behavior type robot. An imaging control unit for controlling the camera;
A recognition unit that discriminates a moving object based on a feature vector extracted from a captured image of the moving object;
According to the determination result, an operation selection unit that selects the motion of the robot,
A drive mechanism for executing the motion selected by the operation selection unit;
An operation detection unit for detecting a touch by a moving object,
The recognition unit sets a captured image when the touch is detected as a master image, and sets a moving object discrimination criterion based on a feature vector extracted from the master image. robot. An imaging control unit for controlling the camera;
A recognition unit that discriminates a moving object based on a feature vector extracted from a captured image of the moving object;
According to the determination result, an operation selection unit that selects the motion of the robot,
A drive mechanism for executing the motion selected by the operation selection unit,
The recognition unit sets, as a master image, an image captured when the moving object is positioned at a predetermined relative point with respect to the robot, and determines a moving object based on a feature vector extracted from the master image Autonomous behavior robot characterized by setting. A computer program for object recognition by a robot,
A function for setting a captured image of a moving object when the robot is held by the moving object as a master image;
A function for setting a moving object discrimination criterion based on a feature vector extracted from the master image;
A behavior control program characterized by causing a robot to exhibit a function of discriminating a moving object based on a feature vector extracted from a captured image of the moving object. A computer program for object recognition by a robot,
A function for setting a captured image of a moving object when the robot touches the moving object as a master image;
A function for setting a moving object discrimination criterion based on a feature vector extracted from the master image;
A behavior control program characterized by causing a robot to exhibit a function of discriminating a moving object based on a feature vector extracted from a captured image of the moving object.

Images