JP6671577B2 - An autonomous robot that identifies people - Google Patents

Description

  The present invention relates to a robot that autonomously selects an action according to an internal state or an external environment.

  Humans acquire various information from the external environment through sensory organs and select actions. There are conscious action choices and unconscious action choices. Repetitive behavior eventually becomes unconscious, and new behavior stays in the conscious domain.

  We believe that we have the will to choose our actions freely, that is, free will. Humans have feelings of love and hatred for others because they believe that others have free will. A person who has free will, at least a person who can be assumed to have free will, will also heal the loneliness of a person.

  Humans keep pets because they provide healing rather than whether they help. Pets can be good companions to humans because they are more or less free-willing.

  On the other hand, many people give up pets for various reasons, such as not having enough time to care for pets, lack of living environment for pets, allergies, and difficult bereavement. If there is a robot that plays the role of a pet, healing may be provided to a person who cannot keep a pet, as provided by the pet (see Patent Document 1).

JP 2000-323219 A

In recent years, robot technology has been advancing rapidly, but has not yet achieved a presence as a companion like a pet. This is because the robot does not seem to have free will. By observing the behavior of a pet that only seems to have a free will, a human feels the free will of the pet, sympathizes with the pet, and is healed by the pet.
Therefore, it is considered that if the robot can express human / biological behavior, particularly if the robot changes its behavior depending on the partner, the empathy with the robot can be greatly increased.

  In order to realize the above behavioral characteristics, the robot must have the ability to identify a human. In the face authentication technology, a captured image to be a reference of a known person A (hereinafter, referred to as a “master image”) is compared with a captured image of an unconfirmed person X (hereinafter, referred to as an “inspection image”). It is determined whether the person A and the person X are the same person. When acquiring a master image, the system often instructs a person to be a subject about a posture and an expression at the time of imaging.

  Although a high-quality master image is required to improve the accuracy of identifying a person, it is not preferable to place an excessive burden on the user to obtain the master image. In particular, forcing a user to perform a behavior in a robot that should realize biological behavior characteristics may cause the user to feel the non-living nature of the robot.

  The present invention has been completed based on the above recognition, and a main object of the present invention is to provide a technique for improving the identification ability of a robot while suppressing a burden on a user.

An autonomous behavior robot according to an aspect of the present invention, an imaging control unit that controls a camera, a recognition unit that determines a moving object based on a feature vector extracted from a captured image of the moving object, An operation selection unit that selects a motion of the robot, a drive mechanism that executes the motion selected by the operation selection unit, and an operation detection unit that detects the holding of the robot by the moving object.
The recognition unit sets a captured image when the robot is held by the moving object as a master image, and sets a criterion for determining the moving object based on a feature vector extracted from the master image.

An autonomous behavior robot according to another aspect of the present invention includes an imaging control unit that controls a camera, a recognition unit that determines a moving object based on a feature vector extracted from a captured image of the moving object, , A motion selecting unit for selecting a motion of the robot, a driving mechanism for executing the motion selected by the motion selecting unit, and a motion detecting unit for detecting a touch by the moving object.
The recognizing unit sets a captured image when a touch is detected as a master image, and sets criteria for determining a moving object based on a feature vector extracted from the master image.

An autonomous behavior robot according to another aspect of the present invention includes an imaging control unit that controls a camera, a recognition unit that determines a moving object based on a feature vector extracted from a captured image of the moving object, , A motion selecting unit for selecting a motion of the robot, and a driving mechanism for executing the motion selected by the motion selecting unit.
The recognition unit sets an image captured when the moving object is located at a predetermined relative point with respect to the robot as a master image, and sets a determination criterion for the moving object based on a feature vector extracted from the master image. I do.

The behavior control program according to an aspect of the present invention is a computer program for object recognition by a robot.
This program has a function of setting a captured image of a moving object when the robot is held by the moving object as a master image, and a function of setting a criterion for determining a moving object based on a feature vector extracted from the master image. And a function of determining a moving object based on a feature vector extracted from a captured image of the moving object.

An action control program according to another aspect of the present invention is a computer program for object recognition by a robot.
A function to set a captured image of the moving object when the robot is touched to the moving object as a master image, a function to set a criterion of the moving object based on a feature vector extracted from the master image, and an image of the moving object The function of discriminating a moving object based on a feature vector extracted from an image is provided to the robot.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes easy to raise the identification capability of a robot, suppressing the burden on a user.

It is a front external view of a robot. It is a side appearance view of a robot. It is sectional drawing which represents the structure of a robot schematically. It is a block diagram of a robot system. It is a conceptual diagram of an emotion map. FIG. 2 is a hardware configuration diagram of a robot. It is a functional block diagram of a robot system. It is an image figure when carrying a robot. It is a data structure figure of master information. FIG. 4 is a first schematic diagram for explaining a user identification method. FIG. 9 is a second schematic diagram for explaining a user identification method. It is a flowchart which shows the extraction process of a master vector. It is a schematic diagram which shows the image tracking method of a user. FIG. 6 is a schematic diagram for explaining a method of extracting a master vector from a remote place.

FIG. 1A is a front external view of the robot 100. FIG. 1B is a side external view of the robot 100.
The robot 100 according to the present embodiment is an autonomous behavior type robot that determines a behavior or a gesture based on an external environment and an internal state. The external environment is recognized by various sensors such as a camera and a thermosensor. The internal state is quantified as various parameters expressing the emotion of the robot 100. These will be described later.

  As a general rule, the robot 100 sets the home of the owner's home as the range of action. Hereinafter, a person related to the robot 100 is referred to as a “user”, and a user who is a member of the home to which the robot 100 belongs is referred to as an “owner”. The “moving object” to be identified by the robot 100 includes both a human and a pet, but in the present embodiment, a description will be given of a human (user).

  The body 104 of the robot 100 has a rounded shape as a whole, and includes an outer skin made of a soft and elastic material such as urethane, rubber, resin, or fiber. The robot 100 may be dressed. By making the body 104 round, soft, and comfortable, the robot 100 provides the user with a sense of security as well as a comfortable tactile sensation.

  The robot 100 has a total weight of 15 kg or less, preferably 10 kg or less, more preferably 5 kg or less. By 13 months of age, the majority of babies will start walking alone. The average weight of a 13-month-old baby is over 9 kilograms for boys and less than 9 kilograms for girls. Therefore, if the total weight of the robot 100 is 10 kg or less, the user can hold the robot 100 with substantially the same effort as holding a baby who cannot walk alone. The average weight of babies less than two months old is less than 5 kilograms for both men and women. Therefore, if the total weight of the robot 100 is 5 kg or less, the user can hold the robot 100 with the same effort as holding an infant.

  By various attributes such as moderate weight, roundness, softness, and good touch, the effect that the user can easily hold the robot 100 and want to hold the robot 100 is realized. For the same reason, it is desirable that the height of the robot 100 is 1.2 meters or less, preferably 0.7 meters or less. An important concept for the robot 100 in the present embodiment is that it can be held.

  The robot 100 includes three wheels for traveling on three wheels. As shown, it includes a pair of front wheels 102 (left wheel 102a, right wheel 102b) and one rear wheel 103. The front wheel 102 is a driving wheel, and the rear wheel 103 is a driven wheel. Although the front wheel 102 does not have a steering mechanism, the rotation speed and the rotation direction can be individually controlled. The rear wheel 103 is formed of a so-called omni wheel, and is rotatable to move the robot 100 forward, backward, left, and right. By increasing the number of rotations of the right wheel 102b compared to the left wheel 102a, the robot 100 can turn left or rotate counterclockwise. By making the rotation speed of the left wheel 102a higher than that of the right wheel 102b, the robot 100 can turn right or rotate clockwise.

  The front wheel 102 and the rear wheel 103 can be completely housed in the body 104 by a driving mechanism (rotating mechanism, link mechanism). During running, most of the wheels are hidden by the body 104. However, when the wheels are completely stored in the body 104, the robot 100 cannot move. That is, the body 104 descends as the wheels are stored, and sits on the floor F. In this seated state, the flat seating surface 108 (ground contact bottom surface) formed on the bottom of the body 104 abuts on the floor surface F.

  The robot 100 has two hands 106. The hand 106 does not have a function of holding an object. The hand 106 can perform simple operations such as raising, shaking, and vibrating. The two hands 106 can also be individually controlled.

  An image can be displayed on the eye 110 by a liquid crystal element or an organic EL element. The robot 100 is equipped with various sensors such as a microphone array, an ultrasonic sensor, an odor sensor, a distance measurement sensor, and an acceleration sensor capable of specifying a sound source direction. Further, the robot 100 has a built-in speaker and can emit a simple voice. A capacitive touch sensor is installed on the body 104 of the robot 100. With the touch sensor, the robot 100 can detect a user's touch.

  A horn 112 is attached to the head of the robot 100. As described above, since the robot 100 is lightweight, the user can lift the robot 100 by grasping the horn 112. A celestial sphere camera is attached to the horn 112, and can image the entire upper region of the robot 100 at a time.

FIG. 2 is a cross-sectional view schematically illustrating the structure of the robot 100.
As shown in FIG. 2, the body 104 of the robot 100 includes a base frame 308, a main body frame 310, a pair of resin wheel covers 312, and an outer skin 314. The base frame 308 is made of metal, constitutes the axis of the body 104, and supports an internal mechanism. The base frame 308 is configured by vertically connecting an upper plate 332 and a lower plate 334 with a plurality of side plates 336. A sufficient space is provided between the plurality of side plates 336 so as to allow ventilation. The battery 118, the control circuit 342, and various actuators are housed inside the base frame 308.

  The main body frame 310 is made of a resin material, and includes a head frame 316 and a body frame 318. The head frame 316 has a hollow hemisphere shape and forms a head skeleton of the robot 100. The torso frame 318 has a stepped cylindrical shape and forms a torso skeleton of the robot 100. The body frame 318 is fixed integrally with the base frame 308. The head frame 316 is attached to the upper end of the body frame 318 so as to be relatively displaceable.

  The head frame 316 is provided with three axes of a yaw axis 320, a pitch axis 322, and a roll axis 324, and an actuator 326 for driving each axis to rotate. The actuator 326 includes a plurality of servomotors for individually driving each axis. The yaw axis 320 is driven for swinging, the pitch axis 322 is driven for nodding, and the roll axis 324 is driven for tilting.

  A plate 325 that supports the yaw axis 320 is fixed to an upper part of the head frame 316. A plurality of ventilation holes 327 are formed in the plate 325 to ensure ventilation between the upper and lower sides.

  A metal base plate 328 is provided to support head frame 316 and its internal mechanism from below. The base plate 328 is connected to the plate 325 via a cross link mechanism 329 (pantograph mechanism), and is connected to the upper plate 332 (base frame 308) via a joint 330.

  The body frame 318 houses the base frame 308 and the wheel drive mechanism 370. The wheel driving mechanism 370 includes a rotation shaft 378 and an actuator 379. The lower half of the body frame 318 has a small width to form a storage space S for the front wheel 102 with the wheel cover 312.

  The outer cover 314 is made of urethane rubber and covers the main body frame 310 and the wheel cover 312 from outside. Hand 106 is integrally formed with outer skin 314. An opening 390 for introducing outside air is provided at the upper end of the outer cover 314.

FIG. 3 is a configuration diagram of the robot system 300.
The robot system 300 includes the robot 100, the server 200, and a plurality of external sensors 114. A plurality of external sensors 114 (external sensors 114a, 114b,..., 114n) are installed in the house in advance. The external sensor 114 may be fixed to a wall surface of a house or may be mounted on a floor. The position coordinates of the external sensor 114 are registered in the server 200. The position coordinates are defined as x and y coordinates in a house that is assumed as an action range of the robot 100.

The server 200 is installed indoors. Normally, the server 200 and the robot 100 in this embodiment correspond one-to-one. The server 200 determines the basic behavior of the robot 100 based on information obtained from a sensor built in the robot 100 and a plurality of external sensors 114.
The external sensor 114 is for reinforcing the sensory organs of the robot 100, and the server 200 is for reinforcing the brain of the robot 100.

The external sensor 114 periodically transmits a wireless signal (hereinafter, referred to as “robot search signal”) including the ID of the external sensor 114 (hereinafter, referred to as “beacon ID”). Upon receiving the robot search signal, the robot 100 returns a wireless signal including a beacon ID (hereinafter, referred to as a “robot response signal”). The server 200 measures the time from when the external sensor 114 transmits the robot search signal to when the robot response signal is received, and measures the distance from the external sensor 114 to the robot 100. By measuring respective distances between the plurality of external sensors 114 and the robot 100, the position coordinates of the robot 100 are specified.
Of course, a method in which the robot 100 periodically transmits its own position coordinates to the server 200 may be used.

FIG. 4 is a conceptual diagram of the emotion map 116.
Emotion map 116 is a data table stored in server 200. The robot 100 selects an action according to the emotion map 116. The emotion map 116 shown in FIG. The x-axis and y-axis of the emotion map 116 indicate two-dimensional space coordinates. The z-axis indicates the magnitude of evil feelings. When the z value is a positive value, the user has a good feeling for the place, and when the z value is a negative value, the user dislikes the place.

In the emotion map 116 of FIG. 4, the coordinate P1 is a point (hereinafter, referred to as a “favorable point”) in the indoor space managed by the server 200 as an action range of the robot 100, where the favorable emotion is high. The favored point may be a “safe place” such as behind a sofa or under a table, a place where people can easily gather like a living room, or a lively place. In addition, it may be a place that has been gently stroked or touched in the past.
The definition of what kind of place the robot 100 prefers is arbitrary, but in general, it is desirable to set a place favored by small children, small animals such as dogs and cats as favor points.

The coordinates P2 are points where bad emotions are high (hereinafter, referred to as “dislike points”). Dislike points are places that emit loud noises, such as near a TV, places that are easily wet such as baths and toilets, enclosed spaces and dark places, and places that are connected to unpleasant memories that have been treated violently by users. There may be.
The definition of what kind of place the robot 100 dislikes is arbitrary, but in general, it is desirable to set a place where small animals such as small children, dogs and cats are afraid as disgust points.

  The coordinates Q indicate the current position of the robot 100. The server 200 specifies the position coordinates of the robot 100 based on a robot search signal periodically transmitted by the plurality of external sensors 114 and a robot response signal thereto. For example, when the external sensor 114 with beacon ID = 1 and the external sensor 114 with beacon ID = 2 respectively detect the robot 100, the distance of the robot 100 is obtained from the two external sensors 114, and the position coordinates of the robot 100 are obtained therefrom. .

Alternatively, the external sensor 114 with the beacon ID = 1 transmits a robot search signal in a plurality of directions, and the robot 100 returns a robot reply signal when receiving the robot search signal. In this way, the server 200 may know from which external sensor 114 the robot 100 is located in which direction and how long. Further, in another embodiment, the moving distance of the robot 100 may be calculated from the rotation speed of the front wheel 102 or the rear wheel 103 to specify the current position, or the current position may be determined based on an image obtained from a camera. It may be specified.
When the emotion map 116 shown in FIG. 4 is given, the robot 100 moves in a direction to be drawn to the favored point (coordinate P1) and in a direction away from the disliked point (coordinate P2).

  The emotion map 116 changes dynamically. When the robot 100 reaches the coordinate P1, the z value (favorable emotion) at the coordinate P1 decreases with time. Thereby, the robot 100 can reach the favorable point (coordinate P1) and emulate the biological behavior of "satisfied with emotion" and eventually "gets tired" of the place. Similarly, the bad feeling at the coordinate P2 is alleviated with time. With the passage of time, new favorable points and dislike points are created, whereby the robot 100 makes a new action selection. The robot 100 is "interested" in the new favor point and constantly selects an action.

  The emotion map 116 expresses the undulation of emotion as an internal state of the robot 100. The robot 100 aims at the favored point, avoids the disliked point, stays at the favored point for a while, and eventually takes the next action. By such control, the action selection of the robot 100 can be made human and biological.

  The map that influences the behavior of the robot 100 (hereinafter, collectively referred to as an “action map”) is not limited to the emotion map 116 of the type shown in FIG. For example, a variety of action maps can be defined, such as curiosity, fear avoidance, desire for security, and desire for physical comfort such as quietness and dimness, coolness and warmth. Then, the destination point of the robot 100 may be determined by weighting and averaging the z values of each of the plurality of action maps.

  The robot 100 has parameters indicating the size of various emotions and sensations, separately from the behavior map. For example, when the value of the emotion parameter of loneliness is increasing, the weighting coefficient of the behavior map for evaluating a place where people feel safe is set large, and the value of the emotion parameter is reduced by reaching the target point. Similarly, when the value of the parameter indicating the feeling of boring is increasing, the weighting coefficient of the action map for evaluating a place satisfying curiosity may be set to a large value.

FIG. 5 is a hardware configuration diagram of the robot 100.
The robot 100 includes an internal sensor 128, a communication device 126, a storage device 124, a processor 122, a driving mechanism 120, and a battery 118. The drive mechanism 120 includes the wheel drive mechanism 370 described above. The processor 122 and the storage device 124 are included in the control circuit 342. Each unit is connected to each other by a power supply line 130 and a signal line 132. The battery 118 supplies power to each unit via a power supply line 130. Each unit transmits and receives a control signal via a signal line 132. The battery 118 is a lithium ion secondary battery, and is a power source of the robot 100.

  The internal sensor 128 is an aggregate of various sensors built in the robot 100. Specifically, it includes a camera (omnidirectional camera), a microphone array, a distance measurement sensor (infrared sensor), a thermo sensor, a touch sensor, an acceleration sensor, an odor sensor, a touch sensor, and the like. The touch sensor is installed between the outer skin 314 and the main body frame 310, and detects a user's touch. The odor sensor is a known sensor to which the principle of changing the electric resistance by the adsorption of the molecule that causes the odor is applied. The odor sensor classifies various odors into a plurality of categories.

  The communication device 126 is a communication module that performs wireless communication with various external devices such as the server 200, the external sensor 114, and a portable device owned by the user. The storage device 124 includes a nonvolatile memory and a volatile memory, and stores a computer program and various setting information. The processor 122 is a means for executing a computer program. The drive mechanism 120 is an actuator that controls an internal mechanism. In addition, displays and speakers are also installed.

  The processor 122 selects an action of the robot 100 while communicating with the server 200 and the external sensor 114 via the communication device 126. Various external information obtained by the internal sensor 128 also affects the action selection. The drive mechanism 120 mainly controls the wheels (the front wheels 102) and the head (the head frame 316). The drive mechanism 120 changes the moving direction and the moving speed of the robot 100 by changing the rotating speed and the rotating direction of each of the two front wheels 102. The drive mechanism 120 can also raise and lower the wheels (the front wheel 102 and the rear wheel 103). When the wheels rise, the wheels are completely housed in the body 104, and the robot 100 comes into contact with the floor surface F at the seating surface 108 to be in a seated state.

FIG. 6 is a functional block diagram of the robot system 300.
As described above, the robot system 300 includes the robot 100, the server 200, and the plurality of external sensors 114. Each component of the robot 100 and the server 200 includes: an arithmetic unit such as a CPU (Central Processing Unit) and various coprocessors; a storage device such as a memory and a storage; hardware including a wired or wireless communication line connecting them; This is realized by software that is stored in the device and supplies processing instructions to the arithmetic unit. The computer program may be configured by a device driver, an operating system, various application programs located in an upper layer thereof, and a library that provides a common function to these programs. Each block described below is not a configuration in a hardware unit but a block in a functional unit.
Some of the functions of the robot 100 may be realized by the server 200, and some or all of the functions of the server 200 may be realized by the robot 100.

(Server 200)
The server 200 includes a communication unit 204, a data processing unit 202, and a data storage unit 206.
The communication unit 204 handles communication processing with the external sensor 114 and the robot 100. The data storage unit 206 stores various data. The data processing unit 202 performs various processes based on the data acquired by the communication unit 204 and the data stored in the data storage unit 206. The data processing unit 202 also functions as an interface between the communication unit 204 and the data storage unit 206.

  In the present embodiment, the communication unit 204 of the server 200 is connected to the communication unit 142 of the robot 100 by two types of communication lines, a first communication line and a second communication line. The first communication line is a 920 MHz ISM frequency (Industrial, Scientific and Medical Band) communication line. The second communication line is a 2.4 GHz communication line. Since the first communication line has a lower frequency than the second communication line, radio waves are likely to wrap around, but the communication speed is slow.

The data storage unit 206 includes a motion storage unit 232, a map storage unit 216, and a personal data storage unit 218.
The robot 100 has a plurality of motion patterns (motions). Various motions are defined, such as shaking the hand 106, meandering toward the owner, and staring at the owner with his neck stuck.

  The motion storage unit 232 stores a “motion file” that defines the control content of the motion. Each motion is identified by a motion ID. The motion file is also downloaded to the motion storage unit 160 of the robot 100. Which motion is executed may be determined by the server 200 or may be determined by the robot 100.

  Many of the motions of the robot 100 are configured as a composite motion including a plurality of unit motions. For example, when the robot 100 approaches the owner, it may be expressed as a combination of a unit motion facing the owner, a unit motion approaching with the hand raised, a unit motion approaching while shaking the body, and a unit motion sitting with the hands raised. . By such a combination of the four motions, a motion of "approaching the owner, raising his hand on the way, and finally sitting with his body shaken" is realized. In the motion file, a rotation angle, an angular velocity, and the like of an actuator provided in the robot 100 are defined in association with a time axis. Various motions are expressed by controlling each actuator over time according to the motion file (actuator control information).

The transition time when changing from the previous unit motion to the next unit motion is called an “interval”. The interval may be defined according to the time required for changing the unit motion or the content of the motion. The length of the interval is adjustable.
Hereinafter, settings relating to the behavior control of the robot 100, such as when and which motion is selected, output adjustment of each actuator for realizing the motion, etc., are collectively referred to as "behavior characteristics". The behavior characteristics of the robot 100 are defined by a motion selection algorithm, a motion selection probability, a motion file, and the like.

  The motion storage unit 232 stores, in addition to the motion file, a motion selection table that defines a motion to be executed when various events occur. In the motion selection table, one or more motions and their selection probabilities are associated with events.

  The map storage unit 216 stores, in addition to a plurality of action maps, a map indicating an arrangement state of obstacles such as chairs and tables. The personal data storage unit 218 stores information of a user, particularly, an owner. Specifically, it stores master information indicating intimacy with the user and physical and behavioral characteristics of the user. Other attribute information such as age and gender may be stored. Details of the master information will be described later with reference to FIG.

The robot system 300 (the robot 100 and the server 200) identifies the user based on the physical and behavioral characteristics of the user. The robot 100 captures an image of the periphery with a spherical camera. Then, the physical and behavioral characteristics of the person appearing in the image are extracted. Physical features include eye-to-eye spacing, eye-to-mouth-to-nose balance, height, preferred clothing, glasses, skin color, hair color, ear size, etc. It may be a visual feature associated with the body, or may include other features such as average body temperature, odor, voice quality, and the like. Specifically, the behavioral features are features accompanying the behavior, such as a place preferred by the user, activeness of the movement, and the presence or absence of smoking. For example, behavior characteristics such as an owner who is identified as a father often not staying at home and often not moving on a sofa at home, but a mother often staying in a kitchen and having a wide range of activity are extracted.
The robot system 300 according to the present embodiment extracts a plurality of parameters indicating physical characteristics from a master image described later, and identifies a user based on the master image. Hereinafter, the process of identifying a user based on the master image is referred to as “user identification process”. Details of the user identification processing will be described later.

  The robot 100 has an internal parameter called intimacy for each user. When the robot 100 recognizes an action that favors itself, such as holding up or calling out, the intimacy with the user increases. The intimacy with a user who is not involved in the robot 100, a user who works violently, or a user who meets with a low frequency is low.

The data processing unit 202 includes a position management unit 208, a map management unit 210, a recognition unit 212, an operation control unit 222, a closeness management unit 220, and an emotion management unit 244.
The position management unit 208 specifies the position coordinates of the robot 100 by the method described with reference to FIG. The position management unit 208 may also track the position coordinates of the user in real time.

  The emotion management unit 244 manages various emotion parameters indicating the emotion (loneliness, enjoyment, fear, etc.) of the robot 100. These emotion parameters are always fluctuating. The importance of the plurality of action maps changes according to the emotion parameter, the movement target point of the robot 100 changes according to the action map, and the emotion parameter changes according to the movement of the robot 100 and the passage of time.

  For example, when the emotion parameter indicating loneliness is high, the emotion management unit 244 sets a large weighting coefficient of the action map that evaluates a place where people feel safe. When the robot 100 reaches a point where loneliness can be eliminated in the action map, the emotion management unit 244 decreases the emotion parameter indicating loneliness. Various emotion parameters also change depending on a response action described later. For example, the emotion parameter indicating loneliness decreases when the owner “holds”, and the emotion parameter indicating loneliness gradually increases when the owner is not visually recognized for a long time.

The map management unit 210 changes the parameters of each coordinate for the plurality of action maps by the method described with reference to FIG. The map management unit 210 may select any one of the plurality of action maps, or may perform a weighted average of z values of the plurality of action maps. For example, in the action map A, the z values at the coordinates R1 and R2 are 4 and 3, and in the action map B, the z values at the coordinates R1 and R2 are -1 and 3. In the case of the simple average, since the total z value of the coordinates R1 is 4-1 = 3 and the total z value of the coordinates R2 is 3 + 3 = 6, the robot 100 goes not in the direction of the coordinate R1 but in the direction of the coordinate R2.
When the action map A is 5 times more important than the action map B, the total z value of the coordinates R1 is 4 × 5-1 = 19 and the total z value of the coordinates R2 is 3 × 5 + 3 = 18. Head in the direction of.

  The recognition unit 212 recognizes an external environment. The recognition of the external environment includes various recognitions such as recognition of weather and season based on temperature and humidity, and recognition of a shadow (safety zone) based on light amount and temperature. The recognizing unit 156 of the robot 100 acquires various types of environmental information by the internal sensor 128, performs primary processing on the acquired information, and transfers the information to the recognizing unit 212 of the server 200. The recognition unit 156 of the robot 100 extracts a moving object, in particular, an image region corresponding to a person or an animal from the image, and extracts a “feature vector” indicating the physical characteristics and the behavioral characteristics of the moving object from the extracted image region. I do. The robot 100 transmits the feature vector to the server 200.

The recognition unit 212 of the server 200 further includes a person recognition unit 214 and a response recognition unit 228.
The person recognizing unit 214 compares the feature vector extracted from the image captured by the built-in camera of the robot 100 with the feature vector of the user registered in advance in the personal data storage unit 218 to determine which person the captured user is. Is determined (user identification processing). The person recognition unit 214 includes a facial expression recognition unit 230. The facial expression recognition unit 230 estimates the emotion of the user by image-recognizing the facial expression of the user.
Note that the person recognizing unit 214 also performs user identification processing on a moving object other than a person, for example, a cat or dog that is a pet.

  As described above, in the present embodiment, the recognition unit 156 of the robot 100 extracts an image region corresponding to a moving object (person and animal) from a captured image, and extracts a feature vector from the extracted captured image. In the personal data storage unit 218 of the server 200, feature vectors of a plurality of users (hereinafter, referred to as “master vectors”) are registered in advance. The master vector is a feature vector extracted based on the master image of the user. The person recognizing unit 214 of the server 200 identifies the user by comparing the feature vector sent from the robot 100 with the master vector.

  Hereinafter, a user whose master vector is registered in the personal data storage unit 218 is referred to as a “registered user”, and an unidentified user who is a target of the user identification process recognized by the camera is referred to as an “unknown user”. If the master vector of the registered user A and the feature vector of the unknown user X (hereinafter, also referred to as “test vector”) match or are similar, it is determined that the unknown user X is the same person as the registered user A.

The response recognition unit 228 recognizes various response actions performed on the robot 100 and classifies the actions into pleasant and unpleasant actions. The response recognition unit 228 also classifies the response into an affirmative / negative response by recognizing the owner's response to the behavior of the robot 100.
Pleasant or unpleasant behavior is determined based on whether the user's response behavior is comfortable or unpleasant as a living thing. For example, being held is a pleasure for the robot 100, and being kicked is a discomfort to the robot 100. The affirmative / negative response is determined based on whether the user's response action indicates a pleasant feeling or a discomfort feeling of the user. For example, being hugged is a positive response indicating a user's pleasant feeling, and kicking is a negative reaction indicating a user's unpleasant feeling.

  The operation control unit 222 of the server 200 determines the motion of the robot 100 in cooperation with the operation control unit 150 of the robot 100. The operation control unit 222 of the server 200 creates a movement target point of the robot 100 and a movement route therefor based on the action map selection by the map management unit 210. The operation control unit 222 may create a plurality of travel routes, and then select any one of the travel routes.

The operation control unit 222 selects a motion of the robot 100 from a plurality of motions in the motion storage unit 232. Each motion is associated with a selection probability for each situation. For example, a selection method is defined in which the motion A is executed with a probability of 20% when a pleasant action is performed by the owner, and the motion B is executed with a probability of 5% when the temperature exceeds 30 degrees. .
A movement target point and a movement route are determined in the action map, and a motion is selected by various events described later.

  The closeness management unit 220 manages the closeness for each user. As described above, the intimacy is registered in the personal data storage unit 218 as a part of personal data. When a pleasant activity is detected, the intimacy management unit 220 increases the intimacy with the owner. The intimacy drops when an unpleasant activity is detected. In addition, the intimacy of owners who have not viewed for a long time gradually decreases.

(Robot 100)
The robot 100 includes a communication unit 142, a data processing unit 136, a data storage unit 148, an internal sensor 128, and a driving mechanism 120.
The communication unit 142 corresponds to the communication device 126 (see FIG. 5), and is in charge of communication processing with the external sensor 114, the server 200, and the other robot 100. The data storage unit 148 stores various data. The data storage unit 148 corresponds to the storage device 124 (see FIG. 5). The data processing unit 136 performs various processes based on the data acquired by the communication unit 142 and the data stored in the data storage unit 148. The data processing unit 136 corresponds to the processor 122 and a computer program executed by the processor 122. The data processing unit 136 also functions as an interface of the communication unit 142, the internal sensor 128, the driving mechanism 120, and the data storage unit 148.

The data storage unit 148 includes a motion storage unit 160 that defines various motions of the robot 100.
Various motion files are downloaded from the motion storage unit 232 of the server 200 to the motion storage unit 160 of the robot 100. A motion is identified by a motion ID. A state in which the front wheel 102 is housed, in which the robot 100 rotates by seating and sitting on the front wheel 102, lifting the hand 106, or by rotating the two front wheels 102 in reverse or by rotating only one of the front wheels 102. In order to express various motions such as shaking by rotating the front wheel 102 and stopping and turning back when leaving the user, the operation timing, operation time, operation direction, etc. of various actuators (drive mechanism 120) are expressed. Time series is defined in the motion file.

  Various data may be downloaded from the map storage unit 216 and the personal data storage unit 218 to the data storage unit 148.

  Internal sensor 128 includes a camera 134. The camera 134 in this embodiment is a spherical camera attached to the horn 112.

The data processing unit 136 includes a recognition unit 156, an operation control unit 150, an operation detection unit 152, an imaging control unit 154, and a distance measurement unit 158.
The operation control unit 150 of the robot 100 determines the motion of the robot 100 in cooperation with the operation control unit 222 of the server 200. Some motions may be determined by the server 200, and other motions may be determined by the robot 100. Further, the robot 100 determines the motion, but when the processing load of the robot 100 is high, the server 200 may determine the motion. The base motion may be determined in the server 200, and the additional motion may be determined in the robot 100. How the motion determination process is shared between the server 200 and the robot 100 may be designed according to the specifications of the robot system 300.

  The operation control unit 150 of the robot 100 determines the moving direction of the robot 100 together with the operation control unit 222 of the server 200. The movement based on the action map may be determined by the server 200, and the immediate movement such as avoiding an obstacle may be determined by the operation control unit 150 of the robot 100. The drive mechanism 120 drives the front wheel 102 in accordance with an instruction from the operation control unit 150, thereby moving the robot 100 to the movement target point.

  The operation control unit 150 of the robot 100 instructs the drive mechanism 120 to execute the selected motion. The drive mechanism 120 controls each actuator according to the motion file.

  The motion control unit 150 can execute a motion of raising both hands 106 as a gesture to hug "Hug" when a user with high intimacy is nearby, and when the user is tired of "Hug", the left and right front wheels 102 can be moved. By alternately repeating the reverse rotation and the stop with the stowed, it is possible to express a motion that hugs the hug. The drive mechanism 120 causes the robot 100 to express various motions by driving the front wheel 102, the hand 106, and the neck (the head frame 316) according to an instruction from the operation control unit 150.

  The motion detection unit 152 detects “lifting” and “holding down” of the robot 100 in addition to the touch by the user. “Lift” is typically an act of the user lifting the robot 100 with both hands attached to the body 104 of the robot 100. The “holding down” is typically an action in which the user puts both hands on the body 104 of the robot 100 and lowers the robot 100 on the floor F. The motion detection unit 152 detects a user's touch with a touch sensor installed below the outer skin 314 of the robot 100. The operation detection unit 152 determines that “lifting” has been performed on condition that the acceleration sensor detects an increase in a touched state. Similarly, when a descent is detected by the acceleration sensor in a touched state, or when a load on the seating surface 108 or the front wheel 102 is detected, the motion detection unit 152 determines that “holding down” has been performed. The camera 134 may capture a moving image of the outside world, and recognize “lift” and “down” by recognizing the rise and fall of the robot 100 from changes in the image.

  The imaging control unit 154 controls the camera 134. The imaging control unit 154 captures an image of a subject when the object is held up, held down, or a touch is detected, or at various timings described later.

  The distance measuring unit 158 detects a distance to a moving object (person and pet) as a subject by using a distance measuring sensor (infrared sensor) included in the internal sensor 128. The recognition unit 156 also detects the relative angle between the robot 100 and the subject by recognizing the subject as an image. A captured image obtained when the robot 100 is located at a predetermined relative point with respect to the subject can be set as a master image candidate (hereinafter, referred to as a “master candidate image”). A method for acquiring the master candidate image based on the distance measurement will be described later with reference to FIG.

The recognition unit 156 of the robot 100 interprets external information obtained from the internal sensor 128. The recognition unit 156 can perform visual recognition (visual unit), odor recognition (olfactory unit), sound recognition (auditory unit), and tactile recognition (tactile unit).
The recognizing unit 156 periodically captures an image of the outside world using a built-in spherical camera, and detects a moving object such as a person or a pet. The feature vector extracted from the captured image of the moving object by the recognition unit 156 is transmitted to the server 200, and the person recognition unit 214 of the server 200 identifies the user. The recognition unit 156 of the robot 100 also detects the smell of the user and the voice of the user. Odors and sounds (voices) are classified into a plurality of types by a known method.

When a strong impact is given to the robot 100, the recognizing unit 156 recognizes the strong impact by the built-in acceleration sensor, and the response recognizing unit 228 of the server 200 recognizes that a "violent act" has been performed by a nearby user. When the user grasps the horn 112 and lifts the robot 100, it may be recognized as a violent act. When the user facing the robot 100 utters in the specific volume region and the specific frequency band, the response recognition unit 228 of the server 200 may recognize that the “houting action” for itself has been performed. Further, when a temperature around the body temperature is detected, it may be recognized that the “contact action” by the user has been performed.
In summary, the robot 100 acquires the user's action as physical information by using the internal sensor 128, the action detection unit 152 determines an action such as “lifting” or “holding down”, and the response recognition unit 228 of the server 200 is free. -After determining discomfort, the recognition unit 212 of the server 200 executes a user identification process based on the feature vector.

  The response recognition unit 228 of the server 200 recognizes various responses of the user to the robot 100. Pleasant or unpleasant, affirmative or negative are associated with some typical responses among various responses. In general, most of the responding acts that are pleasant are positive reactions, and most of the disturbing acts are negative reactions. Pleasant or unpleasant behavior is related to intimacy, and affirmative or negative reactions affect the robot 100's action selection.

  Among a series of recognition processes including detection, analysis, and determination, the recognition unit 156 of the robot 100 selects and extracts information necessary for recognition, and interpretation processing such as determination is performed by the recognition unit 212 of the server 200. . The recognition process may be performed only by the recognition unit 212 of the server 200, may be performed only by the recognition unit 156 of the robot 100, or may be performed by performing the above-described recognition process while sharing roles. Is also good.

  In accordance with the response action recognized by the recognition unit 156, the familiarity management unit 220 of the server 200 changes the familiarity with the user. In principle, the intimacy with the user who performed the pleasure increases, and the intimacy with the user who performs the pleasure decreases.

  The recognizing unit 212 of the server 200 determines the pleasure or discomfort according to the response, and the map management unit 210 changes the z value of the point where the pleasure or discomfort is performed in the action map expressing “attachment to the place”. You may. For example, when a pleasant activity is performed in the living room, the map management unit 210 may set a favorable point in the living room with a high probability. In this case, the robot 100 prefers the living room and receives a pleasant act in the living room, thereby realizing a positive feedback effect that the robot 100 prefers the living room.

  The intimacy with a moving object (user) changes depending on what action is taken by the user.

  The robot 100 sets a high degree of intimacy with a person who frequently meets, a person who frequently touches, and a person who speaks frequently. On the other hand, the intimacy of those who rarely see, those who do not touch much, those who are violent, and those who scold loudly is low. The robot 100 changes intimacy for each user based on various kinds of external information detected by sensors (visual, tactile, and auditory).

  The actual robot 100 autonomously performs complicated action selection according to the action map. The robot 100 behaves while being influenced by a plurality of behavior maps based on various parameters such as loneliness, boredom, curiosity, and the like. If the effect of the action map is excluded or the robot 100 is in an internal state where the effect of the action map is small, the robot 100 will, in principle, try to approach a person with high intimacy and move away from a person with low intimacy. And

The behavior of the robot 100 is categorized as follows according to the intimacy level.
(1) User with Very High Intimacy The robot 100 approaches the user (hereinafter, referred to as “proximity action”) and performs an affection gesture that is defined in advance as a gesture showing favor with the human, thereby making the friendship affect. Express strongly.
(2) The user robot 100 whose intimacy is relatively high performs only the proximity action.
(3) User with relatively low intimacy The robot 100 does not take any special action.
(4) The user robot 100 whose intimacy is particularly low performs the leaving behavior.

  According to the above control method, the robot 100 approaches a user with a high degree of intimacy, and moves away from a user with a low degree of intimacy. With such a control method, a so-called “shyness” can be expressed as an action. Also, when a visitor (user A with a low degree of intimacy) appears, the robot 100 may move away from the guest and head toward a family (user B with a high degree of closeness). In this case, the user B can sense that the robot 100 is shy and anxious and relies on himself. Such an action expression evokes the joy of being selected and relied on by the user B, and the feeling of attachment accompanying it.

  On the other hand, if the user A who is a visitor frequently visits, calls and touches, the intimacy of the robot 100 with the user A gradually increases, and the robot 100 does not perform a shyness action (withdrawal action) with respect to the user A. . The user A can also have an attachment to the robot 100 by feeling that the robot 100 has become familiar to him.

  Note that the above action selection is not always performed. For example, when the internal parameter indicating the curiosity of the robot 100 is high, the behavior map for finding a place that satisfies the curiosity is emphasized. Therefore, the robot 100 may not select the behavior affected by the intimacy. . When the external sensor 114 installed at the entrance detects that the user has returned home, the user's greeting action may be executed with the highest priority.

FIG. 7 is an image diagram when the robot is carried.
The robot 100 has a round, soft, soft-touching body 104 and an appropriate weight, and recognizes a touch as a pleasure. Therefore, it is easy for the user to embrace the feeling of wanting to hold the robot 100. The robot 100 applies this feeling of wanting to be involved to the user identification processing.

In order for the robot 100 to identify a user, information as a clue is necessary. For example, eyebrow thickness, eye size, eye shape, skin color, skin brightness, wrinkle shape, hair brightness, bangs length, eyes and nose size in the entire face Cue is based on physical characteristics such as proportion, eye-to-eye spacing. In the present embodiment, first, the robot 100 acquires a master image. The recognition unit 156 of the robot 100 extracts a feature vector (master vector) from the master image. The feature vector has a plurality of vector components. The feature vector component is a numerical value that quantifies the various physical features described above. For example, the width of the eyes is quantified in the range of 0 to 1 and these form a feature vector component. The method of extracting a feature vector from a captured image of a person is an application of a known face recognition technology. The master vector of the user A is stored as the master information 224 of the personal data storage unit 218.
Hereinafter, the process of extracting a feature vector from a captured image is referred to as “vector extraction process”.

  When the robot 100 captures an image of the unknown user X, the recognition unit 156 extracts a feature vector (inspection vector) from a captured image (inspection image) of the unknown user X. If the test vector of the unknown user X and the master vector of the registered user A are similar, the person recognizing unit 214 of the server 200 determines that the unknown user X and the registered user A are the same person.

  In order to increase the identification accuracy, it is necessary to image a user at a high-quality master image from which a master vector can be easily extracted, more specifically, at a short distance. The recognizing unit 156 in the present embodiment sets a captured image when the motion detecting unit 152 detects the holding of the robot 100 as a master image. When the robot 100 is being held, the robot 100 can take an image with high accuracy by the built-in camera 134. This is because when the robot 100 is lifted, the distance between the user's face and the camera 134 built into the robot 100 falls within a certain range. Instead of giving the user an "action instruction" to capture the master image, the user can obtain a good-quality master image without imposing a burden on the user at the timing of holding the robot 100 with his / her own will.

FIG. 8 is a data structure diagram of the master information.
The master information 224 is stored in the response recognition unit 228. In FIG. 8, three master vectors are associated with a user with user ID = 01 (hereinafter, referred to as “user (01)”). The master image is acquired not only from the front of the user (01), but also from a profile such as a right side or a left side. Therefore, by imaging the user from a plurality of angles and a plurality of distances, a plurality of master vectors are associated with one registered user. The master vector is identified by a master ID. The master vector (01) is extracted from the master image when the face of the user (01) is imaged from the front, and the master vector (02) is extracted from the master image when the face of the user (01) is imaged from the right. .

  For the sake of simplicity, the master vector shown in FIG. 8 will be described as a five-dimensional vector having five vector components. The five vector components a to e correspond to arbitrary feature amounts such as an eye-to-eye distance and a skin color. The master vector (01) includes feature amounts a1, b1, and c1 corresponding to three vector components a to c. On the other hand, no feature amount is set for the vector components d and e. For example, when the vector component d is a feature amount indicating the size of the ear, the component d may not be able to be extracted from the front master image.

  The master vector (02) includes the feature amounts a2, c2, and e2 corresponding to the three vector components a, c, and e, but does not include the feature amounts corresponding to the vector components b and d. The master vector (03) includes feature amounts a3, b3, d3, and e3 corresponding to four vector components a, b, d, and e, but does not include feature amounts corresponding to the vector component c. If a plurality of master images are obtained by imaging the user (01) from a plurality of directions, the physical characteristics of the user (01) can be grasped three-dimensionally.

  The person recognizing unit 214 calculates the center-of-gravity vector MB by arithmetically averaging the three master vectors of the user (01). The vector component a of the centroid vector MB is an average value of the a components (a1, a2, a3) of the three master vectors. Since only the master vector (03) has the vector component d, the vector component d of the center-of-gravity vector MB becomes the feature amount d3 of the master vector (03). The person recognition unit 214 performs a user identification process based on the master vector or the center of gravity vector MB (described later).

It is assumed that there is no registered user.
The motion detection unit 152 acquires a master image when the unknown user A holds the user. The person recognizing unit 214 records the user ID = 01 in the master information 224 in association with the master vector (01) extracted from the master image of the unknown user A. At this time, the person recognizing unit 214 also records the acquisition date and time of the master vector (01). Through the above processing, unknown user A is registered in master information 224 as registered user (01). In FIG. 8, the master vector (01) was acquired on June 7, 2016.

  After the registration of the user (01), even when a new unknown user X carries the robot 100, the motion detection unit 152 acquires a master image. The person recognizing unit 214 compares the master vector MX extracted from the master image of the unknown user X with the master vector (01) of the registered user (01).

(1) When Not Registered The person recognizing unit 214 determines that the unknown user X is different from the user (01) if the vector distance between the master vector MX and the master vector (01) is equal to or longer than a predetermined distance. The feature vector distance may be calculated as a Euclidean distance, or may be a distance calculation based on another definition such as a Chebyshev distance. The person recognizing unit 214 registers the unknown user X in the master information 224 as a new registered user (02), assigns a user ID = 02 to the user X, and assigns a master ID = 04 to the master vector MX. By the above processing, the user (01) and the user (02) are registered in the master information 224.

(2) In the case of already registered If the distance between the master vector MX and the master vector (01) is less than the predetermined distance, the person recognizing unit 214 determines that the unknown user X and the registered user (01) are the same person. The person recognizing unit 214 sets the master ID = 02 in the master vector MX and associates it with the user (01). The master vector of the user (01) is two, and the information for identifying the user (01) is enriched.

  Since a high quality master vector is obtained from the master image, it is possible to easily determine whether the user holding the robot 100 is a registered user or a person by comparing the master vectors.

  When there are a plurality of registered users, the master vector of each registered user is to be compared. When one registered user has two or more master vectors, the center of gravity vector of the registered user and the master vector of the unknown user are compared.

FIG. 9 is a first schematic diagram for explaining a user identification method.
9 and 10, two vector components a and b among the vector components included in the feature vector will be described in order to illustrate the principle of the user identification process. The processing method is the same when there are three or more vector components.
In FIG. 9, one master vector MA and one master vector MB are extracted for each of the registered user A and the registered user B. The master vector MA = (a1, b1) and the master vector MB = (a2, b2). In such a situation, it is assumed that the robot 100 has acquired a captured image (inspection image) of the unknown user X who is walking from the front. The recognizing unit 156 determines whether the unknown user X shown in the inspection image is the registered user A or B. Since the user is not hugged, the feature vector (inspection vector) obtained from the inspection image of the unknown user X usually does not have as high accuracy as the master vector.

  The recognizing unit 156 first extracts a test vector DX = (ax, bx) from the test image of the unknown user X. The communication unit 142 of the robot 100 transmits the inspection vector DX to the communication unit 204 of the server 200. The person recognizing unit 214 of the server 200 calculates the distance ra between the test vector DX and the master vector MA, and the distance rb between the test vector DX and the master vector MB.

  When rb <ra and rb <rm when an arbitrary threshold value rm is set, the person recognizing unit 214 determines that the unknown user X is the registered user B. On the other hand, if ra <rb and ra <rm, the person recognizing unit 214 determines that the unknown user X is the registered user A. On the other hand, when ra> rm and rb> rm, unknown user X does not correspond to any of registered users A and B. When it is determined that the unknown user X is the registered user A with a high degree of familiarity, the operation control unit 150 may select a close action such as running up to the unknown user X. On the other hand, when it is determined that the unknown user X is the registered user B with low intimacy, the operation control unit 150 may select an avoidance action such as escaping from the unknown user X.

  When the unknown user X cannot be identified, the person recognizing unit 214 notifies the robot 100 that the unknown user X has not been confirmed, and the motion control unit 150 of the robot 100 may select a motion that hugs the unknown user X. Specifically, motions such as approaching the unknown user X, raising the hand 106, and sitting down in front of the unknown user X are conceivable.

  When the unknown user X holds the robot 100 and the motion detection unit 152 detects “holding”, the imaging control unit 154 controls the camera 134 to image the unknown user X from a short distance. The recognition unit 156 extracts the master vector MX from the unknown user's master image obtained at the time of lifting. The person recognizing unit 214 of the server 200 may execute the user identification process again by comparing the master vector MX of the unknown user X with the existing master vectors MA and MB. Since comparison is made between master vectors, more accurate identification is possible. When the unknown user X is also determined to be a different person from the registered users A and B by the comparison of the master vectors, the person recognizing unit 214 sets the unknown user X as the third registered user in the master information 224 together with the master vector MX. sign up.

  When the unknown user X is determined to be the registered user A, the person recognizing unit 214 may register the test vector obtained from the test image of the unknown user X as a new master vector of the registered user A.

FIG. 10 is a second schematic diagram for explaining the user identification method.
In FIG. 10, a plurality of master vectors are extracted for each of the registered user A and the registered user B. The person recognizing unit 214 calculates a center-of-gravity vector MB (A) of the registered user A and a center-of-gravity vector MB (B) of the registered user B. In such a situation, it is assumed that the robot 100 has acquired a captured image (inspection image) of the unknown user X who is walking from the front. The recognizing unit 156 determines whether the unknown user X shown in the inspection image is the registered user A or B.

  The recognizing unit 156 first extracts a test vector DX = (ax, bx) from the test image of the unknown user X. The communication unit 142 of the robot 100 transmits the inspection vector DX to the communication unit 204 of the server 200. The person recognizing unit 214 of the server 200 calculates ra which is the distance between the test vector DX and the centroid vector MB (A), and rb which is the distance between the test vector DX and the centroid vector MB (B).

  When rb <ra and rb <rm when an arbitrary threshold value rm is set, the person recognizing unit 214 determines that the unknown user X is the registered user B. On the other hand, if ra <rb and ra <rm, the person recognizing unit 214 determines that the unknown user X is the registered user A. On the other hand, when ra> rm and rb> rm, unknown user X does not correspond to any of registered users A and B.

FIG. 11 is a flowchart showing a master vector extraction process.
When the motion detection unit 152 of the robot 100 detects that the robot 100 is held up, the vector extraction processing of FIG. 11 is executed. The operation control unit 150 executes a predetermined guidance motion when the holding is detected (S10). The guidance motion is a motion defined in advance to make the user pay attention. Specifically, non-verbal motions such as shaking the hand 106, shaking the body 104, turning the head frame 316 toward the user, and shaking the head frame 316 up and down or left and right are assumed. Guided motion is not limited to mechanical motion. The operation control unit 150 causes the eye 110 to display a “pupil” as an image using the organic EL element. The operation control unit 150 may instruct image control such as opening the pupil, shaking the pupil, and causing the pupil to wink by enlarging the pupil image.

  The user's face is directed to the robot 100 by attracting the user with the guided motion. Further, by preparing various guidance motions, various facial expressions of the user can be extracted, and various master vectors corresponding to various facial expressions can be extracted. For example, a characteristic amount specific to a smile, such as a laugh wrinkle or a dimple, may be included as a vector component of the master vector.

  After executing the guided motion, the imaging control unit 154 controls the camera 134 to image the user (S12). The captured image at this time is the “master candidate image”. By imaging the user at the timing when the user looks at the robot 100 by the guided motion, it is possible to acquire a high-quality master candidate image in which the user's face can be easily recognized.

  The recognizing unit 156 determines the quality of the master candidate image (S14). Hereinafter, the quality determination of the master candidate image is referred to as “quality inspection”. A master candidate image that has passed the quality inspection is set as a master image. If the quality inspection fails (N in S14), the process returns to S10, and reacquires the master candidate image. At this time, another type of guided motion may be executed. For quality inspection, a plurality of evaluation items are set in advance for the user's face size, light amount, facial expression, and the like. For example, when the user has closed eyes, when the master candidate image is too dark or too bright, or when the master candidate image is out of focus, the quality inspection fails. What evaluation items are set for quality inspection is optional.

  The recognizing unit 156 adopts the master candidate image that has passed the quality inspection as a formal master image (Y in S14). The recognizing unit 156 extracts a master vector from the master image (S16). The communication unit 142 transmits the master vector to the server 200 (S18).

  The person recognizing unit 214 compares the newly obtained master vector with the master vector already registered in the master information 224 (S20). When the distance between the newly obtained master vector and the registered master vector is short (Y in S20), the master vector is additionally registered (S22). For example, when a master vector similar to the master vector (01) of the user (01) is obtained, a new master vector is also associated with the user (01). When it is not close to any of the registered master vectors (N in S20), a new user ID and a master ID are assigned to newly register the master vector (S24).

  In S20, the registered master vector may be compared with the newly extracted master vector, or the registered center-of-gravity vector may be compared with the newly extracted master vector as described with reference to FIG.

  The extraction process of the master vector is not limited to the hug, and may be executed when the user touches the robot 100. When the user touches the robot 100, there is a possibility that a good master image can be obtained because the user is near the robot 100.

  Also when the motion detection unit 152 detects that the robot 100 is held down, the recognition unit 156 executes a master vector extraction process. The movement detection unit 152 continuously captures an image of the user when the holding down is detected. The recognition unit 156 sequentially performs quality inspection on the plurality of master candidate images obtained at this time, and extracts a plurality of master vectors. At the time of holding down, physical characteristics such as chin, waist and feet can be imaged at a short distance.

  The master vector obtained when the user lifts the robot 100 is referred to as a “first master vector”, and the master vector obtained when the user lowers the robot 100 or after the robot 100 is lowered is referred to as a “second master vector”. . After obtaining the first master vector of the user (01), the recognizing unit 156 also extracts one or more second master vectors from the master image at the time of holding down. When the first master vector with high accuracy is obtained in this manner, the master vector of the user (01) can be enriched by acquiring the second master vector even when the user holds the master vector down. The “second master vector” here includes master vectors obtained from various distances and angles, even after the robot 100 is held down, including the back of the user. The first master vector and the second master vector are associated with each other for one user as shown in the master information 224.

FIG. 12 is a schematic diagram showing a method of tracking an image of a user.
Even after the robot 100 is lowered onto the floor F, the imaging control unit 154 further tracks the user with the camera 134 (omnidirectional camera). The celestial sphere imaging range 418 shown in FIG. The spherical camera can capture an image of substantially the entire upper hemisphere of the robot 100 at a time. The recognition unit 156 of the robot 100 tracks the user in the celestial sphere imaging range 418 for a predetermined period, for example, about 10 seconds after the first master vector is extracted. The imaging control unit 154 captures a master image of the user from various angles and various distances during tracking. For example, the length of the hair, the thinness of the waist, and the like are information that cannot be obtained without leaving the user. The recognition unit 156 enriches the master vectors by extracting various second master vectors from the master image obtained during tracking. These second master vectors are managed in association with the first master vector. In addition to tracking the user on the image in the celestial sphere imaging range 418, the operation control unit 150 may execute a tracking action such as following the user or moving around the user. Then, even during the tracking action, the imaging control unit 154 may enhance the master vector by imaging the user. The tracking action may be instructed by the operation control unit 150, or the operation control unit 222 of the server 200 may instruct the operation control unit 150.

FIG. 13 is a schematic diagram for explaining a method of remotely extracting a master vector.
The imaging control unit 154 captures a master candidate image when the user is located at a predetermined relative position with respect to the robot 100, in addition to holding and touching. Here, the relative point includes both the distance between the user and the robot 100 and the relative angle. The distance measuring unit 158 periodically measures one or more users recognized in the celestial sphere imaging range 418. In FIG. 13, the robot 100 is located at a relative point of a horizontal angle a with respect to the front direction of the user, an elevation angle b with respect to the position of the user's face, and a distance r from the user. The recognition unit 156 determines the orientation of the user's body by image recognition. When the distance, the horizontal angle, and the elevation angle are within a predetermined range (hereinafter, referred to as a “master shot range”), the imaging control unit 154 captures a master candidate image. The recognizing unit 156 inspects the quality of the master candidate image, and extracts a master vector if the master candidate image passes.

  In the distance measuring section 158, a plurality of master shot ranges are set in advance. The ranging unit 158 notifies the imaging control unit 154 every time the user enters the master shot range, and the imaging control unit 154 acquires a master candidate image. For example, when the master shot ranges R1 to R3 are defined and the new user C enters the master shot range R1, a master candidate image corresponding to the master shot range R1 is acquired. In this manner, master vectors corresponding to the respective master shot ranges R1 to R3 are extracted. The user C can be three-dimensionally grasped by capturing the user C from a plurality of master shot ranges, in other words, from a plurality of relative points and obtaining master vectors from multiple directions.

  Since the user can be photographed from a close distance at the time of contact such as holding or touching, it is easy to obtain high-quality information on the user's face. On the other hand, even when the user is not hugging or touching, when the distance measuring unit 158 detects a user at a short distance, the imaging control unit 154 may obtain the master candidate image. For example, even if a small child has resistance to holding or touching, a master vector can be extracted when approaching with interest. In addition, in order to obtain information on the length and body type of the user's hair, the robot 100 must be separated from the user to some extent. By setting various master shot ranges, various master vectors including not only the user's face but also the body shape can be obtained.

  The first master vector is a feature vector useful for identifying a user because the first master vector is based on a master image of the user taken from a close distance. On the other hand, the second master vector often does not clearly show the physical characteristics of the user as much as the first master vector. Then, triggered by the extraction of the first master vector (A) from the master image A at the time of holding or touching, the imaging control unit 154 enters the tracking mode. The tracking mode may continue for a predetermined time. The imaging control unit 154 acquires the master image B1 when detecting the holding down, for example. A second master vector (B1) is extracted from the master image B1, and is associated with the first master vector (A) extracted earlier. The tracking mode continues even after the holding down, and when the user enters the master shot range, the master image B2 is further acquired. The second master vector (B2) obtained from the master image B2 is also associated with the first master vector (A) that triggered the tracking mode. In this way, various second master vectors obtained after that are associated with the first master vector (A) that can easily identify the user. Even in the case of the second master vector, whose characteristic is unlikely to appear like “back view”, the master vector group corresponding to one user can be enriched by associating the second master vector with the first master vector that triggered the acquisition. .

The robot 100 and the robot system 300 including the robot 100 have been described above based on the embodiment.
In the face recognition technology, a master image is often acquired after giving a user a language instruction such as "look at the front" or "look at the camera". Such language instructions such as voices and characters tend to burden the user. Further, the language instruction for acquiring the master image is also undesirable in that the user is made aware of the inanimateness of the robot 100. The robot 100 according to the present embodiment can casually acquire the master image at the timing when the user holds the robot 100. The robot 100 has a shape such as small, soft, light, and round that makes a human touch. Instead of forcing the user to take any action, it is possible to acquire a high-quality master image by capturing the timing when the user naturally "carryes". The master image can be casually acquired by utilizing the characteristics of the robot 100 that stimulates the desire to hold and touch.

  The robot 100 further draws the user's attention by non-verbal guided motion such as flapping the hand 106. Since this is a method in which the user is noticed by non-verbal communication, the user is less likely to have a sense of being forced.

  The robot 100 captures an image of the user when being held, and acquires a first master vector. Further, the robot 100 can also acquire a plurality of second master vectors by capturing a master image when the robot is held down or after the robot is held down. By accumulating the second master vector at the timing when the first master vector is extracted, it becomes easier to grasp the physical characteristics of the user from multiple sides.

  According to the present embodiment, it is easy to establish a fine criterion for user identification processing based on high quality and a large number of master images. The user identification process is a prerequisite for recognizing a response action and calculating intimacy. By identifying the user with high accuracy based on the master vector, the robot 100 can change the behavior characteristics according to the user.

  Note that the present invention is not limited to the above embodiments and modified examples, and can be embodied by modifying the constituent elements without departing from the gist. Various inventions may be formed by appropriately combining a plurality of components disclosed in the above embodiments and modifications. In addition, some components may be deleted from all the components shown in the above-described embodiment and the modified examples.

  Although one robot 100, one server 200, and a plurality of external sensors 114 have been described as constituting the robot system 300, some of the functions of the robot 100 may be realized by the server 200, or the functions of the server 200. May be allocated to the robot 100. One server 200 may control a plurality of robots 100, or a plurality of servers 200 may control one or more robots 100 in cooperation.

  A third device other than the robot 100 and the server 200 may perform some of the functions. An aggregate of each function of the robot 100 and each function of the server 200 described in FIG. 6 can be globally grasped as one “robot”. How to allocate a plurality of functions necessary to implement the present invention to one or a plurality of hardware is determined in consideration of the processing capacity of each hardware, specifications required for the robot system 300, and the like. It only has to be determined.

  As described above, the “robot in a narrow sense” refers to the robot 100 that does not include the server 200, whereas the “robot in a broad sense” refers to the robot system 300. Many of the functions of the server 200 may be integrated into the robot 100 in the future.

  The master vector may include a feature amount other than the feature amount extracted from the master image as a vector component. For example, the vector component may include the odor detected by the odor sensor, the voice quality detected by the microphone, and the body temperature detected by the temperature sensor. In particular, at the time of holding, it is easy to detect the user's odor, body temperature, and the like with high accuracy. The master image may be a moving image (hereinafter, referred to as “master moving image”) instead of a still image. The recognizing unit 156 may extract habits such as a user's walking style and poor extortion from the master moving image, and include these pieces of characteristic information in a master vector component.

  Although the camera 134 in the present embodiment is a spherical camera, the camera 134 may be a normal camera. The camera 134 may be built in the horn 112 or may be built in the eye 110. Further, both a spherical camera and a normal camera may be incorporated.

  In FIG. 8, the center of gravity vector is formed by arithmetic mean of a plurality of master vectors, but as a modification, the median of the plurality of master vectors may be used as a component of the center of gravity vector. For example, if a1 <a2 <a3 in FIG. 8, the a component of the center of gravity vector may be a2.

  In quality inspection of the master candidate image, a plurality of evaluation items may be weighted. As the evaluation items, (E1) facing the front, (E2) light amount is appropriate, (E3) whether eyes are opened, and the like can be considered. The master candidate image may be graded for each evaluation item, and the quality of the master candidate image may be determined by weighted averaging of these item points. For example, the coefficients of p1, p2, and p3 are set to E1 to E3 (p1 + p2 + p3 = 1), and if the item values of E1 to E3 are s1, s2, and s3, the total point is p1 · s1 + p2 · s2 + p3 · s3. . If the total score is equal to or larger than a predetermined threshold, the master image is adopted as a master image, and the master vector is registered in the master information 224.

  The guided motion may be executed other than when the user is hugged. When the distance between the robot 100 and the user is within a predetermined range, the operation control unit 150 may execute a guided motion. For example, a master candidate image may be taken after a guided motion is executed when a user is in the master shot range shown in FIG. In general, the robot 100 extracts a master vector without missing a “photo opportunity” that is effective in grasping the physical and behavioral characteristics of the user while engaging in various ways with the user. Various and high-quality master vectors can be collected without making the user aware. In addition, a non-verbal approach through guided motion can positively create “shutter opportunities”.

  Guided motion in the present embodiment is a type of non-verbal communication. Here, the non-verbal motion may include a sound that does not make sense as a language, such as the sound of an animal. As a variant, the robot 100 may call the user in a simple language.

  When held by the user, the robot 100 may acquire three face images of a front face, a right face, and a left face as master images. The recognition unit 156 may determine from which direction the user is looking by recognizing the user's ears and nose. As one of the internal sensors 128, the robot 100 may include a gyroscope. The recognizing unit 156 may use a gyroscope to detect the tilt direction of the robot 100 when held by the user, and thereby determine from which direction the user is looking.

  In addition to the method shown in FIGS. 9 and 10 (hereinafter, referred to as “distance determination method”), user identification may be performed based on Mahalanobis distance. In FIG. 10, when a plurality of master vectors are obtained, the person recognition unit 214 obtains a Mahalanobis' Distance between the test vector DX and the master vector group of the user A in consideration of the variance. . Similarly, the person recognizing unit 214 obtains a Mahalanobis distance between the inspection vector DX and the master vector group of the user B. Then, based on the Mahalanobis distance for each group, it may be determined whether the unknown user X is the user A or the user B by a known discriminant analysis method (hereinafter, “Maharanobis determination method”). Call).

  The person recognizing unit 214 may form a neural network using the master vector group of each user as teacher data, and may execute user identification based on the goodness of fit between the test vector of the unknown user X and the master vector. (Hereinafter, it is called “neural network judgment method”).

  The person recognizing unit 214 may identify the user by combining a plurality of the distance determination method, the Mahalanobis determination method, and the neural network determination method. In addition to comparing the test vector with the master vector, the similarity determination may be performed by the above-described various methods when comparing the master vector of the registered user with the master vector of the unknown user.

  In the present embodiment, the description has been given assuming that the imaging control unit 154 acquires the master image at the timing of holding or touching. As a modification, the imaging control unit 154 may periodically image the user, and the recognizing unit 156 may select a large number of captured images as master candidate images. For example, the user is imaged at a timing of once every 10 seconds, and the recognition unit 156 performs quality inspection as a master candidate image. The recognizing unit 156 extracts a master vector from the passed master image. According to such a method, a master vector can be extracted from a high-quality captured image obtained by accident.

  When the number of master vectors becomes equal to or more than a predetermined number, the person recognizing unit 214 may delete the old master vector from the personal data storage unit 218. Alternatively, an old master vector, for example, a master vector obtained three years or more ago may be deleted. According to such a control method, not only can the data amount of the personal data storage unit 218 be suppressed, but it is also possible to cope with changes in physical characteristics due to aging and growth of the user.

  In the present embodiment, it has been described that the user is identified by extracting the feature vector in the robot 100 and comparing the feature vector in the server 200. As a modification, the robot 100 may send the captured image to the server 200, and the person recognizing unit 214 of the server 200 may perform both the extraction of the feature vector and the user identification. Alternatively, the robot 100 may execute the user identification process in the recognition unit 156 without depending on the processing capability of the server 200. In this case, the robot 100 may manage the master vector of each user in the data storage unit 148 of the robot 100.

  Not only the camera and various sensors built into the robot 100 but also the sensor built into the external sensor 114 may extract the physical and behavioral characteristics of the user. The external sensor 114 captures an image of the user when the user is nearby, and transmits the captured image to the robot 100. The recognition unit 156 of the robot 100 may execute the quality inspection and the component extraction of the captured image.

  In the present embodiment, the personal data storage unit 218 stores the master vector instead of the master image. However, the personal data storage unit 218 may store both the master image and the master vector.

  The robot system 300 does not need to have a user identification function using a master vector from the time of shipment from the factory. For example, the robot system 300 may perform user identification by a clustering technique using deep learning. After shipping the robot system 300, the function of the robot system 300 may be enhanced by downloading an action control program for realizing a user identification function using a master vector via a communication network.

  As described above, the recognition unit 156 selects a captured image when the robot 100 is held up as a master candidate image. The recognition unit 156 may detect the position and orientation of the user's face using a temperature sensor such as a thermal camera, or may detect the distance between the user and the robot 100 using a distance measurement sensor. The recognizing unit 156 may select, as a master candidate image, a captured image when a predetermined specific condition is satisfied for both or one of the temperature information from the thermal camera and the distance information from the distance measurement sensor. For example, the recognizing unit 156 can use a thermal camera to confirm that the user is facing the robot 100, and use a distance measurement sensor to set a captured image when the distance between the user and the robot 100 is within a predetermined range as a master candidate image. You may choose. According to such a control method, it becomes easy to carefully select an appropriate master candidate image based on a plurality of types of sensors.

  The camera mounted on the robot 100 may be a spherical camera. When the robot 100 is held by the user from the back side, in other words, even when the robot 100 and the user are not directly facing each other, the robot 100 can photograph the user behind using the spherical camera. Therefore, even when the robot 100 is hung from the back side, the recognition unit 156 can acquire an appropriate master candidate image, so that the opportunity of acquiring the master candidate image can be expanded. Alternatively, the recognition unit 156 may specify the master candidate image on condition that the robot 100 and the user face each other.

  When a plurality of users appear in the captured image when the user is hugged, the recognition unit 156 may not select the captured image as a master candidate image.

  When the registered user P1 is detected in a captured image including a plurality of users, the recognizing unit 156 extracts a feature vector of the registered user P1 from the captured image and additionally registers the feature vector as a new master vector of the registered user P1. It may be. When a registered user is not detected in a captured image including a plurality of users, in other words, when a captured image including only a plurality of unknown users is obtained, the recognition unit 156 may perform a predetermined operation such as facing the front. A feature vector may be extracted for an unknown user P2 that satisfies the condition, and this may be newly registered as a master vector of the unknown user P2.

  The recognizing unit 156 may also acquire the voice (voice information) of the user by using a microphone at the time of shooting. The master vector is not limited to image information, and may include a feature vector based on audio information. Similarly, the recognition unit 156 may acquire the user's smell (smell information) by the odor sensor at the time of photographing. As described above, the information for specifying the registered user may include various sensor information such as audio information and olfactory information in addition to the image information.

  The robot 100 may include a plurality of microphones. At the time of voice registration, voice may be detected only from a microphone corresponding to the direction in which the user exists, for example, a microphone attached in front of the robot 100. The recognition unit 156 may invalidate other microphones. According to such a control method, it becomes difficult for environmental sounds other than the user to be captured in the master vector. It is desirable that the microphone, especially the microphone mounted in front, has directivity.

  The recognition unit 156 may acquire the voice information of the user as a part of the master vector on condition that the voice information is the voice information when the movement of the lips of the user shown in the captured image is detected. When an unknown user is detected, the recognizing unit 156 may execute a motion approaching the unknown user and hugging.

Claims (9)

An imaging control unit that controls the camera;
A recognition unit that determines a moving object based on a feature vector extracted from a captured image of the moving object,
An operation selection unit that selects a motion of the robot according to the determination result;
A drive mechanism for executing the motion selected by the operation selection unit,
A motion detection unit that detects the holding of the robot by the moving object,
The recognition unit sets a captured image when the robot is held by the moving object as a master image, and sets a determination criterion for the moving object based on a feature vector extracted from the master image. Autonomous robot.   The autonomous robot according to claim 1, wherein the recognition unit sets a plurality of captured images of the moving object from a plurality of angles as a master image. The operation selection unit causes the drive mechanism to execute a predetermined guidance motion,
The autonomous robot according to claim 1, wherein the imaging control unit captures a master image of the moving object when the guidance motion is executed.   The autonomous robot according to claim 3, wherein the guidance motion is a non-verbal motion.   The autonomous robot according to claim 1, wherein the recognizing unit further sets a captured image when the robot is held down by the moving object as a master image. The operation selection unit allows the user to select a motion that tracks the moving object after the robot is held down by the moving object,
The autonomous robot according to claim 1, wherein the imaging control unit captures a master image of the moving object during tracking. The imaging control unit tracks the moving object even after the robot is held down by the moving object, and captures a second master image of the moving object at a predetermined timing,
The recognition unit may extract a plurality of feature vectors related to the moving object by associating a first master image acquired when the robot is held by the moving object with the second master image. The autonomous behavior robot according to claim 1, wherein   The recognition unit determines a moving object based on a feature vector extracted from both a captured image of the moving object and audio information, and also acquires audio information from the moving object when acquiring the master image. The autonomous robot according to claim 1, wherein a criterion for determining a moving object is set based on a feature vector extracted from the voice information. A computer program for object recognition by a robot,
A function to set a captured image of the moving object when the robot is held by the moving object as a master image,
A function of setting a determination criterion for a moving object based on a feature vector extracted from the master image,
A behavior control program characterized by causing a robot to perform a function of determining a moving object based on a feature vector extracted from a captured image of the moving object.

Images