JP2024055866A - Robot autonomously selecting behavior according to internal state or external environment - Google Patents

Abstract

To provide a user with sense of security when living with robots.SOLUTION: An autonomous robot includes: an operation control unit which selects the motion of the robot; a driving mechanism which executes the motion selected by the operation control unit; a recognition unit which determines whether or not a target object satisfies a predetermined watching condition; a mode setting unit which sets a watching mode of the target object when the watching condition has been established; and a communication unit which transmits a taken image of the target object to a predetermined communication terminal in the watching mode.SELECTED DRAWING: Figure 1A

Description

The present invention relates to a robot that autonomously selects actions depending on its internal state or external environment.

Humans keep pets to find comfort. However, many people give up on having pets for a variety of reasons, including not having enough time to look after them, not having a living environment that allows them to keep a pet, allergies, and the pain of losing a pet. If there were a robot that could play the role of a pet, it might be able to provide the same comfort that pets provide to people who cannot keep pets (see Patent Documents 1 and 2).

JP 2000-323219 A International Publication No. 2017/169826

In recent years, robot technology has been advancing rapidly, but we have yet to realize a robot that can act as a companion like a pet. This is because we do not believe that a robot has free will. By observing the behavior of our pets, which we can only assume to be of free will, humans can sense the presence of free will in their pets, empathize with them, and feel comforted by them.

The present invention was completed based on the recognition of the above problems, and its first objective is to provide technology that provides a variety of feelings of security to users when living with a robot. The second objective is to provide technology that enables a robot to express affection for its user in a rich way. The third objective is to provide technology that enables multiple robots to act in cooperation with each other.

In one aspect of the present invention, an autonomous robot includes a motion control unit that selects the motion of the robot, a drive mechanism that executes the motion selected by the motion control unit, a recognition unit that determines whether a subject satisfies a predetermined monitoring condition, a mode setting unit that sets the subject to a monitoring mode when the monitoring condition is met, and a communication unit that transmits a captured image of the subject to a predetermined communication terminal in the monitoring mode.

This invention makes it easier to enhance the robot's presence.

The above objects, as well as other objects, features and advantages, will become more apparent from the preferred embodiments described below and the accompanying drawings.

FIG. FIG. FIG. 2 is a cross-sectional view that illustrates a schematic structure of the robot. FIG. 2 is a diagram showing the hardware configuration of the robot in the basic configuration. FIG. 2 is a functional block diagram of the robot system in a basic configuration. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over a baby. 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. 1A and 1B are schematic diagrams for explaining behavioral scenes of a robot when the robot is left at home alone. FIG. 11 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. FIG. 11 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. FIG. 11 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. FIG. 11 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. FIG. 11 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. FIG. 1 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. FIG. 11 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. FIG. 11 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. FIG. 11 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. FIG. 11 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. FIG. 11 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. FIG. 11 is a schematic diagram for explaining a behavior scene when a user who is out remotely controls a robot. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person. FIG. 13 is a schematic diagram for explaining a behavioral scene when a plurality of robots are watching over an elderly person.

The robot 100 in this embodiment shares daily life with the user, sometimes being considerate of the user, sometimes striving to be helpful to the user, and actively seeking the user's affection, thereby exerting its presence as a member of the family.
The basic configuration of the robot 100 will be described below with reference to FIGS. 1 to 4, and then various behavioral scenes of the robot 100 will be described.

[Basic configuration]
Fig. 1 is a diagram showing the external appearance of a robot 100. Fig. 1A is a front view, and Fig. 1B is a side view.
The robot 100 is an autonomous robot that determines its behavior based on the external environment and its internal state. The external environment is recognized by various sensors such as a camera and a thermosensor 115. The internal state is quantified as various parameters that express the emotions of the robot 100. The robot 100's range of movement is within the interior of the owner's home. Hereinafter, a human being related to the robot 100 is referred to as a "user." Among the users, the owner or manager of the robot 100 is referred to as the "owner."

The body 104 of the robot 100 has an overall rounded shape and includes an outer skin 314 formed from a soft, elastic material such as urethane, rubber, resin, or fiber. The robot 100 may be dressed in clothing. The total weight of the robot 100 is approximately 5 to 15 kilograms, and the height is approximately 0.5 to 1.2 meters. The appropriate weight, roundness, softness, and pleasant feel make the robot 100 easy for the user to hold and make the user want to hold it.

The robot 100 includes a pair of front wheels 102 (left wheel 102a, right wheel 102b) and one rear wheel 103. The front wheels 102 are driving wheels, and the rear wheels 103 are driven wheels. The front wheels 102 do not have a steering mechanism, but the rotation speed and direction of the left and right wheels can be controlled individually. The rear wheels 103 are casters and can rotate freely to move the robot 100 forward, backward, left and right. The rear wheels 103 may be omniwheels. By increasing the rotation speed of the right wheel 102b compared to the left wheel 102a, the robot 100 can turn left or rotate counterclockwise. By increasing the rotation speed of the left wheel 102a compared to the right wheel 102b, the robot 100 can turn right or rotate clockwise.

The front wheel 102 and rear wheel 103 can be completely stored in the body 104 by a drive mechanism (rotation mechanism, link mechanism). A pair of left and right covers 312 are provided on the lower half of the body 104. The covers 312 are made of a flexible and elastic resin material (rubber, silicone rubber, etc.), which forms a soft body and can store the front wheel 102. The covers 312 are formed with slits 313 (openings) that open from the side to the front, and the front wheel 102 can be advanced through the slits 313 and exposed to the outside.

Even when the robot 100 is moving, most of the wheels are hidden by the body 104, but when the wheels are completely retracted into the body 104, the robot 100 is unable to move. That is, as the wheels are retracted, the body 104 descends and sits on the floor surface F. In this seated state, the flat seating surface 108 (ground contact surface) formed on the bottom of the body 104 abuts against the floor surface F.

The robot 100 has two arms 106. There are hands at the ends of the arms 106, but they do not have the function of grasping objects. The arms 106 can perform simple movements such as lifting, bending, waving, and vibrating when driven by actuators described below. The two arms 106 can each be controlled individually.

A facial region 116 is exposed on the front of the head of the robot 100. Two eyes 110 are provided on the facial region 116. The eyes 110 are capable of displaying images using liquid crystal elements or organic EL elements, and are devices for expressing gaze and facial expressions by moving the pupils and eyelids displayed as an image. A nose 109 is provided in the center of the facial region 116. An analog stick is provided on the nose 109, which can detect all directions, up, down, left, and right, as well as the direction in which it is pressed. The robot 100 is also provided with multiple touch sensors, which can be used to detect the head, torso,
The robot 100 can detect the touch of the user on almost all areas of the robot 100, including the buttocks and arms. The robot 100 is equipped with various sensors, such as a microphone array and an ultrasonic sensor, that can identify the direction of a sound source. The robot 100 also has a built-in speaker and can emit simple sounds.

A horn 112 is attached to the head of the robot 100. A panoramic camera 113 is attached to the horn 112, which can capture an image of the entire upper area of the robot 100 at once. The horn 112 also has a built-in thermosensor 115 (thermocamera). The horn 112 is also provided with multiple modules (not shown) for communication using infrared rays, which are installed in a ring shape facing the surroundings. This allows the robot 100 to communicate by infrared rays while recognizing its direction. Furthermore, the horn 112 is provided with an emergency stop switch, and the user can bring the robot 100 to an emergency stop by pulling out the horn 112.

FIG. 2 is a cross-sectional view that shows a schematic structure of the robot 100. As shown in FIG.
The body 104 includes a main body frame 310, a pair of arms 106, a pair of covers 312, and an outer skin 314. The main body frame 310 includes a head frame 316 and a torso frame 318. The head frame 316 is hollow and hemispherical, and forms the head skeleton of the robot 100. The torso frame 318 is rectangular and tubular, and forms the torso skeleton of the robot 100. A lower end of the torso frame 318 is fixed to a lower plate 334. The head frame 316 is connected to the torso frame 318 via a connection mechanism 330.

The torso frame 318 forms the axis of the body 104. The torso frame 318 is formed by fixing a pair of left and right side plates 336 to a lower plate 334, and supports the pair of arms 106 and the internal mechanism. Inside the torso frame 318, the battery 118, a control circuit 342, various actuators, etc. are housed. The bottom surface of the lower plate 334 forms the seating surface 108.

The torso frame 318 has an upper plate 332 at its top. A cylindrical support 319 with a bottom is fixed to the upper plate 332. The upper plate 332, the lower plate 334, the pair of side plates 336, and the support 319 constitute the torso frame 318. The outer diameter of the support 319 is smaller than the distance between the left and right side plates 336. The pair of arms 106 are integrally assembled with the ring-shaped member 340 to constitute the arm unit 350. The ring-shaped member 340 has a circular ring shape, and the pair of arms 106 are attached to the center line of the ring-shaped member 340 so as to be radially spaced apart. The ring-shaped member 340 is inserted coaxially through the support 319 and placed on the upper end surfaces of the pair of side plates 336. The arm unit 350 is supported from below by the torso frame 318.

The head frame 316 has a yaw axis 321, a pitch axis 322, and a roll axis 323. Rotation (yawing) of the head frame 316 around the yaw axis 321 allows for head swinging, rotation (pitching) around the pitch axis 322 allows for nodding, looking up, and looking down, and rotation (rolling) around the roll axis 323 allows for tilting the head left and right. The position and angle of each axis can change in three-dimensional space depending on the driving mode of the connection mechanism 330. The connection mechanism 330 is composed of a link mechanism and is driven by multiple motors installed in the torso frame 318.

The trunk frame 318 houses a wheel drive mechanism 370. The wheel drive mechanism 370 includes a front wheel drive mechanism and a rear wheel drive mechanism that respectively move the front wheels 102 and the rear wheels 103 in and out of the body 104. The front wheels 102 and the rear wheels 103 function as a "movement mechanism" that moves the robot 100. The front wheel 102 has a direct drive motor at its center. This allows the left wheel 102a and the right wheel 102b to be driven separately. The front wheel 102 is rotatably supported by a wheel cover 105, which is rotatably supported by the trunk frame 318.

The pair of covers 312 are provided to cover the body frame 318 from the left and right, and have a smoothly curved shape to give a rounded outline to the body 104. A closed space is formed between the body frame 318 and the cover 312, and this closed space serves as the storage space S for the front wheel 102. The rear wheel 103 is stored in a storage space provided at the lower rear of the body frame 318.

The outer skin 314 covers the main body frame 310 and the pair of arms 106 from the outside. The outer skin 314 is thick enough that a person can feel its elasticity, and is made of a stretchy material such as urethane sponge. This allows the user to feel a moderate softness when hugging the robot 100, and allows natural skinship, just as a person would with a pet. The outer skin 314 is attached to the main body frame 310 in a manner that exposes the cover 312. An opening 390 is provided at the upper end of the outer skin 314. The horn 112 passes through this opening 390.

Touch sensors are disposed between the main body frame 310 and the outer skin 314. Touch sensors are embedded in the cover 312. All of these touch sensors are capacitance sensors that detect touches over almost the entire area of the robot 100. The touch sensors may be embedded in the outer skin 314 or disposed inside the main body frame 310.

The arm 106 has a first joint 352 and a second joint 354, with the arm 356 between the two joints and a hand 358 at the end of the second joint 354. The first joint 352 corresponds to the shoulder joint, and the second joint 354 corresponds to the wrist joint. A motor is provided at each joint, and drives the arm 356 and the hand 358, respectively. The drive mechanism for driving the arm 106 includes these motors and their drive circuits 344.

FIG. 3 is a diagram showing the hardware configuration of the robot 100.
The robot 100 includes an internal sensor 128, a communication device 126, a memory device 124, a processor 122, a drive mechanism 120, and a battery 118. The drive mechanism 120 includes the above-mentioned connection mechanism 330 and wheel drive mechanism 370. The processor 122 and the memory device 124 are included in a control circuit 342. Each unit is connected to each other by a power line 130 and a signal line 132. The battery 118 supplies power to each unit via the power line 130. Each unit transmits and receives control signals via the signal line 132. The battery 118 is a lithium ion secondary battery, and is the power source of the robot 100.

The internal sensor 128 is a collection of various sensors built into the robot 100. Specifically, it includes a camera, a microphone array, a distance sensor (infrared sensor), a thermosensor 115, a touch sensor, an acceleration sensor, an air pressure sensor, and an odor sensor. The touch sensor covers most of the area of the body 104, and detects the user's touch based on changes in capacitance. The odor sensor is a known sensor that applies the principle that electrical resistance changes due to the adsorption of odor-causing molecules.

The communicator 126 is a communication module that performs wireless communication with various external devices. The storage device 124 is composed of non-volatile memory and volatile memory, and stores computer programs and various setting information. The processor 122 is a means for executing computer programs. The drive mechanism 120 includes multiple actuators. In addition, a display, a speaker, etc. are also installed.

The driving mechanism 120 mainly controls the wheels and the head.
The driving mechanism 120 can change the direction and speed of movement of the robot 100, as well as raise and lower the wheels. When the wheels are raised, they are completely stored in the body 104, and the robot 100 comes into contact with the floor surface F at the seating surface 108, and is in a seated state. The driving mechanism 120 also controls the arms 106.

FIG. 4 is a functional block diagram of the robot system 300.
The robot system 300 includes a robot 100, a server 200, and a plurality of external sensors 114. Each component of the robot 100 and the server 200 is realized by hardware including computing units such as a CPU (Central Processing Unit) and various coprocessors, storage devices such as memory and storage, and wired or wireless communication lines connecting them, and software stored in the storage devices and supplying processing instructions to the computing units. The computer program may be configured by a device driver, an operating system, various application programs located at higher layers thereof, and a library that provides common functions to these programs. Each block described below shows a functional block, not a hardware-based configuration.
A part of the functions of the robot 100 may be realized by the server 200, and a part or all of the functions of the server 200 may be realized by the robot 100.

Multiple external sensors 114 are installed in the house beforehand. The server 200 manages the external sensors 114 and provides the robot 100 with detection values acquired by the external sensors 114 as necessary. The robot 100 determines basic behavior based on information obtained from the internal sensor 128 and the multiple external sensors 114. The external sensors 114 are intended to reinforce the sensory organs of the robot 100, and the server 200 is intended to reinforce the processing capabilities of the robot 100. The communicator 126 of the robot 100 may periodically communicate with the server 200, and the server 200 may be responsible for processing to identify the position of the robot 100 using the external sensors 114 (see also Patent Document 2).

(Server 200)
The server 200 includes a communication unit 204 , a data processing unit 202 , and a data storage unit 206 .
The communication unit 204 is responsible for communication processing with the external sensor 114 and the robot 100. The data storage unit 206 stores various data. The data processing unit 202 executes various processes based on the data acquired by the communication unit 204 and the data stored in the data storage unit 206. The data processing unit 202 also functions as an interface between the communication unit 204 and the data storage unit 206.

The data store 206 includes a motion store 232 and a personal data store 218 .
The robot 100 has a plurality of movement patterns (motions). Various motions are defined, such as shaking the arms 106, approaching the owner while meandering, and gazing at the owner with the head tilted.

The motion storage unit 232 stores "motion files" that define the control content of a motion. Each motion is identified by a motion ID. The motion files are also downloaded to the motion storage unit 160 of the robot 100. The motion to be executed may be determined by the server 200 or by the robot 100.

Most of the motions of the robot 100 are configured as composite motions including a plurality of unit motions. For example, when the robot 100 approaches the owner, it may be expressed as a combination of a unit motion of turning toward the owner, a unit motion of approaching while raising hands, a unit motion of approaching while shaking the body, and a unit motion of sitting down while raising both hands. A combination of these four motions realizes a motion of "approaching the owner, raising hands in the middle, and finally sitting down after shaking the body." In the motion file, the rotation angle, angular velocity, etc. of the actuators provided in the robot 100 are defined in association with the time axis. Various motions are expressed by controlling each actuator over time according to the motion file (actuator control information).

The transition time from one unit motion to the next is called an "interval." The interval can be defined according to the time required to change the unit motion and the content of the motion. The length of the interval can be adjusted.
Hereinafter, settings related to behavior control of the robot 100, such as when and which motion to select, and adjustment of the output of each actuator to realize the motion, will be collectively referred to as "behavior characteristics." The behavior characteristics of the robot 100 are defined by a motion selection algorithm, a motion selection probability, a motion file, etc.

The motion storage unit 232 stores motion files as well as a motion selection table that defines the motions to be executed when various events occur. In the motion selection table, one or more motions and their selection probabilities are associated with each event.

The personal data storage unit 218 stores information about the user. Specifically, it stores master information indicating the degree of intimacy with the user and the physical and behavioral characteristics of the user. Other attribute information such as age and gender may also be stored.

The robot 100 has an internal parameter called intimacy for each user. When the robot 100 recognizes actions that show affection toward the user, such as picking up the robot 100 or talking to the user, the intimacy with that user increases. The intimacy with users who do not interact with the robot 100, users who are violent, and users the robot encounters infrequently decreases.

The data processing unit 202 includes a position management unit 208 , a recognition unit 212 , a motion control unit 222 , an intimacy management unit 220 , and a state management unit 244 .
The position management unit 208 specifies the position coordinates of the robot 100. The state management unit 244 manages various internal parameters such as various physical states such as the charging rate, internal temperature, and the processing load of the processor 122. The state management unit 244 also manages various emotion parameters indicating the emotions of the robot 100 (loneliness, curiosity, desire for recognition, etc.). These emotion parameters are constantly fluctuating. The movement destination of the robot 100 changes depending on the emotion parameters. For example, when the robot 100 feels increasingly lonely, it sets the location of the user as the movement destination.

Emotion parameters change over time. In addition, various emotion parameters also change depending on the interaction, as described below. For example, if the owner "holds" the robot, the emotion parameter indicating loneliness will decrease, and if the robot does not see the owner for an extended period of time, the emotion parameter indicating loneliness will gradually increase.

The recognition unit 212 recognizes the external environment. Recognition of the external environment includes a variety of recognitions, such as recognition of weather and season based on temperature and humidity, and recognition of shade (safety zones) based on the amount of light and temperature. The recognition unit 156 of the robot 100 acquires various types of environmental information using the internal sensor 128, performs initial processing on this information, and then transfers it to the recognition unit 212 of the server 200.

Specifically, the recognition unit 156 of the robot 100 extracts an image area corresponding to a moving object, particularly a person or an animal, from the image, and extracts a "feature vector" from the extracted image area as a set of feature amounts indicating the physical and behavioral characteristics of the moving object. A feature vector component (feature amount) is a numerical value that quantifies various physical and behavioral characteristics. For example, the width of a human eye is quantified in the range of 0 to 1 to form one feature vector component. A method for extracting a feature vector from a captured image of a person is an application of known face recognition technology. The robot 100 transmits the feature vector to the server 200.

The recognition unit 212 of the server 200 compares the feature vector extracted from the image captured by the built-in camera of the robot 100 with the feature vectors of users (clusters) preregistered in the personal data storage unit 218 to determine which person the captured user corresponds to (user identification process). The recognition unit 212 also estimates the user's emotions by performing image recognition of the user's facial expressions. The recognition unit 212 also performs user identification process on moving objects other than people, such as pet cats and dogs.

The recognition unit 212 recognizes various behaviors given to the robot 100 and classifies them into pleasant or unpleasant behaviors. The recognition unit 212 also recognizes the owner's behavior in response to the behavior of the robot 100 and classifies it into positive or negative reactions.
A pleasant or unpleasant behavior is determined based on whether the user's behavior is pleasant or unpleasant for the living organism. For example, being held is a pleasant behavior for the robot 100, and being kicked is an unpleasant behavior for the robot 100. A positive or negative reaction is determined based on whether the user's behavior indicates a pleasant emotion or an unpleasant emotion. Being held is a positive reaction that indicates a pleasant emotion for the user, and being kicked is a negative reaction that indicates an unpleasant emotion for the user.

The motion control unit 222 of the server 200 cooperates with the motion control unit 150 of the robot 100 to determine the motion of the robot 100. The motion control unit 222 of the server 200 creates a movement target point for the robot 100 and a movement route therefor. The motion control unit 222 may create multiple movement routes and then select one of the movement routes.

The motion control unit 222 selects a motion for the robot 100 from a number of motions in the motion storage unit 232. A selection probability is associated with each motion depending on the situation. For example, a selection method is defined such that when the owner performs a pleasurable action, motion A is executed with a 20% probability, and when the temperature reaches 30 degrees or higher, motion B is executed with a 5% probability.

The intimacy management unit 220 manages the intimacy level for each user. As described above, the intimacy level is registered as part of the personal data in the personal data storage unit 218. When a pleasant behavior is detected, the intimacy level management unit 220 increases the intimacy level with the owner. When an unpleasant behavior is detected, the intimacy level decreases. In addition, the intimacy level of an owner who has not been viewed for a long period of time gradually decreases.

(Robot 100)
The robot 100 includes a communication unit 142 , a data processing unit 136 , a data storage unit 148 , an internal sensor 128 , and a drive mechanism 120 .
The communication unit 142 corresponds to the communicator 126 (see FIG. 3) and is responsible for communication processing with the external sensor 114, the server 200, and other robots 100. The data storage unit 148 stores various data. The data storage unit 148 corresponds to the memory device 124 (see FIG. 3). The data processing unit 136 executes various processes based on the data acquired by the communication unit 142 and the data stored in the data storage unit 148. The data processing unit 136 corresponds to the processor 122 and the computer program executed by the processor 122. The data processing unit 136 also functions as an interface between the communication unit 142, the internal sensor 128, the drive mechanism 120, and the data storage unit 148.

The data storage unit 148 includes a motion storage unit 160 that defines various motions of the robot 100 .
Various motion files are downloaded from the motion storage unit 232 of the server 200 to the motion storage unit 160 of the robot 100. The motions are identified by motion IDs. In order to express various motions such as sitting down with the front wheels 102 folded, lifting the arms 106, rotating the two front wheels 102 in the opposite direction or rotating only one of the front wheels 102 to cause the robot 100 to rotate, trembling by rotating the front wheels 102 with the front wheels 102 folded, stopping once and looking back when moving away from the user, etc., the operation timing, operation time, operation direction, etc. of various actuators (drive mechanisms 120) are defined in a time series in the motion files.
Various data may also be downloaded to the data storage unit 148 from the personal data storage unit 218 .

The data processing unit 136 includes a recognition unit 156 and an operation control unit 150 .
The motion control unit 150 of the robot 100 decides the motions of the robot 100 in cooperation with the motion control unit 222 of the server 200. Some motions may be decided by the server 200, and other motions may be decided by the robot 100. Alternatively, the robot 100 may decide the motions, but when the processing load of the robot 100 is high, the server 200 may decide the motions. A base motion may be decided in the server 200, and additional motions may be decided in the robot 100. How the motion decision process is shared between the server 200 and the robot 100 may be designed according to the specifications of the robot system 300.

The motion control unit 150 of the robot 100 instructs the driving mechanism 120 to execute the selected motion. The driving mechanism 120 controls each actuator according to the motion file.

The motion control unit 150 can execute a motion of lifting both arms 106 as a gesture of asking to be held when a user with whom the robot has a high level of intimacy is nearby, and when the robot gets tired of being held, it can express a motion of refusing to be held by alternately rotating in the opposite directions and stopping the left and right front wheels 102 while keeping them retracted. The drive mechanism 120 drives the front wheels 102, arms 106, and neck (head frame 316) according to instructions from the motion control unit 150, causing the robot 100 to express various motions.

The recognition unit 156 of the robot 100 interprets external information obtained from the internal sensor 128. The recognition unit 156 is capable of visual recognition (visual unit), smell recognition (olfactory unit), sound recognition (auditory unit), and tactile recognition (tactile unit).

The recognition unit 156 extracts a feature vector from the captured image of the moving object. As described above, the feature vector is a collection of parameters (feature values) that indicate the physical and behavioral characteristics of the moving object. When a moving object is detected, physical and behavioral characteristics are also extracted from an odor sensor, a built-in sound collecting microphone, a temperature sensor, and the like. These features are also quantified and become feature vector components. The recognition unit 156 identifies the user from the feature vector based on known technology described in Patent Document 2, etc.

In the series of recognition processes including detection, analysis, and judgment, the recognition unit 156 of the robot 100 selects and extracts information necessary for recognition, and the recognition unit 212 of the server 200 interprets the information and performs judgment.
The recognition process may be performed by only the recognition unit 212 of the server 200, or by only the recognition unit 156 of the robot 100, or the recognition process may be performed by both of them with each other sharing the roles as described above.

When a strong impact is given to the robot 100, the recognition unit 156 recognizes this by the touch sensor and the acceleration sensor, and the recognition unit 212 of the server 200 recognizes that a "violent act" has been committed by a nearby user. When a user grabs the horn 112 and lifts the robot 100, this may also be recognized as a violent act. When a user facing the robot 100 speaks in a specific volume range and a specific frequency band, the recognition unit 212 of the server 200 may recognize that a "calling act" has been performed on the robot 100. In addition, when a temperature of about body temperature is detected, it recognizes that a "contact act" has been performed by the user, and when an upward acceleration is detected in a state where contact has been recognized, it recognizes that a "hug" has been performed. Physical contact when the user lifts the body 104 may be sensed, or a hug may be recognized by a decrease in the load on the front wheels 102.
In summary, the robot 100 acquires the user's action as physical information by the internal sensor 128, and the recognition unit 212 of the server 200 judges whether the action is comfortable or uncomfortable. The recognition unit 212 of the server 200 also executes a user identification process based on the feature vector.

The recognition unit 212 of the server 200 recognizes various responses of the user to the robot 100. Some typical responses among the various responses are associated with pleasant or unpleasant, positive or negative. Generally, most pleasant responses are positive reactions, and most unpleasant responses are negative reactions. Pleasant and unpleasant actions are related to the degree of intimacy, and positive and negative reactions affect the behavior selection of the robot 100.

In response to the interaction behavior recognized by the recognition unit 156, the intimacy management unit 220 of the server 200 changes the intimacy with the user. In principle, the intimacy with a user who performs a pleasant behavior increases, and the intimacy with a user who performs an unpleasant behavior decreases.

Each function of the server 200 is realized by loading a program for implementing that function into memory and instantiating it. The processing power of the server 200 supplements the various processes performed by the robot 100. The server 200 can be used as a resource for the robot 100. How the resources of the server 200 are used is dynamically determined according to a request from the robot 100. For example, if the robot 100 needs to continuously generate complex motions according to detection values from multiple touch sensors, the processing of the processor 122 in the robot 100 can be preferentially assigned to selecting and generating motions, and the processing for image recognition of the surrounding situation can be performed by the recognition unit 212 of the server 200. In this way, various processes of the robot system 300 can be distributed between the robot 100 and the server 200.

A single server 200 can also control multiple robots 100. In this case, each function of the server 200 is realized independently for each robot 100. For example, the server 200 may provide a recognition unit 212 for robot 100B in addition to the recognition unit 212 (instance object) for robot 100A.

Given the above basic configuration, next we will explain the implementation of the robot 100 in this embodiment, focusing in particular on the features and purpose of this implementation and the differences from the basic configuration.

[SLAM]
The robot 100 of this embodiment acquires a large number of captured images (still images) by periodically capturing images of the surroundings using the omnidirectional camera 113. The robot 100 forms a memory based on the captured images (hereinafter referred to as an "image memory").

The image memory is a collection of multiple key frames. A key frame is distribution information of feature points (feature values) in a captured image. The robot 100 of this embodiment forms key frames using a graph-based SLAM (Simultaneous Localization and Mapping) technique that uses image features, more specifically, a SLAM technique based on ORB (Oriented FAST and Rotated BRIEF) features (see Patent Document 3).

By periodically forming key frames while moving, the robot 100 forms an image memory as a collection of key frames, in other words, a distribution of image features. The robot 100 estimates its current location by comparing the key frame acquired at its current location with the many key frames it already possesses. In other words, the robot 100 performs "spatial recognition" by comparing the captured image it is actually viewing with captured images (memory) it has previously viewed, and aligning its current situation with its past memory. The image memory formed as a collection of feature points becomes what is known as a map. The robot 100 updates the map while moving and estimating its current location.

The basic configuration of the robot 100 is based on the premise that it recognizes its location using the external sensor 114, not using key frames. The robot 100 of this embodiment will be described as recognizing its location based only on key frames.

In this embodiment, the robot 100 is equipped with a "mode setting unit" for setting various modes.

<Baby monitoring>
5 to 7 are schematic diagrams for explaining behavioral scenes when a plurality of robots 100 are watching over a baby.
First, in the child's room, the baby to be watched over (hereafter referred to as the "subject") is sleeping in a bouncer (Figure 5A).
There are two robots 100 in this house. Each of the two robots 100 recognizes the presence and position of a baby (infant) through image recognition. The two robots 100 may share the baby's position through mutual communication. The two robots 100 determine the gaze points of each robot from the baby's position. The two robots 100 correct the gaze points through mutual communication so that the gaze points of each robot are the same or within a predetermined range. The two robots 100 control the movements and parts of the robots 100 so that their heads face the determined gaze points, thereby realizing an action of the two robots 100 peering at the baby ( FIG. 5B ).
One of the two robots 100, robot 100A, continues to watch over the baby by turning its head in the direction of the baby according to the baby's position. Robot 100A keeps a constant distance from the baby so as not to leave the baby's side. The other robot, robot 100B, performs actions as if it has stopped watching over the baby and started playing after a while by sharing roles with robot 100A through communication (Fig. 5C).
The mother is cooking in the kitchen, leaving the robots 100 to watch over the baby (Figure 5D).

The mother places a smartphone (communication terminal) in the kitchen (FIG. 6A). The robot 100A (watching over the baby) captures images of the baby using the omnidirectional camera 113 and continues to send the captured images to the smartphone as live video. The smartphone displays the area of the omnidirectional image in which the baby appears. At this time, the smartphone may display an image that has been corrected so that the distortion of the omnidirectional image is displayed on a flat surface.

At this point, the baby suddenly begins crying (Figure 6B).
When the robot 100B, which was playing, hears the baby's cry through the microphone (collects the sound), it approaches the baby based on the baby's location (FIG. 6C). The robot 100B executes an interference motion. The robot 100B executes an interference motion when at least one of the following conditions is met: that a sound like a cry has been collected, and that the baby is determined to be in a specific state such as crying through image analysis. The interference motion is a motion for soothing the baby, such as outputting a predetermined sound that attracts the baby's attention, outputting a sound like a lullaby, waving the arms 106, rocking the head or body, or rocking the bouncer. The interference motion distracts the baby. Also, by executing the interference motion, a third party can have the impression that the robot 100B is struggling to do something about the crying baby. Instead of or in addition to the robot B, the robot A may execute the interference motion.
The baby's crying image is shown on the mother's smartphone through live video relayed from robot 100A (Figure 6D).
The video before and after the baby starts crying (the video from a predetermined time before to a predetermined time after the baby starts crying) may be made permanent by being stored on a HDD or the like.

The mother was so surprised that she instinctively picked up her smartphone (Figure 7A).
The mother rushes to the child's room and rushes to her baby (Figure 7B).
The mother holds the baby in her arms (Figure 7C).
The robots 100 stand beside the mother and baby and look at them. The two robots recognize the respective positions of the mother and baby through image analysis, and set a gaze point from the respective positions. The two robots 100 share the gaze points through communication and correct them as necessary so that the gaze points are the same or within a predetermined range. The two robots 100 perform an action of directing their heads to their respective gaze points. As a result, the two robots 100 look at the mother and baby side by side, and the mother can have the impression that the robots 100 are concerned about the baby.
The baby falls asleep again in the mother's arms (Figure 7D).
Once the robots 100 confirm that the baby has fallen asleep, they leave the bouncer (Figure 7E).

When the robot 100 is manually set to the "monitoring mode" by the mother via an input switch, a smartphone, or the like, the robot 100 may search for the baby to start the above-mentioned monitoring action. When the robot 100 detects that the baby is in a bouncer, the mode setting unit may automatically set the robot 100 to the monitoring mode. When the robot 100 detects an infant by image recognition and there is no guardian near the infant, the robot 100 may automatically switch to the monitoring mode. When there is no guardian near the infant, for example, there may be a case where an image of a person of a certain age or older is not detected, or an image of a person associated with the infant is not detected. In addition, when multiple robots 100 are in the same room, all the robots 100 may be set to the monitoring mode, or only some of the robots 100 may be set to the monitoring mode.

The robot 100 may transition to the monitoring mode when the monitoring condition is that a "baby" is detected in the captured image, but no other users or a specific user such as the mother or father is detected.

The robot 100 may limit its range of movement to a range where it can visually recognize the baby being watched. Alternatively, the robot 100 may not leave the room where the baby is while watching over the baby. It is desirable that the robot 100 does not take its eyes off the baby while watching over the baby (for example, always captures the baby in the camera's field of view). The robot 100 pays close attention to the condition of the target person using sensors that measure the external environment, such as the omnidirectional camera 113, microphone, and temperature sensor. When the baby moves around, the robot 100 may recognize the baby's position using sensors that measure the external environment, such as the omnidirectional camera 113, microphone, and temperature sensor, and control its own direction to face the baby. In addition, when the baby comes within a predetermined distance of the robot 100, the robot 100 may reduce the points of contact with the baby by, for example, storing the wheels.

When a predetermined alert condition is met, the robot 100 executes a motion to actively engage with the baby. Alternatively, the robot may transmit "alert information" to another communication terminal such as the mother's smartphone. The alert condition can be set arbitrarily, but possible situations that may put the baby in danger include when the baby (the person being watched over) starts crying, approaches stairs, tries to go outside, plays with small objects, or falls. The motion to actively engage with the baby may be the interference motion described above.

When robot 100A transitions to the watch mode, it may notify robot 100B that it has transitioned to the watch mode. Upon receiving the notification, robot 100B may also transition to the watch mode, or may approach robot 100A or the baby. By having multiple robots 100 gather near the baby, the robots 100 can express behavior as if they are concerned about the baby.

At this time, in order not to wake the baby with the operation sounds of the robot 100, the robot 100 operates quietly by suppressing the amount of operation of the drive mechanism 120 (particularly the mechanism related to movement and posture adjustment) compared to the normal mode, so as to make less operation noise than normal. When the monitoring mode is switched to when the baby is awake, the robot B may perform a specific motion that attracts the baby's interest, such as a motion of running around the bouncer or a dance, beside the baby. For example, it may be determined that the baby is awake when a condition is met, such as the baby's voice being detected by audio analysis or an image of the baby with its eyes open being detected by image analysis. Such a motion is not performed when the baby is asleep. The robot B performs an appropriate motion depending on the state of the person being watched over. It is thought that the baby will also feel safe if many robots 100 gather around him.

The robot 100 may switch to the monitoring mode when it receives a specific voice command, such as "Watch the baby," from a specific user, such as the mother, via a microphone or the like. When the robot 100 starts monitoring, it may execute a confirmation action, such as walking around the baby, turning toward the baby, or pointing a hand 358 toward the baby, based on the baby's position detected by image analysis or the like. The confirmation action allows the mother (instructor) to determine whether the robot 100 is watching over the wrong person. If the mother is correct, she is likely to give a positive response, such as "I'll leave it to you," and if the mother is incorrect, she is likely to give a negative response, such as "No, it's not,". Therefore, the robot 100 may determine whether the person being watched over is correct or not based on the mother's response collected by the microphone after the confirmation action.

During the watching over, the robot 100 keeps its gaze on the baby, for example by peering at the baby. The baby feels reassured, believing that the robot 100 is watching over it. The mother's feelings of affection and trust for the robot 100 are stimulated as she sees the robot 100 continuing to watch over the baby with diligence. The watching over behavior also serves as an appeal to the user (witness) that "the robot 100 is working hard."

As described above, while the robot 100A is being watched, the robot 100B may play freely. However, the range of movement of the robot 100B is limited to the range in which the baby or the robot 100A can be seen. The robot 100A may transmit a live image (close-up image) of the baby to the smartphone, and the robot 100B may transmit a live image (wide-angle image) of the robot 100A watching the baby to the smartphone. It is desirable that the robot 100B further restricts the range of movement so as not to interfere with the robot 100A taking a picture of the baby. For example, the robot 100B may identify the positions of the baby and the robot 100A, and act so as not to overlap on the straight line connecting the baby and the robot 100A. In addition, when the robot 100A recognizes the image being taken and the robot 100B is no longer able to recognize the figure of the baby, the robot 100A may notify the robot 100B of a command requesting it to move. In response to the command, the robot 100B moves.

When the image analysis or audio analysis detects that the baby has entered a predetermined state, for example, when the baby starts to cry, the robot 100A notifies the user of the change in the baby's state in addition to the live video. The robot 100B may execute an interference motion. The robot 100A notifies the smartphone of a message that clearly indicates a state associated with an alert condition, such as "the baby is crying" or "the baby is starting to fuss." The robot 100B may perform a dance as an interference motion, or may play music such as a lullaby using a built-in audio player. In either case, the robot 100B comforts the baby by executing an action to distract the baby. When the mother returns (see FIG. 7B), the robot 100B stops the interference motion. Specifically, the robot 100B stops the interference motion when the "mother" is confirmed in the captured image.
The robot 100B may stop the interference motion when it recognizes through a microphone or the like that the mother has uttered a keyword (hereinafter referred to as a "completion command") instructing the completion of the watching action, such as "Thank you" or "It's okay now." At this time, it may be determined whether or not the completion command has been uttered by the person who instructed the watching action to the robot 100A, and if so, the watching action may be completed.

The robot 100 performs image recognition of the baby from the captured image. The method of detecting the "baby" may be realized by applying known face recognition technology. In addition, when the baby is moving, the robot 100 may adjust the direction of movement so that the baby is always visible regardless of whether or not it is being watched over.

Robot 100A and robot 100B may take turns watching over the baby. For example, when robot 100A has watched over the baby for 10 minutes, robot 100B may switch to a watching mode and move freely. It is considered that the baby will not get bored if multiple robots 100 take turns watching over the baby. It is also easier for third parties to get the impression that robot 100A and robot 100B are watching over the baby together.

If the robot 100 watches over the baby, even a mother who is busy raising a child can concentrate on housework with peace of mind. If you are doing housework such as laundry away from the baby, you may not immediately notice that the baby is crying. The baby may start crying in earnest before you realize that it is crying. This situation places a great burden on the mother. By having the robot 100 watch over the baby, it is possible to quickly detect signs of the baby becoming fussy or about to cry loudly.

Not only the mother but also the robot 100 participates in child rearing. It is also expected that the baby who grows up under the watchful eye of the robot 100 will develop a sense of affinity with the robot 100 in the future.

＜Home alone＞
8 to 11 are schematic diagrams for explaining behavioral scenes of the robot 100 when the robot 100 is left at home alone.
Consider a home with two robots, 100A and 100B, and a female owner. The female owner goes out to work (Fig. 8A). At this time, the female owner calls out to the nearby robot 100A, saying, "Please stay at home."
By hearing these words (collecting voice via a microphone), the robot 100A recognizes that the female owner is leaving the house and transitions to "home alone mode" (FIG. 8B).
The female owner leaves through the entrance (FIG. 8C). The robot 100A moves to the entrance and sees the female owner off by executing a predetermined motion.

Meanwhile, robot 100B in the room chases robot 100A and starts playing with it (Figure 8D). When the female owner leaves, both robots 100A and 100B may see her off, or only robots 100 whose intimacy with the female owner is equal to or greater than a predetermined value may see her off.

After a while, the front door opens while you are away (Figure 9A).
When it is determined by voice analysis or image analysis that the front door is open, the two robots 100 move to the front door in "hoping" that the female owner will return home (FIG. 9B).
However, the person who appeared (identified by image analysis, etc.) was not the female owner, but a stranger (suspicious person) carrying a large bag (Figure 9C).
At this time, the robot 100 approaches the suspicious person sufficiently and takes a picture of the suspicious person (FIG. 9D). The robot 100 transitions to the alert mode. When the robot 100 transitions to the alert mode, the robots 100 may move close to the suspicious person and continue taking pictures of the suspicious person.

The female owner is working in the office. The robot 100 transmits a captured image of the suspicious person to the female owner's smartphone (FIG. 10A).
The female owner, who is at work, finds out through her smartphone that the robot 100 has found a suspicious person (FIG. 10B). In this case, it is assumed that the person thought to be suspicious is the female owner's mother. The robots 100 do not know the female owner's mother. The woman notifies the robots 100 through her smartphone that "there is no suspicious person." The female owner may inform the robot 100 by voice that "there is no need to worry, it's your mother." The robot 100 then cancels the alert mode.
The mother cooks a home-cooked meal (Figure 10C).

The female owner (daughter) returns home and chats with her mother while eating a home-cooked meal (Figure 11A).
Robots 100 are playing nearby them (Figures 11B and 11C).

When the robot 100 finds a suspicious person (a person the robot has not seen before or whose intimacy level is below a threshold) while the owner is away from home, the robot 100 takes a picture of the suspicious person and transmits the captured image of the suspicious person to the smartphone of the female owner (specific user). With this control method, the female owner can safely leave the security of the house to the robot 100. In addition, the robot 100 detects various events that occur while the owner is away from home, such as an earthquake that breaks things, a gas leak, or a visitor such as a delivery person (a call from the intercom), through image analysis and voice recognition, and notifies the owner's smartphone of the contents of the event. The robot 100 may also record events that occurred while the owner was away as a life log, and the owner may check the events that occurred while the owner was away by checking the life log via the smartphone after returning home.

The mode setting unit of the robot 100 may set the answering mode based on an operational input from the user. The robot 100 may automatically change the setting to the answering mode when it detects a specific event, such as the user leaving the front door. Alternatively, the robot 100 may automatically change the setting to the answering mode when it is unable to visually confirm the user in the room for a certain period of time or more.

In the alert mode, the robot 100 may set a range of movement to a position where it can photograph a suspicious person. This is to ensure that the suspicious person's actions are not overlooked. In addition, the robot 100 may move away from the suspicious person to prevent violence from the suspicious person. After reporting the suspicious person, when the robot 100 receives a notification from the user that the person is not suspicious, the robot 100 cancels the alert mode and returns to the answering machine mode. After this, the robot 100 may interact with the unidentified person (former suspicious person) in the usual manner. In addition, the robot 100 may remember the appearance of the unidentified person and manage parameters such as intimacy. The robot 100 may become attached to the unidentified person. In the example shown in FIG. 8 to FIG. 11, the mother is initially wary of the robot 100, but after the female owner (daughter) sends an approval notice to the robot 100, the robot 100 begins to cling to the mother. The mother can suddenly feel that she has been accepted and welcomed by the robot 100.

In the away mode, a user (owner) in a remote location may send an instruction to check the interior of the house to the robot 100 via a smartphone. When the robot 100 receives the instruction, it patrols the room according to a map generated based on SLAM. At this time, the robot 100 may send captured images to the user's smartphone, or may notify the user of the presence or absence of any abnormal events. By sending an instruction to check the interior of the house, the user can check the state of their home at any time.

<Remote Control>
12 to 14 are schematic diagrams for explaining behavioral scenes when a user who is out remotely controls the robot 100. FIG.
Two robots 100A and 100B are watching the house while the person is away. There is also a cat in the house (Fig. 12A).
Meanwhile, the rest of the family goes out to town, leaving the robots 100 and the cat behind (Figure 12B). The boy looks gloomy (Figure 12C).
The boy takes out his smartphone (mobile device) and begins to operate it (Figure 12D).

Two robots 100 are playing in the empty house (FIG. 13A).
When the robot 100A starts to move, the robot 100B follows the robot 100A (FIG. 13B). The robots 100A and 100B play a game of tag.
The mother is concerned about her son's condition (Figure 13C).
When the boy leaves, he is worried that his cat does not seem very energetic (Figure 13D).

The boy sends a command to the robot 100 from his smartphone saying, "Check on the cat." The robots 100A and 100B move to photograph the location or object indicated in the command, and take pictures as appropriate. The captured images of the robots 100A and 100B are sent to the smartphone. The cat is playing happily in the cat tree (FIG. 14A).
The family feels relieved to see the cat looking healthy (Figure 14B).
The robot 100 takes a picture of a cat playing on a cat tower. The robot 100 takes a close-up picture of the cat, and the cat looks at the robot 100 (FIGS. 14C and 14D).

In this way, the user can send various instructions to the robot 100 from the smartphone. In particular, the user can command the robot 100 to check the room. When the command "Check on the cat" is sent as in the above embodiment, the robot 100 detects an object corresponding to the "cat" from the captured image and transmits the captured image with the cat at the center to the smartphone. Such commands may be voice commands or may be input from a graphical user interface provided in the smartphone. The user may be able to operate the robot 100 like a radio-controlled car (hereinafter, such an operation method is referred to as "remote operation").

The images captured by the robot 100 are displayed on a smartphone, and the user may enlarge and display a portion of the captured image that is being live-streamed to the smartphone that the user particularly wishes to see. The robot 100 may transmit the panoramic image itself to the smartphone, and the user may check what the robot 100 "saw" through the panoramic image.

The robot 100 can recognize not only species such as humans and cats, but also individual levels such as "who" and "which". Black cats and white cats, and big cats and small cats are treated as different cats. The robot 100 also learns the names of cats based on the user's calls to the cat. For example, a model may be generated by machine learning that extracts an image of a cat from the name of the cat recognized by voice analysis and the captured image when the name of the cat is recognized, and outputs the name of the cat using the image of the cat as input. By using such a model, even if there are multiple cats, if the user issues a command specifying the name of the cat, the robot 100 can select the specified cat as the subject of the photo. The user may register the name of the cat and a photo of the cat in advance via a smartphone, etc.

When the user remotely controls the robot 100A, the robot 100B may move along with the robot 100A. The captured images from the robot 100A and the robot 100B are sent to the user's smartphone. When the robot 100A captures an image of the cat, the robot 100B, which is near the robot 100A, also captures an image of the cat using the omnidirectional camera 113. By simply remotely controlling the robot 100A, the user can obtain captured images of the cat not only from the robot 100A but also from the robot 100B. By remotely controlling only the robot 100A, the user can also indirectly remotely control the robot 100B. This is because the robot 100B has a "following function."

A user can set the robot 100 to a remote control mode from a smartphone. A user can also end the remote control mode of the robot 100 from the smartphone. The robot 100 in the remote control mode may change the display of the eyes 110. For example, the robot 100 may change the eyes 110 to red, or may display an icon on the eyes 110 to visually express that it is "controlled (remotely controlled)". When the remote control mode ends, the robot 100 returns the eyes 110 to the normal black eye display and returns to the location at the start of the remote control mode. When the remote control mode ends, the robot 100 may sit down, or may vigorously shake its head to show that it has "escaped control and regained its sense of self".

In remote mode, the robot 100 does not change emotion parameters or intimacy.

When switching to the remote operation mode, the robot 100 authenticates the person who has requested the remote operation. Only when the authentication is successful, the robot 100 switches to the remote operation mode. The authentication may be a general authentication method using an account name and a password, or an electronic certificate may be registered in advance in a device used for remote operation, and only access from a device having the electronic certificate may be permitted. Furthermore, authentication may be performed by confirming that the person operating the mobile terminal is the owner of the robot 100 using a camera or a microphone provided in a mobile terminal such as a smartphone. Furthermore, when the remote mode is requested, a user in the vicinity may be asked to approve the switch to the remote mode. In this way, by sufficiently confirming that the person who has requested the remote mode is the owner of the robot 100, unintended remote operation by a third party can be prevented.

<Watching over the elderly>
15 to 18 are schematic diagrams for explaining behavioral scenes when a plurality of robots 100 are watching over an elderly person.
An elderly father lives alone. His only daughter lives far away from him. There are two robots 100 in the father's house (Fig. 15A).
In the living room at home, the daughter is looking at her smartphone (Fig. 15B). The robots 100 record their life with their father as a life log. The life log is a diary that shows what is happening around the father in a way that respects his privacy.
The daughter checks the life log on her smartphone (Fig. 15C), which contains simple information such as what time her father woke up and whether he had breakfast.
Meanwhile, the father is holding the robot 100 and loving it ( FIG. 15D ). If the father's permission is recognized through image analysis, audio analysis, communication, or the like, the robot 100 may transmit a captured image of the robot 100 playing with the father to the daughter's smartphone. For example, when the robot 100A is held by the father, the robot 100B may act as a cameraman and capture an image of the robot 100A and the father, and transmit the captured image to the daughter's smartphone.

The daughter feels reassured when she sees her father enjoying life with robot 100 in the living room (Figure 16A).

Next, imagine a scenario in which the daughter is working in the office. She suddenly takes out her smartphone and checks her father's life log (Figure 16B).
Let us suppose that there were almost no records about the father in this life log. The daughter suddenly becomes worried about her father (Figure 16C).
The daughter calls her father from the office hallway (Figure 16D).

The father's cheerful voice soon came over the phone (Figure 17A).
The father was at the inn and had just gotten out of the bath when the phone rang (Figure 17B).
The father tells his daughter that he was at the hot spring with friends (Figure 17C).
The daughter was not aware that her father was going to the hot springs, so she feels relieved when she finds out the details (Figure 17D).

The conversation between father and daughter continues (Fig. 18A, Fig. 18B). Two robots 100 are staying at home while the father is away (Fig. 18C).

The robot 100 records various events that occur in the life of the father (the elderly person being monitored) as a life log. This life log records the father's regular daily actions, such as the time he woke up and whether he did his usual exercises today. Through the life log provided by the robot 100, the daughter can check whether her father is living his life as usual.

When an event indicating an interaction between the father and the robot 100 does not occur (is not detected) for a predetermined period of time or more, the robot 100 may send an abnormality notification to the daughter's smartphone. For example, the abnormality notification may be sent when the father has not touched the robot 100 for a while, or when the father is still lying down in the afternoon. Whether the father is lying down can be determined by image analysis, temperature sensor analysis, or the like. The robot 100 may behave in such a way as to increase opportunities for the father to be seen by actively moving around the room.

The life log, which abstracts information, allows the daughter to check on her father's daily life while protecting his privacy. When watching over an elderly person, the robot 100 does not need to constantly visually monitor the elderly person. The elderly person lives an independent life, and the robot 100 should basically be able to act autonomously. It is considered preferable for the elderly person and the robot 100 to maintain a reasonable distance. The robot 100 may record a life log at all times, not just when watching over the elderly person, if the user so desires. The robot 100 only needs to notify the daughter when something unusual occurs in the elderly person's life.

Expressions of jealousy
People cannot remain indifferent to the affection shown to them. When multiple people show affection to the same person, jealousy is likely to arise. Therefore, when the robot 100A and the robot 100B live with a user, one of the robots 100 may take a behavior that makes the user feel jealous of the other robot 100.

For example, when the robot 100B is being held by the user, the state management unit 244 increases the recognition desire value (desire to be recognized), which is a type of emotional parameter of the robot 100A. When the recognition desire value increases, the operation control unit 150 of the robot 100A asks the user to hold it. The robot 100A may stare at the user, may approach the user, or may wander around the user's vicinity to ask to be held. When the user walks, the robot 100A may move following the user. The increase in the recognition desire value is externally expressed as a behavioral characteristic of the robot 100, as if it were jealousy aroused.

When the user continues to hold robot 100B, robot 100A may actively express "strong jealousy" by clinging to the user. Alternatively, robot 100A may passively express jealousy by moving away from the user and looking at the user from afar. Such manner of expressing jealousy is determined according to the personality of each robot 100 (initial personality or cultivated personality). Jealousy may be expressed by "sulking," or for a certain period of time after the occurrence of an event that makes the robot jealous, the robot may act in such a way that it moves away even if the user approaches. Robot 100 may act in a "sulking" manner by temporarily refusing to be held.

The robot 100 may behave in a manner that makes it feel jealous as the level of intimacy increases. For example, assume that the robot 100A has a high level of intimacy with the user P1 and a relatively low level of intimacy with the user P2. In this case, when the user P1 holds the robot 100B, the robot may have a higher need for recognition than when the user P2 holds the robot 100B. This control method makes it possible for the robot to behave as if it has a possessive desire to monopolize the affection of a user that it particularly likes.

The robot 100 may notify other robots 100 of its own state (emotion parameters, intimacy, events, etc.) (hereinafter, such notifications are referred to as "state notifications"). Based on the state notifications, the robots 100 may be able to understand each other's states. For example, the robot 100A notifies the robot 100B of states such as "being held by the user," "being petted by the user," and "having been changed by the user," so that the robot 100B can understand the state of the robot 100A. While the recognition request value (desire to be recognized) of the robot 100A decreases due to an event such as being held, when the recognition request value of the robot 100B is in a high state above a threshold, the robot 100B shows a unique behavioral characteristic that expresses jealousy.

In this embodiment, the state management unit 244 of the server 200 collectively manages the emotion parameters of each robot 100. In this case, the state management unit 244 may internally notify the robot 100B of the value of the emotion parameter of the robot 100A, or may change the emotion parameter of the robot 100B based on the emotion parameter of the robot 100A. In this case, the emotion parameter of the robot 100B may be changed on the condition that the robot 100A is in a position visible from the robot 100B. This is to allow the robot 100B, which is close to the robot 100A, to visually sense the change in the emotion of the robot 100A and change its own emotion parameter.

In addition to emotion parameters, robot 100A may notify robot 100B by short-distance wireless communication such as infrared. In this case, robot 100B can only receive status notification from robot 100A when it is near robot 100A and there are no obstacles blocking its line of sight, so it is possible to express a state in which "the robot can sense the status only when it is close enough to be seen." Robot 100B may also detect events such as robot 100A being held or stroked from captured images. Robot 100B may change emotion parameters when it recognizes a pleasant behavior toward robot 100A through image recognition.

The robot 100 may not only be jealous of other robots 100, but also of pets or children. For example, the robot 100's need for recognition value may be increased when the user holds a cat. The user may also need to be considerate, such as loving a pet when the robot 100 is not looking, or loving both the pet and the robot 100 equally, in order to avoid making the robot 100 jealous. By actively creating opportunities for the user to consider the feelings of the robot 100, the user's attachment to the robot 100 can be deepened.

The robot 100 of this embodiment can express its feelings by its actions without conversation. The robot 100A may receive the feelings (emotion parameters) of the robot 100B. The server 200 may reflect the change in the emotion parameters of the robot 100B in the behavior of the robot 100A. For example, when the recognition desire value of the robot 100B suddenly drops (when it is considered that something good has happened to the robot 100B), the robot 100A may move closer to the robot 100B. When the robot 100A is notified that the recognition desire of the robot 100B has been satisfied, the robot 100A may increase its own recognition desire value (a desire to be recognized). According to this control method, even when the user secretly pets the robot 100B, the robot 100A can realize a behavioral expression as if it has sensed something. In other words, a mysterious behavioral expression can be realized as if the robots 100 are communicating with each other telepathically.

<Multiple robots staring at the same thing>
The robots 100A and 100B may continue to gaze at the same object. For example, when the robot 100A gazes at a relaxing user, the robot 100B may also gaze at the same user. The robot 100B may detect that the robot 100A gazes at the relaxing user (the head of the robot 100A faces the direction of the user) through communication with the robot 100A or through image analysis. The robot B may move close to the robot A and then gaze at the user. The user can sense the multiple gazes and therefore feel that the robots 100 are very interested in him/her. When the user does not pay attention to the robots 100 for a long time, the robots 100A and 100B may silently seek "interaction" by gazing at the user at the same time.

Assume that the robot 100A has a high intimacy level with the user P1 that is equal to or higher than a predetermined value, and the robot 100B also has a high intimacy level with the user P1 that is equal to or higher than a predetermined value. The robot 100 has more opportunities to stare at the user with a higher intimacy level. For this reason, in the above situation, the robot 100A and the robot 100B may have an opportunity to stare at the user P1 at the same time. The robot 100A may notify the robot 100B of its state that it is staring at the user P1. When the robot 100B receives a state notification from the robot 100A that "(the robot 100A) is also staring at the user P1" while it is staring at the user P1, it may perform a specific motion such as a surprised motion or turning its gaze in the direction of the robot 100A to create a "coincidence". In addition, when both robots are staring at the same user, the robots 100A and 100B may approach each other and perform a motion to stare at the user P1 side by side. By moving both robots 100 to a position where the user's face is as large as possible and staring at the user side by side, strong pressure can be exerted on the user.

In addition, when an insect gets into the house (when an insect is detected in the house by image analysis or sound analysis, etc.), robot 100A and robot 100B may share the target insect, and robot 100A and robot 100B may simultaneously stare at the insect, thereby expressing an extraordinary interest in the insect. Furthermore, by gazing at each other, robot 100A and robot 100B can realize a behavioral expression as if the robots 100 are showing each other something.

When the value of the emotion parameter indicating the curiosity of the robot 100A exceeds a threshold, the robot 100A may notify the robot 100B of its state of "increased curiosity." At this time, the robot 100B may perform a motion as if it wants to know the source of the robot 100A's curiosity, such as approaching the robot 100A and moving its hands to touch the robot 100A. The robot 100A may notify the robot 100B of an object of interest in the panoramic image and its direction. When the robot 100B receives this notification, the robot 100B may gaze at the same object as the object of the robot 100A by directing its head or gaze toward the same object.

<Appeal>
When a predetermined appeal condition is met, for example, when the recognition desire value exceeds a threshold, the robot 100 performs a strong appeal behavior toward the user. The appeal behavior here refers to an action that actively seeks the user's interaction with the robot 100, such as touching, calling out to the user, or holding the user. For example, assume that the user is doing exercise such as yoga indoors. When the user is engrossed in yoga, and the appeal condition of the robot 100 is met, the robot 100 may perform an appeal behavior such as continuing to stare at the user or wandering around the user, to request the user to stop yoga.

FOLLOW-UP
As described above, when the robot 100A moves, the robot 100B may move to follow the robot 100A from behind while keeping a constant distance from the robot 100A. The robot 100A may similarly move to follow a user or a pet. For example, when a dog is following a user, the robot 100A may follow the dog or the user. When the robot 100A is following a dog, the robot 100B may follow the robot 100A. The distance between the robot and a target to be followed (e.g., a dog following a user) may be equal to the distance between the target to be followed and a target to be followed by the target to be followed (e.g., a user), or may be shorter than the distance by a predetermined length, or may be longer than the distance by a predetermined length.

When the robot 100 detects that the moving objects Q1 and Q2 are moving in the same direction for a predetermined period of time or more, it determines that "following" is occurring. With this control method, when following is occurring, it is possible for the robot 100 to express a behavior that conveys its instinct to want to follow. The sight of multiple robots 100 performing following behavior is considered to be effective in appealing to the user how cute the robots 100 are.

The robot 100 may execute the following behavior on the condition that the emotional parameter of the robot 100, for example, an emotional parameter indicating curiosity, is equal to or lower than a threshold value. This control method enables the robot 100 to express the behavior of executing a following behavior toward another robot when the robot is bored due to fading curiosity, and not executing the following behavior when the robot is curious. When the robot executes the following behavior, the robot may end the following behavior on the condition that the robot's curiosity is increased to or higher than a threshold value due to various events. The source of the behavior of an autonomous robot is a predetermined parameter indicating an internal state. In this embodiment, the parameter indicating curiosity contributes greatly to the source of the behavior, but if there is little change in the external environment, the curiosity parameter may approach 0. In such a case, the robot can actively change its own parameters by piggybacking on the behavior of the other robot, rather than waiting for the parameter to change.

As described above, the multiple robots 100 may take the same action, such as following behavior, or may change their behavioral characteristics while being influenced by each other's actions. In order to increase the cooperation and interlocking of the multiple robots 100, it is desirable for the robots 100 to be able to grasp each other's states within the server 200 or among themselves. Having grasped the state of robot 100A, robot 100B may behave in sync with the state of robot 100A, or may behave independently without being synchronized. An example of robot 100B's behavior in sync with robot 100A is when robot 100B performs a motion of the same category on the same object as robot 100A, such as robot 100B looking at the same thing that robot A is looking at.

When multiple robots 100 behave cooperatively, the user is likely to find the robots 100 cute. The user may want to take a picture of the robots 100 behaving cooperatively. When the omnidirectional camera 113 recognizes that the user is holding the camera, the robot 100 may maintain at least one of its own behavior and state until the user finishes taking pictures. In this case, instead of maintaining the behavior or state, the robot 100 may select a specific motion. For example, the robot 100 may turn its body toward the user, or may cooperate with the user in taking pictures by temporarily stopping the cooperative behavior. In this way, the robot 100 may temporarily stop its operation when it detects a photographing behavior. In addition, the robot 100 may pose or temporarily stop its behavior when it detects a user's photographing behavior, not only when it is acting cooperatively. According to such a control method, it becomes easier for the user to upload a cute photograph of the robot 100 to a social networking service (SNS) or the like. It is also believed that it will be easier to take a variety of best shot images of the robot 100 interacting with various household items (pets, children, toys, furniture, etc.).

<Structure of the outer skin>
The outer skin 314 of the robot 100 is constructed by housing an elastic base material in a cloth bag. The bag may be made of a soft material that is warm and pleasant to the touch for the user. The base material is preferably a flame-retardant material, and more preferably a material that emits self-extinguishing gas when heated. For example, the base material is made of a flame-retardant sponge. The outer skin 314 is formed so that the flame-retardant base material is wrapped in a cloth bag, so that even if the cloth bag ignites, the base material emits self-extinguishing gas, preventing the spread of fire in the cloth bag. The threshold temperature at which the base material generates self-extinguishing gas is preferably lower than the ignition temperature of the cloth. In this case, when the cloth becomes hot, the self-extinguishing gas is generated before the cloth ignites, so that the cloth can be prevented from catching fire. By making the outer skin 314 a double structure of a flame-retardant base material and a soft bag, the warmth of the robot 100 and safety against high temperatures can be achieved at the same time.

The present invention is not limited to the above-described embodiment and modifications, and can be embodied by modifying the components without departing from the spirit of the invention. Various inventions may be formed by appropriately combining multiple components disclosed in the above-described embodiment and modifications. In addition, some components may be deleted from all the components shown in the above-described embodiment and modifications.

Although the robot system 300 has been described as being composed of one or more robots 100 and one server 200, some of the functions of the robot 100 may be realized by the server 200, or some or all of the functions of the server 200 may be assigned to the robot 100. One server 200 may control multiple robots 100, or multiple servers 200 may cooperate to control one or more robots 100.

A third device other than the robot 100 and the server 200 may also be responsible for some of the functions. The collection of functions of the robot 100 and the server 200 described in FIG. 4 can be understood as a single "robot" in a broader sense. How to allocate the multiple functions required to realize the present invention to one or more pieces of hardware can be determined in consideration of the processing capacity of each piece of hardware, the specifications required for the robot system 300, etc.

As mentioned above, a "robot in the narrow sense" refers to the robot 100 that does not include the server 200, while a "robot in the broad sense" refers to the robot system 300. It is conceivable that many of the functions of the server 200 may be integrated into the robot 100 in the future.

Claims (11)

A motion control unit that selects the motion of the robot;
A drive mechanism that executes the motion selected by the motion control unit;
A recognition unit that determines whether a target person satisfies a predetermined monitoring condition;
The robot includes a mode setting unit that sets the robot to a monitoring mode for the target person when the monitoring condition is met. It has a head,
2. The robot according to claim 1, wherein the driving mechanism executes a motion of keeping the head facing the direction of the target person during the monitoring mode. The robot according to claim 2, in the monitoring mode, shares the position of the subject with another robot, and faces the head toward a gaze point that is the same as that of the other robot and is determined based on the subject, or a gaze point within a predetermined distance from the gaze point of the other robot. The robot according to claim 2 or 3, wherein in the monitoring mode, at least one of the other robot and the robot itself is configured to turn the head toward the subject. The robot according to any one of claims 1 to 4, which controls the distance to the object to within a predetermined distance during the monitoring mode. The robot according to any one of claims 1 to 5, wherein during the monitoring mode, the amount of movement of the mobility mechanism is reduced compared to at least one mode when the monitoring condition is not satisfied. The robot according to any one of claims 1 to 6, characterized in that, when it is determined that the object satisfies a predetermined condition during the monitoring mode, a motion is executed for the object. The robot according to any one of claims 1 to 7, characterized in that, when it is determined that the object satisfies a predetermined condition during the monitoring mode, a motion is executed for the object. The robot according to any one of claims 1 to 9, characterized in that the mode setting unit is configured to set the robot to the monitoring mode when, in addition to the monitoring condition, a condition that no person of a certain age or older is detected in the vicinity is satisfied. The robot according to any one of claims 1 to 9, characterized in that the mode setting unit is configured to set the robot to the monitoring mode when, in addition to the monitoring condition, a condition that a person associated with the target person is not detected during the week is satisfied. The robot according to any one of claims 1 to 10, characterized in that it is provided with a communication unit that transmits a captured image of the subject to a predetermined communication terminal in the monitoring mode.