JP7189620B2 - An image processing device that corrects the distortion of a celestial image and a robot equipped with it - Google Patents

Description

The present invention relates to technology for supporting image recognition based on celestial images.

Humans keep pets for comfort. On the other hand, many people give up pets for a variety of reasons, such as not having enough time to take care of pets, lack of living environments that allow pets, allergies, and the pain of bereavement. If there is a robot that can play the role of a pet, it may be possible to give people who cannot have pets the comfort that pets give them (see Patent Document 1).

JP-A-2000-323219 JP 2013-198062 A

The premise of the behavior of living things is the cognition of the external world. In order to realize human-like and biological behavior in robots, it is necessary to make robots accurately recognize the external world. The inventors of the present invention conceived that if a robot is equipped with an omnidirectional camera (see Patent Document 2) capable of capturing an image of the celestial sphere centering on the user at once, the cognitive ability of the robot can be dramatically expanded. However, since the celestial image obtained by the omnidirectional camera is highly distorted, image recognition processing for the celestial image is difficult.

The present invention is an invention that was completed based on the above-mentioned ideas of the present inventors and the recognition of problems associated therewith, and the main purpose thereof is to provide a technique for correcting distortion in celestial images, in particular, correcting distortion in celestial images. To provide a technology for assisting a robot in recognizing the external world based on a celestial image.

An image processing apparatus according to an aspect of the present invention includes an image acquisition unit that acquires a celestial image as a captured image of a celestial sphere centered on a camera; an image converter that obtains a plurality of projected images by projecting pixels of the images.

An image processing apparatus according to another aspect of the present invention includes an image acquisition unit that acquires a celestial image as a captured image of a celestial sphere centered on a camera, and a plurality of cylindrical images that surround the camera at different angles with respect to the camera. An image conversion unit that obtains a plurality of projection images by projecting pixels of the celestial image.

A robot according to one aspect of the present invention has a camera capable of simultaneously capturing substantially the entire area above the robot, and a celestial image, which is a captured image of the celestial sphere covering the upper part of the robot, acquired from the camera, and the celestial image is converted into a polyhedral image. An image conversion unit for conversion and a recognition unit for image recognition of an external object from a polyhedral image are provided.
The image conversion unit sets an image whose imaging surface is the polyhedron covering the upper part of the robot as a polyhedral image, and converts the celestial image into a polyhedral image by projecting the pixels of the celestial image onto one of the surfaces of the polyhedron.

According to another aspect of the present invention, a robot acquires a camera capable of simultaneously capturing substantially the entire area above the robot, and a celestial image, which is a captured image of a celestial sphere surface covering the upper part of the robot, from the camera, and obtains the celestial image from the camera. An image conversion unit that converts the image into an image, and a recognition unit that recognizes an external object from the multiple cylindrical images are provided.
The image transforming unit forms a cylindrical image by projecting the pixels of the celestial image onto the side surface of a cylinder surrounding the robot, and transforms a set of a plurality of cylindrical images corresponding to a plurality of relative angles between the cylinder and the celestial sphere into the multiple cylindrical image. The celestial image is converted to a multiple cylindrical image by setting as .

A measurement device according to an aspect of the present invention includes first and second cameras capable of capturing celestial images centering on the cameras, acquiring the celestial images from the first and second cameras, respectively, and obtaining different images for the cameras. an image conversion unit that acquires a plurality of projected images by projecting pixels of a celestial image onto a plurality of planes that face each other at an angle; a recognition unit that recognizes an external object from the plurality of projected images; A first projected image generated based on a celestial image captured by a first camera including an object, and a second projected image generated based on a celestial image captured by a second camera including the same external object. a distance measuring unit that measures the distance to an external object by triangulation based on the distance.

According to the present invention, the celestial image can be subjected to suitable correction for image recognition.

It is a front external view of a robot. It is a side external view of a robot. It is a cross-sectional view schematically showing the structure of the robot. 1 is a configuration diagram of a robot system; FIG. It is a conceptual diagram of an emotion map. It is a hardware block diagram of a robot. 1 is a functional block diagram of a robot system; FIG. FIG. 4 is a schematic diagram for explaining a method of generating a celestial image; FIG. 4 is a schematic diagram of a celestial image; FIG. 3 is a schematic diagram for explaining the relationship between a cylindrical surface and a celestial sphere in the equirectangular projection; FIG. 4 is a schematic diagram for explaining a method of projecting an image from a celestial image onto a single cylindrical image using equirectangular projection. 1 is a perspective view of an orthorhombic octahedron; FIG. It is a developed view of a rhombicoctahedron. It is a side cross-sectional view of the rhombicoctahedron and the celestial sphere. FIG. 10 is a schematic diagram for explaining partial clipping of an enlarged square plane; FIG. 4 is a schematic diagram for explaining a polyhedral projection method; 4 is a flow chart showing an image processing process in the first embodiment; FIG. 3 is a first schematic diagram for explaining the multiple cylindrical projection method; FIG. 4 is a second schematic diagram for explaining the multiple cylindrical projection method; FIG. 2 is a conceptual diagram of a multiple cylindrical image generated by multiple cylindrical projection; 9 is a flow chart showing an image processing process in the second embodiment; It is a front external view of the robot in 3rd Embodiment. It is a side external view of the robot in 3rd Embodiment. FIG. 4 is a schematic diagram for explaining a method of measuring the distance from the robot to an external object; It is a perspective view of a truncated pentagonal pyramid. It is a schematic diagram when a truncated pentagonal pyramid (t1) is viewed from above. It is a schematic diagram when a truncated pentagonal pyramid (t2) is viewed from above. FIG. 4 is a developed view of a truncated pentagonal pyramid; FIG. 4 is a schematic diagram for explaining rotation of an enlarged zenith plane; It is a schematic diagram which shows joining of several enlarged side surfaces.

FIG. 1(a) is a front external view of the robot 100. FIG. FIG. 1B is a side external view of the robot 100. FIG.
The robot 100 in this embodiment is an autonomous robot that determines actions and gestures based on the external environment and internal state. The external environment is recognized by various sensors such as cameras and thermosensors. The internal state is quantified as various parameters that express the robot's 100 emotions. These will be described later.

In principle, the robot 100 has an action range within the owner's house. Hereinafter, a person involved in the robot 100 will be referred to as a "user", and in particular, a user who will be a member of the household to which the robot 100 belongs will be referred to as an "owner".

A body 104 of the robot 100 has an overall rounded shape and includes an outer skin made of a soft and elastic material such as urethane, rubber, resin, or fiber. The robot 100 may be dressed. By making the body 104 round, soft, and pleasant to the touch, the robot 100 provides the user with a sense of security and a pleasant touch.

Robot 100 has a total weight of 15 kilograms or less, preferably 10 kilograms or less, and more preferably 5 kilograms or less. By 13 months of age, the majority of babies begin walking on their own. The average weight of a 13-month-old baby is just over 9 kilograms for boys and just under 9 kilograms for girls. Therefore, if the total weight of the robot 100 is 10 kg or less, the user can hold the robot 100 with approximately the same effort as holding a baby who cannot walk alone. Babies under 2 months of age weigh less than 5 kilograms on average for both sexes. Therefore, if the total weight of the robot 100 is 5 kg or less, the user can hold the robot 100 with the same effort as holding an infant.

Various attributes such as appropriate weight, roundness, softness, and good touch realize the effect that the user can easily hold the robot 100 and want to hold it. For the same reason, it is desirable that the height of the robot 100 is 1.2 meters or less, preferably 0.7 meters or less. It is an important concept for the robot 100 in this embodiment that it can be held.

The robot 100 has three wheels for traveling on three wheels. As shown, it includes a pair of front wheels 102 (left wheel 102a, right wheel 102b) and one rear wheel 103. As shown in FIG. The front wheels 102 are driving wheels and the rear wheels 103 are driven wheels. The front wheels 102 do not have a steering mechanism, but can be individually controlled in rotation speed and rotation direction. The rear wheel 103 is a so-called omni wheel, and is rotatable to move the robot 100 forward, backward, leftward, and rightward. The robot 100 can turn left or rotate counterclockwise by making the rotation speed of the right wheel 102b greater than that of the left wheel 102a. By increasing the rotation speed of the left wheel 102a more than the right wheel 102b, the robot 100 can turn right or rotate clockwise.

The front wheels 102 and rear wheels 103 can be completely housed in the body 104 by means of drive mechanisms (rotating mechanisms, link mechanisms). Most of the wheels are hidden by the body 104 even during running, but when the wheels are completely housed in the body 104, the robot 100 cannot move. That is, the body 104 descends and seats on the floor surface F as the wheels are retracted. In this seated state, a flat seating surface 108 (grounding bottom surface) formed on the bottom of the body 104 contacts the floor surface F. As shown in FIG.

Robot 100 has two hands 106 . The hand 106 does not have the function of grasping an object. The hand 106 can be raised, shaken, or vibrated for simple actions. The two hands 106 are also individually controllable.

The eye 110 can display an image using a liquid crystal element or an organic EL element. The robot 100 is equipped with various sensors such as a microphone array and an ultrasonic sensor that can identify the sound source direction. It also has a built-in speaker that can emit simple sounds.

A horn 112 is attached to the head of the robot 100 . Since the robot 100 is lightweight as described above, the user can lift the robot 100 by grasping the horns 112 . An omnidirectional camera is attached to the horn 112 so that the entire upper area of the robot 100 can be imaged at once.

FIG. 2 is a cross-sectional view schematically showing the structure of the robot 100. As shown in FIG.
As shown in FIG. 2 , the body 104 of the robot 100 includes a base frame 308 , a body frame 310 , a pair of resin wheel covers 312 and an outer skin 314 . The base frame 308 is made of metal, constitutes the axis of the body 104, and supports the internal mechanism. The base frame 308 is configured by vertically connecting an upper plate 332 and a lower plate 334 with a plurality of side plates 336 . Sufficient spacing is provided between the plurality of side plates 336 to allow for ventilation. Inside the base frame 308, the battery 118, the control circuit 342 and various actuators are accommodated.

Body frame 310 is made of a resin material and includes head frame 316 and body frame 318 . The head frame 316 has a hollow hemispherical shape and forms the head skeleton of the robot 100 . The torso frame 318 has a stepped tubular shape and forms the torso skeleton of the robot 100 . A torso frame 318 is fixed integrally with the base frame 308 . The head frame 316 is attached to the upper end of the body frame 318 so as to be relatively displaceable.

The head frame 316 is provided with three axes, a yaw axis 320, a pitch axis 322 and a roll axis 324, and an actuator 326 for rotationally driving each axis. Actuator 326 includes multiple servo motors to drive each axis individually. The yaw axis 320 is driven for the swing motion, the pitch axis 322 is driven for the nod motion, and the roll axis 324 is driven for the tilt motion.

A plate 325 that supports the yaw axis 320 is fixed to the top of the head frame 316 . A plurality of ventilation holes 327 are formed in the plate 325 to ensure ventilation between the upper and lower sides.

A metal base plate 328 is provided to support the head frame 316 and its internal mechanisms from below. The base plate 328 is connected to the plate 325 via a cross link mechanism 329 (pantograph mechanism) and is connected to an upper plate 332 (base frame 308 ) via a joint 330 .

Torso frame 318 houses base frame 308 and wheel drive mechanism 370 . Wheel drive mechanism 370 includes a pivot shaft 378 and an actuator 379 . A lower half portion of the body frame 318 is made narrow in order to form a storage space S for the front wheel 102 between the wheel cover 312 and the wheel cover 312 .

Outer skin 314 is made of urethane rubber and covers body frame 310 and wheel cover 312 from the outside. Hand 106 is integrally molded with skin 314 . An opening 390 for introducing outside air is provided at the upper end of the outer skin 314 .

FIG. 3 is a configuration diagram of the robot system 300. As shown in FIG.
Robotic system 300 includes robot 100 , server 200 and a plurality of external sensors 114 . A plurality of external sensors 114 (external sensors 114a, 114b, . . . , 114n) are installed in advance in the house. The external sensor 114 may be fixed to the wall surface of the house or placed on the floor. The position coordinates of the external sensor 114 are registered in the server 200 . The positional coordinates are defined as x, y coordinates in a house assumed as the action range of the robot 100 .

The server 200 is installed inside the house. The server 200 and the robot 100 in this embodiment are normally in one-to-one correspondence. The server 200 determines the basic actions of the robot 100 based on the information obtained from the sensors built into the robot 100 and the plurality of external sensors 114 .
The external sensor 114 is for reinforcing the sensory organs of the robot 100 , and the server 200 is for reinforcing the brain of the robot 100 .

The external sensor 114 periodically transmits a radio signal (hereinafter referred to as a "robot search signal") including an ID of the external sensor 114 (hereinafter referred to as a "beacon ID"). Upon receiving the robot search signal, the robot 100 returns a radio signal including the beacon ID (hereinafter referred to as "robot response signal"). The server 200 measures the time from when the external sensor 114 transmits the robot search signal to when it receives the robot reply signal, and measures the distance from the external sensor 114 to the robot 100 . By measuring the respective distances between the plurality of external sensors 114 and the robot 100, the position coordinates of the robot 100 are specified.
Of course, a system in which the robot 100 periodically transmits its own position coordinates to the server 200 may be used.

FIG. 4 is a conceptual diagram of the emotion map 116. As shown in FIG.
Emotion map 116 is a data table stored in server 200 . The robot 100 selects actions according to the emotion map 116 . The emotion map 116 shown in FIG. 4 indicates the magnitude of the likes and dislikes for the location of the robot 100 . The x-axis and y-axis of the emotion map 116 indicate two-dimensional spatial coordinates. The z-axis indicates the magnitude of likes and dislikes. A positive z-value indicates a high degree of liking for the location, and a negative z-value indicates dislike of the location.

In the emotion map 116 of FIG. 4, a coordinate P1 is a point with a high degree of favorability (hereinafter referred to as a “favorable point”) in the indoor space managed by the server 200 as the action range of the robot 100 . The favored point may be a "safe place" such as behind a sofa or under a table, or a place where people tend to gather, such as a living room, or a lively place. It may also be a place that was gently stroked or touched in the past.
The definition of what kind of place the robot 100 likes is arbitrary, but generally, it is desirable to set places preferred by small children and small animals such as dogs and cats as favorable points.

Coordinate P2 is a point of high ill feeling (hereinafter referred to as "disgust point"). Disgust points include places with loud noises such as near a TV, places where it is easy to get wet such as baths and washrooms, closed or dark places, and places associated with unpleasant memories of being treated roughly by the user. There may be.
The definition of what kind of place the robot 100 dislikes is arbitrary, but in general, it is desirable to set places where small children and small animals such as dogs and cats are afraid as disgust points.

A coordinate Q indicates the current position of the robot 100 . The server 200 identifies the position coordinates of the robot 100 based on the robot search signal and the robot reply signal corresponding to the robot search signal periodically transmitted by the plurality of external sensors 114 . For example, when the external sensor 114 with the beacon ID=1 and the external sensor 114 with the beacon ID=2 each detect the robot 100, the distance to the robot 100 is obtained from the two external sensors 114, and the position coordinates of the robot 100 are obtained therefrom. .

When the emotion map 116 shown in FIG. 4 is given, the robot 100 moves in the direction of being attracted to the favorable point (coordinates P1) and in the direction away from the disgusting point (coordinates P2).

The emotion map 116 changes dynamically. When the robot 100 reaches the coordinate P1, the z value (positivity) at the coordinate P1 decreases with time. As a result, the robot 100 can emulate the biological behavior of reaching the favored point (coordinate P1), being “satisfied with emotions”, and eventually becoming “bored” at that place. Similarly, the bad feeling at coordinate P2 is mitigated over time. With the passage of time, new points of favor and points of dislike are created, and the robot 100 makes new action selections accordingly. The robot 100 is "interested" in new favored points, and continuously selects actions.

The emotion map 116 expresses emotional ups and downs as the internal state of the robot 100 . The robot 100 aims at the favored point, avoids the disliked point, stays at the favored point for a while, and then takes the next action again. With such control, action selection of the robot 100 can be made human and biological.

Note that the map that affects the action of the robot 100 (hereinafter collectively referred to as the "action map") is not limited to the emotion map 116 of the type shown in FIG. For example, curiosity, avoidance of fear, need for reassurance, need for physical comforts such as silence/darkness, coolness/warmth, etc., can be defined in various behavioral maps. Then, the destination point of the robot 100 may be determined by weighting and averaging the z values of each of the plurality of action maps.

The robot 100 has parameters indicating the magnitude of various emotions and sensations in addition to the action map. For example, when the value of the emotional parameter of loneliness is increasing, the weighting factor of the action map for evaluating places where people feel safe is set large, and the value of this emotional parameter is lowered by reaching the target point. Similarly, when the value of the parameter indicating the sense of boredom is high, the weighting factor of the activity map that evaluates places that satisfy curiosity can be set high.

FIG. 5 is a hardware configuration diagram of the robot 100. As shown in FIG.
Robot 100 includes internal sensors 128 , communicator 126 , storage device 124 , processor 122 , drive mechanism 120 and battery 118 . Drive mechanism 120 includes wheel drive mechanism 370 described above. Processor 122 and storage device 124 are included in control circuitry 342 . Each unit is connected to each other by a power line 130 and a signal line 132 . A battery 118 supplies power to each unit via a power line 130 . Each unit transmits and receives control signals via signal line 132 . The battery 118 is a lithium ion secondary battery and is the power source of the robot 100 .

The internal sensor 128 is a collection of various sensors built into the robot 100 . Specifically, they are cameras (omnidirectional cameras), microphone arrays, ranging sensors (infrared sensors), thermosensors, touch sensors, acceleration sensors, odor sensors, and the like. A touch sensor is installed between the outer skin 314 and the body frame 310 to detect a user's touch. An odor sensor is a known sensor that applies the principle that electrical resistance changes due to the adsorption of odor-causing molecules.

The communication device 126 is a communication module that performs wireless communication with various external devices such as the server 200, the external sensor 114, and a user's portable device. The storage device 124 is composed of non-volatile memory and volatile memory, and stores computer programs and various setting information. The processor 122 is means for executing computer programs. The drive mechanism 120 is an actuator that controls internal mechanisms. It also has a display and speakers.

The processor 122 selects actions of the robot 100 while communicating with the server 200 and the external sensor 114 via the communication device 126 . A variety of external information obtained by internal sensors 128 also influences behavioral choices. Drive mechanism 120 primarily controls the wheels (front wheels 102) and the head (head frame 316). The driving mechanism 120 changes the moving direction and moving speed of the robot 100 by changing the rotating speed and rotating direction of each of the two front wheels 102 . The drive mechanism 120 can also move the wheels (the front wheels 102 and the rear wheels 103) up and down. When the wheels are raised, the wheels are completely housed in the body 104, and the robot 100 comes into contact with the floor surface F on the seating surface 108 and is in a seated state. Drive mechanism 120 also controls hand 106 via wire 135 .

FIG. 6 is a functional block diagram of the robot system 300. As shown in FIG.
As described above, robotic system 300 includes robot 100 , server 200 and a plurality of external sensors 114 . Each component of the robot 100 and the server 200 includes computing units such as a CPU (Central Processing Unit) and various coprocessors, storage devices such as memory and storage, hardware including wired or wireless communication lines connecting them, and storage. It is implemented by software that is stored in the device and that supplies processing instructions to the calculator. A computer program may consist of a device driver, an operating system, various application programs located in their higher layers, and a library that provides common functions to these programs. Each block described below represents a functional block rather than a hardware configuration.
A part of the functions of the robot 100 may be realized by the server 200 , and a part or all of the functions of the server 200 may be realized by the robot 100 .

(Server 200)
Server 200 includes communication unit 204 , data processing unit 202 and data storage unit 206 .
The communication unit 204 is in charge of communication processing with the external sensor 114 and the robot 100 . A data storage unit 206 stores various data. The data processing unit 202 executes various processes based on data acquired by the communication unit 204 and data stored in the data storage unit 206 . Data processing unit 202 also functions as an interface for communication unit 204 and data storage unit 206 .

Data store 206 includes motion store 232 , map store 216 and personal data store 218 .
The robot 100 has a plurality of action patterns (motions). Various motions are defined, such as shaking the hand 106, approaching the owner while meandering, and staring at the owner while tilting the neck.

The motion storage unit 232 stores a “motion file” that defines the content of motion control. Each motion is identified by a motion ID. The motion files are also downloaded to the motion store 160 of the robot 100 . Which motion to perform may be determined by the server 200 or may be determined by the robot 100 .

Most of the motions of the robot 100 are configured as compound motions including multiple unit motions. For example, when the robot 100 approaches the owner, it may be expressed as a combination of a unit motion of turning toward the owner, a unit motion of approaching while raising hands, a unit motion of approaching while shaking the body, and a unit motion of sitting while raising both hands. . By combining these four motions, a motion of "approaching the owner, raising a hand in the middle, finally shaking the body and sitting down" is realized. In the motion file, the rotation angles and angular velocities of the actuators provided in the robot 100 are defined in association with the time axis. Various motions are expressed by controlling each actuator over time according to a motion file (actuator control information).

The transition time when changing from the previous unit motion to the next unit motion is called "interval". The interval may be defined according to the time required to change the unit motion and the contents of the motion. The interval length is adjustable.
Hereinafter, settings related to action control of the robot 100, such as when and which motion to select, output adjustment of each actuator for realizing the motion, etc., are collectively referred to as "behavior characteristics". The behavioral characteristics of the robot 100 are defined by motion selection algorithms, motion selection probabilities, motion files, and the like.

The motion storage unit 232 stores motion files as well as motion selection tables that define motions to be executed when various events occur. In the motion selection table, one or more motions and their selection probabilities are associated with events.

The map storage unit 216 stores a plurality of action maps as well as maps showing the arrangement of obstacles such as chairs and tables. The personal data storage unit 218 stores user information, especially owner information. Specifically, it stores master information indicating familiarity with the user and the user's physical features/behavioral features. Other attribute information such as age and gender may be stored.

The robot 100 has an internal parameter of intimacy for each user. When the robot 100 recognizes an action showing goodwill toward the user, such as picking the user up or calling out to the user, the degree of intimacy with the user increases. The degree of intimacy is low for users who do not interact with the robot 100, users who behave violently, and users whom they rarely meet.

Data processing unit 202 includes location management unit 208 , map management unit 210 , recognition unit 212 , operation control unit 222 , familiarity management unit 220 and state management unit 244 .
The position management unit 208 identifies the position coordinates of the robot 100 by the method described using FIG. The location manager 208 may also track the user's location coordinates in real time.

The state management unit 244 manages various internal parameters such as the charging rate, internal temperature, and various physical states such as the processing load of the processor 122 . State manager 244 includes emotion manager 234 .
The emotion management unit 234 manages various emotion parameters indicating the emotions of the robot 100 (loneliness, curiosity, desire for approval, etc.). These emotional parameters are constantly fluctuating. The importance of a plurality of action maps changes according to the emotion parameter, the movement target point of the robot 100 changes depending on the action map, and the emotion parameter changes according to the movement of the robot 100 and the passage of time.

For example, when the emotion parameter indicating loneliness is high, the emotion management unit 234 sets a large weighting factor for the action map that evaluates the safe place. When the robot 100 reaches a point where loneliness can be resolved on this action map, the emotion management unit 234 lowers the emotion parameter indicating loneliness. In addition, various emotion parameters change depending on the later-described response behavior. For example, when the owner hugs the pet, the emotional parameter indicating loneliness decreases, and when the owner is not seen for a long time, the emotional parameter indicating loneliness gradually increases.

The map management unit 210 changes the parameter of each coordinate by the method described in relation to FIG. 4 for a plurality of action maps. The map management unit 210 may select one of the plurality of action maps, or may weight-average the z values of the plurality of action maps. For example, on action map A, the z values at coordinates R1 and R2 are 4 and 3, and on action map B, the z values at coordinates R1 and R2 are -1 and 3. In the simple average, the total z value of coordinate R1 is 4−1=3 and the total z value of coordinate R2 is 3+3=6, so robot 100 is directed toward coordinate R2 instead of coordinate R1.
When action map A is given five times more weight than action map B, the total z value of coordinate R1 is 4×5−1=19, and the total z value of coordinate R2 is 3×5+3=18. in the direction of

The recognition unit 212 recognizes the external environment. Recognition of the external environment includes various recognitions such as recognition of weather and seasons based on temperature and humidity, and recognition of shadows (safe zones) based on light intensity and temperature. The recognition unit 156 of the robot 100 acquires various environmental information from the internal sensor 128 , performs primary processing on the information, and transfers it to the recognition unit 212 of the server 200 .

Specifically, the recognition unit 156 of the robot 100 extracts an image area corresponding to a moving object, particularly a person or an animal, from the image, and extracts the physical characteristics and behavioral characteristics of the moving object from the extracted image area. Extract a "feature vector" as a set of quantities. A feature vector component (feature amount) is a numerical value obtained by quantifying various physical/behavioral features. For example, the width of the human eye is digitized in the range of 0 to 1 to form one feature vector component. A technique for extracting feature vectors from a captured image of a person is an application of known face recognition technology. Robot 100 transmits the feature vector to server 200 .

The recognition unit 212 of the server 200 further includes a person recognition unit 214 and a reception recognition unit 228 .
The person recognition unit 214 compares the feature vector extracted from the image captured by the built-in camera of the robot 100 with the feature vector of users (clusters) registered in advance in the personal data storage unit 218, thereby identifying the imaged user. to which person corresponds (user identification processing). Person recognition unit 214 includes facial expression recognition unit 230 . The facial expression recognition unit 230 estimates the user's emotion by performing image recognition of the user's facial expression.
The person recognition unit 214 also performs user identification processing on moving objects other than people, such as pets such as cats and dogs.

The response recognition unit 228 recognizes various response actions performed by the robot 100 and classifies them into pleasant and unpleasant actions. The response recognition unit 228 also recognizes the owner's response to the behavior of the robot 100 and classifies it into positive and negative responses.
Pleasant/unpleasant behavior is determined by whether the user's response behavior is biologically pleasant or unpleasant. For example, being hugged is a pleasant action for the robot 100, and being kicked is an unpleasant action for the robot 100. A positive/negative reaction is determined depending on whether the user's behavior indicates a user's pleasant feeling or an unpleasant feeling. For example, being hugged is a positive response that indicates a user's pleasant feeling, and being kicked is a negative response that indicates a user's unpleasant feeling.

The motion control unit 222 of the server 200 cooperates with the motion control unit 150 of the robot 100 to determine the motion of the robot 100 . The operation control unit 222 of the server 200 creates a movement target point for the robot 100 and a movement route therefor based on the action map selection by the map management unit 210 . The operation control unit 222 may create a plurality of moving routes and select one of the moving routes.

The motion control unit 222 selects a motion for the robot 100 from multiple motions in the motion storage unit 232 . Each motion is associated with a selection probability for each situation. For example, a selection method is defined such that when the owner makes a pleasurable act, motion A is executed with a probability of 20%, and when the temperature reaches 30°C or higher, motion B is executed with a probability of 5%. .
A movement target point and a movement route are determined on the action map, and motions are selected according to various events described later.

The familiarity management unit 220 manages the familiarity of each user. As described above, the degree of intimacy is registered as part of personal data in personal data storage unit 218 . When a pleasant act is detected, the degree of intimacy management unit 220 increases the degree of intimacy with the owner. The degree of intimacy is lowered when an unpleasant act is detected. In addition, the intimacy level of the owner who has not seen it for a long time gradually decreases.

(robot 100)
Robot 100 includes communication portion 142 , data processing portion 136 , data storage portion 148 , internal sensors 128 and drive mechanism 120 .
The communication unit 142 corresponds to the communication device 126 (see FIG. 5) and takes charge of communication processing with the external sensor 114, the server 200 and other robots 100. FIG. The data storage unit 148 stores various data. The data storage unit 148 corresponds to the storage device 124 (see FIG. 5). The data processing unit 136 executes various processes based on data acquired by the communication unit 142 and data stored in the data storage unit 148 . The data processing unit 136 corresponds to the processor 122 and computer programs executed by the processor 122 . Data processing unit 136 also functions as an interface for communication unit 142 , internal sensor 128 , drive mechanism 120 and data storage unit 148 .

The data store 148 includes a motion store 160 and an image store 172 that define various motions of the robot 100 .
Various motion files are downloaded from the motion storage unit 232 of the server 200 to the motion storage unit 160 of the robot 100 . A motion is identified by a motion ID. Sitting with the front wheels 102 retracted, lifting the hand 106, rotating the robot 100 by rotating the two front wheels 102 in reverse, or rotating only one of the front wheels 102, with the front wheels 102 retracted. In order to express various motions such as trembling by rotating the front wheel 102 with the , stopping and looking back when moving away from the user, operation timing, operation time, operation direction, etc. of various actuators (driving mechanism 120) are adjusted. It is defined chronologically in the motion file.

Various data may also be downloaded to the data storage unit 148 from the map storage unit 216 and the personal data storage unit 218 . The image storage unit 172 temporarily stores a celestial image (described later) acquired by the omnidirectional camera 134 and various images in the process of the processing.

Internal sensors 128 include omnidirectional camera 134 . The omnidirectional camera 134 in this embodiment is attached to the horn 112 .

Data processing unit 136 includes recognition unit 156 , operation control unit 150 , image acquisition unit 152 , image conversion unit 154 and distance measurement unit 158 .
The motion control unit 150 of the robot 100 determines the motion of the robot 100 in cooperation with the motion control unit 222 of the server 200 . Some motions may be determined by the server 200 and other motions may be determined by the robot 100 . Also, although the robot 100 determines the motion, the server 200 may determine the motion when the processing load of the robot 100 is high. A base motion may be determined at the server 200 and an additional motion may be determined at the robot 100 . How the server 200 and the robot 100 share the motion determination process may be designed according to the specifications of the robot system 300 .

The motion control unit 150 of the robot 100 determines the moving direction of the robot 100 together with the motion control unit 222 of the server 200 . The movement based on the action map may be determined by the server 200 , and the immediate movement such as avoiding obstacles may be determined by the motion control section 150 of the robot 100 . The drive mechanism 120 drives the front wheels 102 in accordance with instructions from the motion control unit 150 to direct the robot 100 to the movement target point.

The motion control unit 150 of the robot 100 instructs the driving mechanism 120 to execute the selected motion. The drive mechanism 120 controls each actuator according to the motion file.

The motion control unit 150 can also execute a motion of raising both hands 106 as a gesture of pleading for a hug when a user with a high degree of intimacy is nearby, and can move the left and right front wheels 102 when the user gets tired of hugging. By alternately repeating reverse rotation and stopping while stored, it is possible to express a motion that does not want to be hugged. The drive mechanism 120 drives the front wheels 102, hands 106, and neck (head frame 316) according to instructions from the motion control unit 150, thereby causing the robot 100 to express various motions.

The image acquisition unit 152 acquires a celestial image from the omnidirectional camera 134 . The image conversion unit 154 converts the celestial image into various projection images (details will be described later). When the robot 100 is equipped with a plurality of omnidirectional cameras 134, the rangefinder 158 measures the distance to an external object based on the principle of triangulation (described later in connection with the third embodiment).

A recognition unit 156 of the robot 100 interprets external information obtained from the internal sensor 128 . The recognition unit 156 is capable of visual recognition (visual unit), smell recognition (olfactory unit), sound recognition (auditory unit), and tactile recognition (tactile unit).

A recognition unit 156 detects a moving object such as a person or a pet based on the celestial image obtained by the omnidirectional camera 134 . Recognition unit 156 includes feature extraction unit 146 . The feature extraction unit 146 extracts feature vectors from the captured image of the moving object. As described above, a feature vector is a set of parameters (feature quantities) representing physical features and behavioral features of a moving object. When a moving object is detected, physical characteristics and behavioral characteristics are also extracted from an odor sensor, a built-in sound collecting microphone, a temperature sensor, and the like. These features are also quantified into feature vector components.

The robot system 300 clusters frequently appearing users as "owners" based on physical and behavioral features obtained from a large amount of image information and other sensing information.
For example, if moving objects with beards (users) tend to be active in the early morning (early risers) and rarely wear red clothes, then a cluster (users) who wake up early in the morning, grow beards and rarely wear red clothes ( user). On the other hand, a moving object wearing glasses often wears a skirt, but if this moving object does not have a beard, a cluster that wears glasses and wears a skirt but never has a beard is found. (user), a second profile is created.
Although the above is a simple example, the method described above creates a first profile corresponding to the father and a second profile corresponding to the mother, and there are at least two users (owners) in this house. The robot 100 recognizes this.

However, robot 100 need not recognize that the first profile is "father". Ultimately, it is sufficient to recognize the image of a person as "a cluster that has a beard, often wakes up early, and rarely wears red clothes." For each profile, a feature vector is defined that characterizes the profile.

Assume that the robot 100 newly recognizes a moving object (user) in a state where such cluster analysis is completed.
At this time, the person recognition unit 214 of the server 200 executes user identification processing based on the feature vector of the new moving object, and determines which profile (cluster) the moving object corresponds to. For example, when a moving object with a beard is detected, there is a high probability that this moving object is the father. If this moving object moves early in the morning, it is more certain that it corresponds to the father. On the other hand, when a moving object wearing glasses is detected, this moving object may be the mother. If this moving object has a beard, it is neither a mother nor a father, so it is determined to be a new person who has not undergone cluster analysis.

Formation of clusters (profiles) by feature extraction (cluster analysis) and fitting to clusters accompanying feature extraction may be executed in parallel.

Among a series of recognition processes including detection, analysis, and judgment, the recognition unit 156 of the robot 100 selects and extracts information necessary for recognition, and interpretation processing such as judgment is executed by the recognition unit 212 of the server 200. . The recognition process may be performed only by the recognition unit 212 of the server 200, or may be performed only by the recognition unit 156 of the robot 100. As described above, the recognition process may be performed while both of them share roles. good too.

When a strong impact is applied to the robot 100, the recognizing unit 156 recognizes this with a built-in acceleration sensor, and the response recognizing unit 228 of the server 200 recognizes that a nearby user has acted violently. When the user grabs the horn 112 and lifts the robot 100, it may be recognized as a violent act. When the user facing the robot 100 speaks in a specific sound volume range and a specific frequency band, the response recognition unit 228 of the server 200 may recognize that the user has made a "calling action" to himself/herself. Also, when a temperature of about body temperature is detected, it is recognized that the user has made a "contact action", and when an upward acceleration is detected while contact is being recognized, it is recognized that a "hold" has been made. Physical contact when the user lifts the body 104 may be sensed, and the holding may be recognized by reducing the load applied to the front wheels 102 .
In summary, the robot 100 acquires the behavior of the user as physical information by the internal sensor 128, the response recognition unit 228 of the server 200 determines whether it is pleasant or unpleasant, and the recognition unit 212 of the server 200 performs user identification processing based on the feature vector. to run.

The response recognition unit 228 of the server 200 recognizes various responses of the user to the robot 100 . Pleasant or unpleasant, affirmative or negative are associated with a part of typical responses among various responses. In general, most of the reception behaviors that are pleasant behaviors are positive reactions, and most of the reception behaviors that are unpleasant behaviors are negative reactions. Pleasant/unpleasant actions are related to intimacy, and affirmative/negative reactions influence the action selection of the robot 100 .

The familiarity management unit 220 of the server 200 changes the familiarity with the user according to the behavior recognized by the recognition unit 156 . In principle, the degree of intimacy with a user who has performed a pleasant act increases, and the degree of intimacy with a user who has performed an unpleasant act decreases.

The recognizing unit 212 of the server 200 determines pleasantness/unpleasantness according to the reception, and the map management unit 210 changes the z value of the point where the pleasantness/unpleasant action is performed in the action map expressing “attachment to the place”. may For example, when a pleasant act is performed in the living room, the map management unit 210 may set a favorable point in the living room with a high probability. In this case, the robot 100 likes the living room, and the positive feedback effect that the robot 100 likes the living room more and more by receiving pleasant behavior in the living room is realized.

The level of intimacy with the user changes depending on what kind of action is performed by the moving object (user).

The robot 100 sets a high degree of intimacy to people who meet frequently, people who often touch them, and people who often talk to them. On the other hand, intimacy is low for people who are rarely seen, rarely touched, violent, or scolded loudly. The robot 100 changes the degree of intimacy for each user based on various external world information detected by sensors (visual, tactile, and auditory).

The actual robot 100 autonomously selects complex actions according to the action map. The robot 100 behaves while being influenced by a plurality of action maps based on various parameters such as loneliness, boredom, and curiosity. If the influence of the action map is excluded, or when the robot 100 is in an internal state where the influence of the action map is small, in principle, the robot 100 will try to approach a person with a high degree of intimacy and move away from a person with a low degree of intimacy. and

The behavior of the robot 100 is categorized as follows according to familiarity.
(1) A user with a very high intimacy The robot 100 approaches a user (hereinafter referred to as "proximity behavior") and performs an affection gesture defined in advance as a gesture of showing goodwill to a person. strongly express.
(2) A user robot 100 with a relatively high degree of intimacy performs only close behavior.
(3) The user robot 100 with relatively low degree of intimacy does not perform any particular action.
(4) A user robot 100 with a particularly low degree of intimacy performs a withdrawal action.

According to the control method described above, when the robot 100 finds a user with a high degree of intimacy, it approaches that user, and conversely, when it finds a user with a low degree of intimacy, it moves away from that user. With such a control method, so-called "shyness" can be expressed in behavior. Also, when a visitor (user A with a low degree of intimacy) appears, the robot 100 may leave the visitor and head toward his family (user B with a high degree of intimacy). In this case, user B can perceive that the robot 100 is shy and feels uneasy, and that it relies on him. Such a behavioral expression arouses user B's joy of being selected and relied upon, and the feeling of attachment associated therewith.

On the other hand, when the user A, who is a visitor, frequently visits, speaks to, and touches the robot 100, the intimacy of the robot 100 with the user A gradually increases, and the robot 100 ceases to behave shyly (withdrawal behavior) toward the user A. . User A can feel attachment to the robot 100 by feeling that the robot 100 has become familiar to him/herself.

Note that the above action selection is not always executed. For example, when the internal parameter indicating the curiosity of the robot 100 is high, there is a possibility that the robot 100 will not select actions that are influenced by intimacy because the action map for searching for a place that satisfies the curiosity is emphasized. . Also, when the external sensor 114 installed at the entrance detects that the user has returned home, the behavior of welcoming the user may be performed with the highest priority.

[General projection processing from celestial images]
First, a general method of generating a projected image by projecting a celestial image, which is an image captured by the omnidirectional camera 134, onto a cylindrical surface based on the equirectangular projection will be described below. Next, the projection method newly employed in the robot system 300 will be described after pointing out the problems of the equirectangular projection.
Hereinafter, the term "projection image" means an image that is generated by transforming a celestial image and is expected to be a direct object of image recognition.

FIG. 7 is a schematic diagram for explaining a method of generating a celestial image.
A celestial sphere 140 is a conceptual imaging plane when the omnidirectional camera 134 is installed at the center point O. FIG. The omnidirectional camera 134 projects the coordinate points on the celestial sphere 140 (imaging plane) onto the imaging plane 138 . Imaging plane 138 is the horizontal plane containing omnidirectional camera 134 . The celestial sphere 140 can be divided into an upper celestial sphere 140a on the upper half side and a lower hemisphere 140b on the lower half side. In FIG. 7, the coordinate point P1 of the upper celestial sphere 140a is projected onto the coordinate point Q1 of the imaging plane 138. In FIG. Similarly, the coordinate points P2, P3, PZ (zenith) of the upper celestial sphere 140a are projected onto the coordinate points Q2, Q3, O (center point) of the imaging plane 138 (surface).

Lower hemisphere 140b is also projected onto image plane 138 in a similar fashion. The coordinate points P4, P5, P6, PN (nadir) of the lower hemisphere 140b are projected onto the coordinate points Q1, Q2, Q3, O (center point) of the imaging plane 138 (back surface).

All coordinate points of the upper celestial sphere 140 a are projected onto the front side of the imaging plane 138 , and all coordinate points of the lower hemisphere 140 b are projected onto the back side of the imaging plane 138 . A two-dimensional “celestial image” is generated by projecting the coordinate points (pixels) on the three-dimensional celestial sphere 140 onto the two-dimensional imaging plane 138 . A method of generating such a celestial image is well known for a fisheye lens or the like.

FIG. 8 is a schematic diagram of the celestial image 170. As shown in FIG.
The celestial image 170 includes a perfectly circular upper celestial image 170a and a lower celestial image 170b. The upper celestial sphere image 170 a is an image obtained by projecting the upper celestial sphere 140 a (upper imaging plane) onto the surface of the imaging plane 138 . A lower celestial sphere image 170 b is an image obtained by projecting the lower hemisphere 140 b (lower imaging surface) onto the back surface of the imaging surface 138 . The center point O of the upper celestial sphere image 170a corresponds to the zenith PZ of the upper celestial sphere 140a, and the center point O of the lower celestial sphere image 170b corresponds to the nadir PN of the lower hemisphere 140b. A celestial sphere centerline 174 is a boundary line between the upper celestial sphere 140a and the lower hemisphere 140b.

Note that the omnidirectional camera 134 of the robot 100 may capture only the upper celestial sphere 140a. In that case, the celestial image 170 is composed only of the upper celestial image 170a. The celestial sphere image 170 may be a captured image that covers a part of the celestial sphere 140 , at least 35% or more of the surface area of the celestial sphere 140 . The celestial image 170 is desirably a captured image covering the area above the robot 100 , that is, the range including the zenith PZ of the celestial surface 140 .

FIG. 9 is a schematic diagram for explaining the relationship between the cylindrical surface 162 and the celestial surface 140 in the equirectangular projection.
First, a cylindrical surface 162 that circumscribes the celestial sphere 140 is assumed. Cylindrical surface 162 surrounds celestial surface 140 and does not have upper and lower surfaces. In the equirectangular projection, the coordinate points of the celestial sphere 140 are projected onto the cylindrical surface 162 . In other words, the coordinate points on cylindrical surface 162 corresponding to the coordinate points on celestial image 170 are identified.

Cylindrical centerline 180 is a line passing through the center of cylindrical surface 162 . In an equirectangular projection, the celestial centerline 174 and the cylinder centerline 180 coincide. Therefore, in the vicinity of the cylinder center line 180 (the celestial sphere center line 174), the celestial sphere surface 140 and the cylindrical surface 162 almost coincide. However, the closer the zenith PZ or the nadir PN (hereinafter collectively referred to as "polar point"), the greater the divergence between the celestial sphere 140 and the cylindrical surface 162. FIG. The equirectangular projection is a projection method that translates the latitude and longitude of a sphere into the X and Y coordinates of a plane (side surface of a cylinder). Therefore, when the image of the celestial sphere 140 is projected onto the cylindrical surface 162, there is a drawback that the projected image is greatly stretched in the lateral direction near the poles.

FIG. 10 is a schematic diagram for explaining an image projection method based on the equirectangular projection from the celestial image 170 to the single cylindrical image 164. FIG.
A single cylindrical image 164 is a projection image from the celestial sphere 140 onto the cylindrical surface 162 . A single cylindrical image 164 is an unfolded image of the cylindrical surface 162 in FIG. Referring to FIG. 10, a method of projecting coordinate points A1 and A2 on the celestial sphere 140 onto a single cylindrical image 164 will be described.

Point A1 is closer to the celestial centerline 174 than point A2. Point A1 on celestial surface 140 corresponds to point B1 on celestial image 170 and to point C1 on single cylinder image 164 . Point A2 corresponds to point B2 on the celestial image 170 and to point C2 on the single cylindrical image 164. FIG. Actually, when obtaining the corresponding point C1 on the single cylindrical image 164 of the point B1 of the celestial sphere image 170, first, the coordinates of the point A1 on the celestial sphere 140 corresponding to the point B1 are obtained, and then from the celestial sphere 140 to the cylinder By transforming the coordinates to the surface 162, the coordinates of the point C1 of the single cylindrical image 164 are calculated.

In the equirectangular projection, the single cylindrical image 164 is greatly distorted in the vicinity of the poles (shaded areas in the figure). Therefore, the point B2 of the celestial image 170 is stretched horizontally in the single cylindrical image 164 . In other words, point C2 corresponding to point B2 is stretched laterally.

Various products are provided as image recognition programs for detecting external objects from captured images. A general image recognition program assumes that there is no distortion in the captured image (projected image) to be analyzed. Therefore, it is difficult to perform image recognition on the single cylindrical image 164 by equirectangular projection, let alone the celestial image 170 . A first countermeasure for this problem is to develop a new program capable of image recognition taking into consideration the distortion of the celestial image 170 and the single cylindrical image 164 . A second countermeasure is to develop an image correction program that generates a projection image with no or little distortion from the celestial image 170, regardless of the equirectangular projection. The second measure has the advantage of being able to use various existing image recognition programs. In this embodiment, the second countermeasure is adopted, and two types of methods for correcting the distortion of the celestial sphere image 170 are shown.

As a method of correcting the distortion of the celestial image 170, a method of projecting onto the rhombicoctahedron (hereinafter referred to as "polyhedral projection method") in the first embodiment will be described. Next, a method of projecting onto multiple cylinders (hereinafter referred to as "multiple cylinder projection method") will be described as a second embodiment.

(First Embodiment: Polyhedral Projection Method)
FIG. 11 is a perspective view of an orthorhombic octahedron 166 . FIG. 12 is a developed view of the rhombicoctahedron 166. As shown in FIG.
In the first embodiment, an oblique cubic octahedron 166 surrounding the omnidirectional camera 134 is assumed. Rhombic octahedron 166 is a polyhedron concentric with celestial sphere 140 . The rhombocubic octahedron 166 has 18 square faces (hereinafter also referred to as "square faces") and 8 equilateral triangular faces (hereinafter also referred to as "regular triangular faces"), for a total of 26 faces. The breakdown of the 18 square surfaces is the top surface T and bottom surface B, the eight central side surfaces F, FR, R, BR, B, BL, L, FL, and the four surfaces connecting the top surface T and the central side surface. Four lower surfaces BF, BR, BB, and BL connecting the upper surface TF, TR, TB, and TL, the lower surface B, and the center surface.

The top surface T is a horizontal plane perpendicular to the axis connecting the center O and the zenith PZ. Similarly, the lower surface B is a horizontal plane perpendicular to the axis connecting the center O and the nadir PN. In the first embodiment, enlarged parallel planes 168 are assumed to be enlarged in the parallel direction with respect to all square planes. Enlarged parallel planes 168 intersect each other. An enlarged parallel plane 168 (TF) shown in FIG. 11 is a plane obtained by parallel-enlarging the upper side surface TF in the vertical and horizontal directions. An enlarged parallel plane 168 (F) is a plane obtained by parallel-enlarging the central side surface F in the vertical and horizontal directions. Enlarged parallel surface 168(TF) and enlarged parallel surface 168(F) intersect each other. Similarly, an enlarged parallel surface 168(B) shown in FIG. 12 is a surface obtained by parallelly enlarging the central side surface B in the vertical and horizontal directions.
In the first embodiment, coordinate points (pixels) on the celestial image 170 are projected onto a total of 18 enlarged parallel planes 168 . In other words, the projection image of the celestial image 170 is an aggregation of the 18 enlarged parallel planes 168 .

FIG. 13 is a side sectional view of rhombicoctahedron 166 and celestial sphere 140. FIG.
The point S (TF) is the point of intersection when a perpendicular line is drawn from the coordinate point A of the celestial sphere 140 to the enlarged parallel plane 168 (TF). The point S(F) is the point of intersection when a perpendicular line is drawn from the coordinate point A of the celestial sphere 140 to the enlarged parallel plane 168(F). Assume that point Q corresponds to point A on the imaging plane 138 (celestial image 170).

Image converter 154 identifies point S(TF) and point S(F) corresponding to point A (point Q) on enlarged parallel plane 168(TF) and enlarged parallel plane 168(F), respectively. More specifically, the transformation function f _TF for calculating the coordinates of point S(TF) on enlarged parallel plane 168(TF), the coordinate of point S(F) on enlarged parallel plane 168(F) is _prepared in advance. The image transformation unit 154 calculates the coordinates of the projection point on each enlarged parallel plane 168 using a plurality of transformation functions corresponding to the multiple enlarged parallel planes 168 .

The image conversion unit 154 copies the pixel (imaging information) of the coordinate point A in the celestial image 170 to the point S(TF) on the enlarged parallel plane 168(TF) and the point S(F) on the enlarged parallel plane 168(F). By doing so, the celestial image 170 is projected onto the enlarged parallel plane 168 . That is, the pixel at the coordinate point Q of the celestial sphere image 170 becomes the pixel at the coordinate point S(TF) on the enlarged parallel plane 168(TF) and the pixel at the coordinate point S(F) on the enlarged parallel plane 168(F). Become. In the case of the polyhedral projection method, one pixel (coordinate point) of the celestial image 170 may be multiple-projected onto a plurality of enlarged parallel planes 168 .

In FIG. 13, the point S(F) is closer to the celestial sphere 140 than the point S(TF). Therefore, S(F) has less image distortion than point S(TF). Also, since point S(TF) is close to the edge of enlarged parallel plane 168(TF) and point S(F) is close to the center of enlarged parallel plane 168(F), the image near point A is the enlarged parallel plane 168 Enlarged parallel plane 168 (F) is likely to be projected better than (TF). Point A is projected onto both enlarged parallel plane 168 (TF) and point A is projected onto enlarged parallel plane 168 (F), but may also be projected onto enlarged parallel plane 168 (FR) (not shown). do not have. In this case, the three enlarged parallel planes 168 allow the object to be recognized from multiple angles. The enlarged parallel plane 168 may also be set for the equilateral triangular plane shown in FIG. In this case, a single object can be projected onto more parallel planes 168 simultaneously.

In the polyhedral projection method, a plurality of enlarging parallel planes 168 envisioned along the celestial sphere 140 surround the center point O (omnidirectional camera 134 ). The zenith PZ is projected onto the enlarged parallel plane 168(T). Unlike the cylindrical surface 162 in FIG. 9, the enlarged parallel surface 168(T) coincides with the celestial surface 140 at the zenith PZ, so the distortion of the projected image (enlarged parallel surface 168(T)) near the zenith PZ is the equirectangular projection. is considerably smaller than that of The same applies to the nadir PN.

FIG. 14 is a schematic diagram for explaining partial cutting of the enlarged parallel plane 168. As shown in FIG.
The image conversion unit 154 may cut out a portion of the enlarged parallel plane 168 instead of the entirety. In FIG. 14, the range of enlarged parallel plane 168(F) is 0≤x≤x(F), 0≤y≤y(F). The image conversion unit 154 cuts out x1≦x≦x2 (0<x1, x2<x(F)) and y1≦y≦y2 (0<y1, y2<y(F)). The image conversion unit 154 may exclude a portion of the enlarged parallel plane 168 that is known to have a large distortion from being converted, and subject only a part of the enlarged parallel plane 168 to image processing. Alternatively, the image conversion unit 154 may exclude portions of the enlarged parallel plane 168 in which important information is not shown, such as monotonous portions in which only the same color is shown over a predetermined range or more, from the object of image processing.

FIG. 15 is a schematic diagram for explaining the polyhedral projection method.
A polyhedral image 190 (projected image) is a collection of 18 enlarged parallel planes 168 . That is, the polyhedral image 190 is a projected image from the celestial sphere 140 onto the enlarged parallel plane 168 of 18 planes. In FIG. 15, it is assumed that the coordinate points A1 and A2 of the celestial sphere 140 are projected onto the polyhedral image 190 respectively.

Point A1 is closer to the celestial centerline 174 than point A2. A point A1 on the celestial sphere 140 corresponds to a point B1 on the celestial image 170, and points S1 and S12 on the polyhedral image 190. FIG. The point A2 corresponds to the point B2 in the celestial image 170, and corresponds to the points S21 and S22 in the polyhedron image 190. FIG. When finding the corresponding points S11 and S12 on the polyhedral image 190 of the point B1 of the celestial sphere image 170, first find the coordinates of the point A1 on the celestial sphere 140 corresponding to the point B1, and then from the celestial sphere 140 to the enlarged parallel plane 168. coordinates of points S11 and S12 on the enlarged parallel plane 168 (polyhedral image 190) are calculated by the coordinate conversion of . The point A2 is also the same. In the case of polyhedral projection, portions of a single external object may be overprojected onto multiple enlarged parallel planes 168 .

FIG. 16 is a flow chart showing the image processing process in the first embodiment.
The image acquisition unit 152 periodically acquires the celestial image 170 . The image processing shown in FIG. 16 is repeatedly executed each time the celestial image 170 is obtained.

First, the image acquisition unit 152 acquires the celestial image 170 from the omnidirectional camera 134 (S10). The image conversion unit 154 converts the celestial sphere image 170 into a polyhedral image 190, that is, an enlarged parallel plane 168 with 18 planes (S12). As described above, a transformation function is defined for each enlarged parallel plane 168 , and the image transformation unit 154 projects points on the celestial sphere image 170 onto one or more enlarged parallel planes 168 using 18 kinds of transformation functions. Before executing image recognition processing for each of the enlarged parallel planes 168 included in the polyhedral image 190, the recognition unit 156 determines the processing order of image recognition (S14).

For example, when the owner X is detected on the enlarged parallel plane 168(R) at the processing time t1 and the image processing is executed again at the next processing time t2, the recognition unit 156 performs the image recognition of the enlarged parallel plane 168(R) to the maximum. set priority. This is because there is a high possibility that the owner X will be detected on the enlarged parallel plane 168(R) even at the processing time t2. Since the robot 100 changes its behavioral characteristics based on the position, movement, facial expression, etc. of the owner X, the recognition unit 156 gives top priority to the owner X's visual recognition. After the enlarged parallel plane 168(R), the recognizing unit 156 assigns the processing order to the enlarged parallel plane 168(TR) adjacent to the enlarged parallel plane 168(R), and so on.

After determining the processing order, the recognition unit 156 executes image recognition in cooperation with the server 200 (S16). Specifically, the recognition unit 156 extracts a moving object such as a user from the enlarged parallel plane 168 ranked first in the processing order, and if the moving object is reflected on this enlarged parallel plane 168, the feature extraction unit 146 extracts a feature vector. do. The communication unit 142 transmits the feature vector to the server 200 and executes user identification processing of the server 200 and the like. The robot 100 determines motion selection and the like according to the identification result. As long as unprocessed enlarged parallel planes 168 remain (N of S18), similar image recognition is performed for other enlarged parallel planes 168 as well. When all enlarged parallel planes 168 have been confirmed (Y in S18), the process ends.

When owner X is reflected in both enlarged parallel plane 168 (TF) and enlarged parallel plane 168 (F), an image of owner X is displayed based on both enlarged parallel plane 168 (TF) and enlarged parallel plane 168 (F). can be processed. For example, it is assumed that owner X's mouth is shown on enlarged parallel plane 168(F), and owner X's entire face is shown on enlarged parallel plane 168(TF). In this case, the recognition unit 156 cannot extract the owner X from the enlarged parallel plane 168(F), but can extract the owner X from the enlarged parallel plane 168(TF). The recognition unit 156 performs recognition by synthesizing the results extracted from each enlarged parallel plane 168 . Moreover, since the arrangement relationship of each enlarged parallel plane 168 is known, the direction in which the owner X exists can also be specified from the enlarged parallel plane 168 (TF) in which the owner X has been successfully extracted. Since the rhombicoctahedron 166 completely surrounds the celestial image 170, the external world can be projected onto the polyhedral image 190 without blind spots. Furthermore, since the enlarged parallel plane 168 is an enlarged square face of the rhomboctahedron 166 , some pixels of the celestial image 170 are multi-projected onto a plurality of enlarged parallel planes 168 . Therefore, it becomes easier to image (capture) an external object from an angle suitable for image recognition.

(Second Embodiment: Multiple Cylindrical Projection Method)
FIG. 17 is a first schematic diagram for explaining the multiple cylindrical projection method.
In the multiple cylindrical projection, the celestial image 170 is projected onto a single cylindrical image 164, similar to the equirectangular projection described in connection with FIG. In the multiple cylindrical projection method, multiple single cylindrical images 164 are generated by setting multiple cylindrical surfaces 162 for one celestial image 170 . 17, similarly to FIG. 10, the cylinder center line 180 of the cylindrical surface 162 and the celestial sphere center line 174 of the celestial sphere surface 140 are aligned. Note that in the multiple cylindrical projection method, there is little distortion near the cylinder center line 180, and the distortion increases as the distance from the cylinder center line 180 increases.

In FIG. 17, point A1 on celestial sphere 140 is closer to cylinder center line 180 than point A2. Point A1 on celestial surface 140 corresponds to point B1 on celestial image 170 and to point S1 on single cylindrical image 164a. Point A2 corresponds to point B2 in celestial image 170, but is not projected onto single cylindrical image 164a. As described above, when image conversion is performed using the equirectangular projection with the cylinder center line 180 and the celestial center line 174 aligned, the single cylindrical image 164a is greatly distorted near the point A2 far from the cylinder center line 180. to put away. Therefore, in the multiple cylindrical projection method, the image conversion unit 154 performs image conversion on only the center partial image 176a (portion with small distortion) of a predetermined width including the cylinder center line 180. FIG. That is, when the cylinder center line 180 and the celestial sphere center line 174 coincide, the multiple cylindrical projection method does not perform image transformation near the poles.

In the multiple cylindrical projection method, while changing the relative angle between the cylinder center line 180 and the celestial sphere center line 174, image conversion is performed only for the central partial image 176 including the cylinder center line 180. FIG. In other words, the image conversion unit 154 performs image conversion only on portions with little distortion during image conversion. A multiple cylindrical image 400 (projection image), which will be described later, is formed as an aggregation of the plurality of central partial images 176 .

FIG. 18 is a second schematic diagram for explaining the multiple cylindrical projection method.
In FIG. 18, the cylinder centerline 180 of the cylindrical surface 162 and the celestial sphere centerline 174 of the celestial sphere 140 intersect perpendicularly. Therefore, the poles of the celestial sphere 140 are located on the cylinder center line 180 .

In FIG. 18, point A3 on celestial sphere 140 is located on cylinder center line 180. In FIG. Point A3 on celestial surface 140 corresponds to point B3 in celestial image 170 and to point S3 in single cylindrical image 164b. On the other hand, point A4, far from cylinder centerline 180, corresponds to point B4 in celestial image 170, but is not projected onto single cylinder image 164b. In FIG. 18 as well, the image conversion unit 154 performs image conversion on only the center partial image 176b of a predetermined width including the cylinder center line 180. FIG.

FIG. 19 is a conceptual diagram of a multiple cylindrical image 400 generated by the multiple cylindrical projection method.
The image conversion unit 154 generates a plurality of central partial images 176 while changing the relative angle between the celestial sphere 140 and the cylindrical surface 162 , in other words, while changing the relative angle between the cylinder center line 180 and the celestial sphere center line 174 . In the second embodiment, four central partial images 176 are obtained for four relative angles of 0 (degrees), 30 (degrees), 60 (degrees), and 90 (degrees) between the cylinder center line 180 and the celestial center line 174. (Central partial images 176a to 176d) are generated. By collecting central partial images 176 with small distortion in this way, it is possible to generate a multiple cylindrical image 400 as a projection image with small distortion. Even with multiple cylindrical projections, portions of a single external object may be overprojected onto multiple central partial images 176 .

FIG. 20 is a flow chart showing the image processing process in the second embodiment.
Also in the second embodiment, the image acquisition unit 152 periodically acquires the celestial image 170 . The image processing shown in FIG. 20 is repeatedly executed each time the celestial image 170 is acquired.

First, the image acquisition unit 152 acquires the celestial image 170 from the omnidirectional camera 134 (S20). The image converter 154 sets the relative angle between the cylindrical surface 162 and the celestial sphere 140 (the cylinder center line 180 and the celestial sphere center line 174) (S22). The image converter 154 image-converts the celestial sphere image 170 into a central partial image 176 at the specified relative angle (S24). If image conversion has not been completed for the above four relative angles (N of S26), the process returns to S22 and the next relative angle is set (S22). When image conversion is completed for all four relative angles (Y of S26), the recognition unit 156 determines the processing order of image recognition for each central partial image 176 (S28). The method of determining the processing order is the same as the method described in relation to the polyhedral projection method.

After determining the processing order, the recognition unit 156 executes image recognition in cooperation with the server 200 (S30). The robot 100 determines motion selection and the like according to the recognition result. As long as an unprocessed central partial image 176 remains (N of S32), similar image recognition is performed for other central partial images 176 as well. When all central partial images 176 have been confirmed (Y of S32), the process ends.

Since the central partial images 176 partially overlap each other, some pixels of the celestial image 170 are multiple projected onto a plurality of central partial images 176 . Therefore, like the polyhedral projection method, the multiple cylindrical projection method also makes it easier to image (capture) an external object from an angle suitable for image recognition.

(Third Embodiment: Triangulation)
In the third embodiment, a method of measuring the distance to an external object by triangulation using a plurality of omnidirectional cameras 134 will be described. Triangulation using two cameras is widely known. However, a general triangulation method that assumes an image without distortion cannot be applied to the celestial image 170 as it is. In the first embodiment, by projecting the celestial image 170 onto the rhombicoctahedron 166, 18 projected images with little distortion were obtained. Substantially, it can be considered that 18 projection images (enlarged parallel plane 168) are captured by 18 cameras. Therefore, if two omnidirectional cameras 134 are used, it can be considered that a pair of images captured by 18×2 cameras in 18 directions is obtained. Triangulation based on the omnidirectional camera 134 becomes possible by obtaining the parallax of the paired projection images for each direction.

FIG. 21(a) is a front external view of the robot 100 in the third embodiment. FIG. 21(b) is a side external view of the robot 100 in the third embodiment.
The robot 100 in the third embodiment has two horns 112 (horns 112a and 112b). Omnidirectional cameras 134 (omnidirectional camera 134a and omnidirectional camera 134b) are incorporated in horns 112a and 112b, respectively. The distance between omnidirectional camera 134a and omnidirectional camera 134b is known.

FIG. 22 is a schematic diagram for explaining a method of measuring the distance from the robot 100 to the external object 250. FIG.
In FIG. 22, there is an external object 250a at coordinate Z1. The external object 250a may be a user, a pet, or a fixed object such as furniture or a window. In the case of the omnidirectional camera 134b built into the horn 112b, the external object 250a is reflected in the enlarged parallel plane 168(F). At this time, let X1(F) be the coordinates (spatial coordinates) of the projection point on the enlarged parallel plane 168(F). Also in the case of the omnidirectional camera 134a built into the horn 112a, the external object 250a is reflected on the enlarged parallel plane 168(F). Let X2(F) be the coordinates (spatial coordinates) of the projection point on the enlarged parallel plane 168(F) at this time. The distance between X1(F) and X2(F) is known, and let this distance be r. X1(F) and X2(F) may be substituted by position coordinates of horns 112b and 112a. The camera B1 located at the horn 112b and taking the enlarged parallel plane 168(F) as the imaging direction and the camera A1 located at the horn 112a and taking the enlarged parallel plane 168(F) as the imaging direction are used to capture the external object 250a. is the same as multiple imaging. Let P be the angle formed by the straight line X1(F)-X2(F) and the straight line X1(F)-Z1. Let Q be the angle formed by the straight line X2(F)-X1(F) and the straight line X2(F)-Z1. If the line segment X1(F)-X2(F) and the angles P and Q at both ends thereof can be defined, the triangle X1(F)-X2(F)-Z1 can be determined. to 250a can be calculated.

External object 250b is at coordinate Z2. The external object 250b is captured on the enlarged parallel plane 168 (FR) of the omnidirectional camera 134b. Also, an external object 250b is captured on the enlarged parallel plane 168 (FR) of the omnidirectional camera 134a. The camera B2 located at the horn 112b and taking the enlarged parallel plane 168 (FR) in the imaging direction and the camera A1 located at the horn 112a and taking the enlarged parallel plane 168 (FR) in the imaging direction are used to capture the external object 250a. is the same as multiple imaging. Therefore, the distance from the robot 100 to the external object 250b can be measured in the same manner as described above. Such a method enables three-dimensional measurement of the surroundings of the robot 100 with a small amount of equipment. Distance information can be used for object extraction, and object recognition accuracy is improved.

The robot 100 and the robot system 300 including the robot 100 have been described above based on the embodiment.
As described above, when the three-dimensional celestial sphere 140 is projected onto the two-dimensional celestial image 170, the celestial image 170 is partially distorted. When the celestial image 170 is further projected onto the single cylindrical image 164 by equirectangular projection, the single cylindrical image 164 is greatly distorted near the poles. In the first embodiment (polygonal projection method), a rhombic octahedron 166 covering the celestial sphere 140 is assumed, and the celestial sphere 140 (celestial sphere image 170) are projected. Since the rhombic octahedron 166 deviates less from the celestial sphere 140 than the cylindrical surface 162, in other words, since there are no coordinates that deviate extremely from the celestial sphere 140, a less distorted polyhedron image 190 (projection image) is obtained. can be obtained.

Existing image recognition programs can be applied if there is a projection image with little distortion. According to the polygonal projection method, the captured image of the celestial sphere 140 covering the robot 100 can be projected onto a plurality of enlarged parallel planes 168 (projected images). can be visually recognized. Correcting the distortion of the celestial image 170 makes it easier to effectively utilize the existing image recognition program than adapting the image recognition program to the distorted celestial image 170 or the like.

In the first embodiment, by setting a plurality of enlarged parallel planes 168 obtained by enlarging the side surfaces of the orthorhombic octahedron 166 , an external object can be overlapped and projected onto the plurality of enlarged parallel planes 168 . For this reason, even if the image is poor on one enlarged parallel plane 168, there is a possibility that the image can be properly recognized on another enlarged parallel plane 168, so that the accuracy of image recognition can be improved.

In a second embodiment (multiple cylindrical projection method), a plurality of central partial images 176 are acquired by varying the relative angles of the celestial sphere 140 and the cylindrical surface 162 . Even in the equirectangular projection, distortion of the image is small near the cylinder center line 180 . Therefore, by generating a plurality of central partial images 176 having a predetermined width near the cylinder center line 180 corresponding to a plurality of relative angles, it is possible to generate a multiple cylindrical image 400 (projection image) with little distortion as a whole.

Image conversion from the celestial sphere image 170 to the enlarged parallel plane 168 and the central partial image 176 may be performed in parallel using a GPU (Graphics Processing Unit). Also, by using a plurality of GPUs, image conversion can be performed at a higher speed. For example, each of the first to fourth GPUs may execute image conversion processing of the central partial images 176a to 176d in parallel. Alternatively, the transformation function prepared for each enlarged parallel plane 168 may be associated with the GPU on a one-to-one basis, and image transformation processing may be executed in parallel by a plurality of GPUs. Since GPUs can perform simple tasks such as image conversion at high speed, it is easy to increase the image processing speed by mounting multiple GPUs on the robot 100 .

As described in the third embodiment, by mounting a plurality of omnidirectional cameras 134 on the robot 100, the distance from the external object 250 to the robot 100 can be measured in all directions. Since a normal camera has a narrower angle of view than the omnidirectional camera 134, the distance measurable range is limited by the two cameras. On the other hand, the omnidirectional camera 134 has substantially no blind spots. Therefore, by mounting the two omnidirectional cameras 134 on the robot 100, most of the external objects 250 around the robot 100 can be used for distance measurement. As a result, it becomes easier for the robot 100 to grasp the surrounding environment more precisely.

It should be noted that the present invention is not limited to the above-described embodiments and modifications, and can be embodied by modifying constituent elements without departing from the scope of the invention. Various inventions may be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments and modifications. Also, some constituent elements may be deleted from all the constituent elements shown in the above embodiments and modifications.

Although the robot system 300 has been described as being composed of one robot 100, one server 200, and a plurality of external sensors 114, part of the functions of the robot 100 may be implemented by the server 200, or the functions of the server 200 may be implemented by the server 200. may be assigned to the robot 100 . One server 200 may control multiple robots 100 , or multiple servers 200 may cooperate to control one or more robots 100 .

A third device other than the robot 100 and the server 200 may take part of the functions. The aggregate of each function of the robot 100 and each function of the server 200 described with reference to FIG. 6 can also be grasped as one "robot" from a broad perspective. How to distribute a plurality of functions necessary for realizing the present invention to one or more pieces of hardware is determined in consideration of the processing capability of each piece of hardware and the specifications required for the robot system 300. It should be decided.

As described above, the “robot in a narrow sense” refers to the robot 100 that does not include the server 200 , but the “robot in a broad sense” refers to the robot system 300 . It is conceivable that many of the functions of the server 200 will be integrated into the robot 100 in the future.

In the first embodiment (polyhedral projection method), the celestial sphere 140 is projected onto the rhombic octahedron 166 . The polyhedral projection method is applicable not only to the rhombicoctahedron 166 but also to various polyhedrons. For example, the celestial sphere 140 may be projected onto a hexahedron or an icosahedron. As the number of projection planes increases, in other words, as the polyhedron approaches a sphere, the divergence between the celestial sphere 140 and the polyhedron becomes smaller, making it easier to suppress distortion of the projected image.

Polyhedral projection is also applicable to non-polygons. Assuming a plurality of planes (projection planes) facing the omnidirectional camera 134 at different angles, and projecting the pixels of the celestial sphere 140 (celestial sphere image 170) onto these planes, a plurality of projection images can be obtained. can be obtained. At this time, it is not essential that a plurality of planes form a polyhedron. For example, if the projection plane is not set behind the robot 100, the omnidirectional camera 134 can take an image of the rear, but cannot be visually recognized. With such a setting, the robot 100 has a blind spot even though it has a wide field of view. The robot 100 with blind spots may feel more alive. Therefore, it is not essential that the robot 100 can visually recognize all directions. On the other hand, when the celestial sphere 140 is covered with a plurality of projection planes, that is, when the plurality of projection planes form a polyhedron, the robot 100 can visually recognize all directions.

The recognition unit 156 may preferentially process a predetermined enlarged parallel plane 168 among the plurality of enlarged parallel planes 168 . For example, the recognition unit 156 may set a higher processing priority to the enlarged parallel plane 168 corresponding to the upper side surface than to the other enlarged parallel planes 168 . Since the robot 100 is shorter than humans, it is highly likely that an external object to be recognized such as a user is normally viewed in the upper lateral direction (diagonally above the robot 100). Therefore, by prioritizing the processing order of the enlarged parallel plane 168 corresponding to the upper side surface such as the enlarged parallel plane 168 (TF), it becomes easier to detect the user to be recognized. The same applies when setting the processing order for the plurality of central partial images 176 .

In the multiple cylindrical projection method, the relative angle between the cylindrical center line 180 and the celestial sphere center line 174 is arbitrarily adjusted so that the multiple cylindrical image 400 includes many skin-colored pixels representing human characteristics, for example, the face. may According to such a control method, a person's face, which is likely to be an object of image recognition, can be easily included in the central partial image 176 .

In the present embodiment, the robot 100 is described as a target, but the functions of the image acquisition unit 152, the image conversion unit 154, and the recognition unit 156 may be componentized as an image recognition device.

The robot system 300 does not need to be equipped with image correction functions such as polyhedral projection and multiple cylindrical projection when shipped from the factory. After the robot system 300 is shipped, the robot system 300 may be enhanced in functionality by installing an image processing program that implements the image correction function in the robot 100 .

(Fourth Embodiment: Rotational Polyhedral Projection Method)
In the fourth embodiment, a "rotational polyhedral projection method", which is an improvement of the polyhedral projection method described in the first embodiment, will be described.
In the rotating polyhedral projection method (fourth embodiment), the celestial image 170 is projected onto each face of the polyhedron, as in the polyhedral projection method (first embodiment). In the fourth embodiment, the surface of the polyhedron on which the celestial image 170 is projected is called a "development surface", and the photographed image projected onto the development surface of the polyhedron is called a "development image". In the polyhedral projection method (first embodiment), since the joint between the developed image (projected image) and the developed image tends to be a blind spot in image recognition, the expanded plane is enlarged to generate an enlarged parallel plane 168 . In other words, it is a system in which the celestial sphere 140 is largely and thoroughly wrapped by the large enlarged parallel surface 168 .

In the rotating polyhedral projection method (fourth embodiment), instead of eliminating blind spots with a large enlarged parallel plane 168, the polyhedron is moved to change the position of the joint between the developed images. If the seams (blind spots) between the developed images are shifted in time series, for example, during image conversion (during projection) at one point in time, the projected object at the seams (blind spots) will be displayed at the time of the next image conversion. It can be projected in the center of the developed image. Since the system eliminates blind spots by rotating the polyhedron, there is no need to prepare a large enlarged parallel surface 168 . For this reason, the overlapped portion generated in the enlarged parallel surface 168, the so-called "overlap portion" can be eliminated or reduced.
In the fourth embodiment, the celestial image 170 is projected onto the truncated pentagonal pyramid 402 instead of the rhombicoctahedron 166 (see FIG. 11).

FIG. 23 is a perspective view of a truncated pentagonal pyramid 402. FIG.
The truncated pentagonal pyramid 402 is a three-dimensional shape obtained by cutting off the upper part of the pentagonal pyramid. The pentagonal truncated pyramid 402 has a total of six surfaces, including five side surfaces 404 (trapezoid) and one zenith surface 406 (regular pentagon). The bottom is not assumed. Assume a square enlarged side surface 408 containing side surface 404 (trapezoid) and a square enlarged zenith surface 410 containing zenith surface 406 (regular pentagon). The five enlarged side surfaces 408 and the one zenith surface 406 are developed surfaces. Also in the fourth embodiment, the celestial image 170 is projected onto five enlarged side surfaces 408 and one enlarged zenith plane 410 by the polyhedral projection method shown in the first embodiment.

The image acquisition unit 152 (omnidirectional camera 134) of the fourth embodiment acquires about 30 to 50 celestial images 170 per second. The number of celestial images 170 acquired by the image acquisition unit 152 per second is called a “celestial image rate”. The celestial images 170 continuously acquired at acquisition timings tc1, tc2, . The celestial image rate may vary depending on the status of computational resources such as CPUs.

The image conversion unit 154 does not convert all the celestial images 170 acquired by the image acquiring unit 152, but selects some of the continuously acquired celestial images 170 as conversion targets. For example, the image conversion unit 154 picks up the celestial image 170 to be converted once every five times, such as the celestial image 170 (tc1), the celestial image 170 (tc5), and the celestial image 170 (tc9). You may In this case, the celestial image 170 (tc2) and the like are excluded from the image conversion processing and image recognition processing.

Hereinafter, the timings at which the image conversion unit 154 picks up the celestial image 170 to be converted are expressed as "t1, t2, t3...". In the above example, t1=tc1, t2=tc5, and t3=tc9. Image conversion and image recognition processing of the celestial image 170 take more time than the image acquisition unit 152 acquires the celestial image 170 . Also, the higher the thinning rate, the more the processing load of image conversion and image recognition can be reduced. The celestial image 170 acquired at time t1 (tc1) is hereinafter referred to as "celestial image 170 (t1)".
Hereinafter, the number of times the image conversion unit 154 performs image conversion on the celestial image 170 per second will be referred to as an “image conversion rate”. Image conversion is performed while thinning out the celestial image 170 acquired by the image acquiring unit 152, so the image conversion rate is smaller than the celestial image rate.

FIG. 24 is a schematic diagram of the truncated pentagonal pyramid 402(t1) viewed from above.
The pentagonal truncated pyramid 402 has five enlarged side surfaces 408 (development surfaces) so as to surround the omnidirectional camera 134 located at the center point O. FIG. In FIG. 24, an external object 412 (eg, a person) exists beyond the boundary line 414 between enlarged side 408(F) and enlarged side 408(FL). Therefore, at time t1, the external object 412 appears near the boundary between the captured images of both the enlarged side surface 408(F) and the enlarged side surface 408(FL).

The image conversion unit 154 rotates the truncated pentagonal pyramid 402 by 36 degrees. Specifically, first, the image conversion unit 154 projects the celestial image 170(t1) onto the six development planes of the pentagonal frustum 402(t1) at time t1. A total of six developed images projected onto five enlarged side surfaces 408 and one enlarged zenith surface 410 of the pentagonal frustum 402(t1) are subjected to image recognition processing. Six developed images generated based on one celestial image 170 are collectively referred to as a "developed image group".

Next, at time t2, the image conversion unit 154 changes the relative angle of the pentagonal frustum 402(t1) with respect to the omnidirectional camera 134 by 36 degrees. In other words, the truncated pentagonal pyramid 402 is rotated about the center point O by 36 degrees. As a result, an object projected on the boundary of a developed image group at a certain point in time is projected onto an area other than the boundary in any of the developed images included in the temporally adjacent developed image group. Since the truncated pentagonal pyramid 402 is rotated by 36 degrees (hereinafter referred to as "partial rotation"), the truncated pentagonal pyramid 402 rotates once by ten partial rotations. Therefore, the pentagonal frustum 402(t11) at time t11 coincides with the pentagonal frustum 402(t1) again.

FIG. 25 is a schematic diagram of the truncated pentagonal pyramid 402(t2) viewed from above.
As noted above, the pentagonal frustum 402(t2) is partially rotated counterclockwise by 36 degrees from the pentagonal frustum 402(t1). Considering the movement of general objects, the image conversion rate is relatively high (short period), so the external object 412 hardly moves from time t1 to time t2. Since the central angle of the pentagon is 72 degrees, the enlarged side surface 408 (F) of the pentagonal frustum 402 (t 2 ) faces the external object 412 . Thus, in the pentagonal frustum 402(t2), the external object 412 is projected onto the enlarged side surface 408(F).

FIG. 26 is an exploded view of the truncated pentagonal pyramid 402 .
In FIG. 26, the five enlarged side surfaces 408 of the pentagonal truncated pyramid 402(t1) and the five enlarged side surfaces 408 of the pentagonal truncated pyramid 402(t2) are displayed side by side. Since the truncated pentagonal pyramid 402(t1) and the truncated pentagonal pyramid 402(t2) are shifted by 36 degrees in rotation angle (angle relative to the center point O), the position of the boundary line 414 is different between the time t1 and the time t2.

In the pentagonal frustum 402(t1), the external object 412 is reflected on the boundary 414 between the enlarged side 408(F) and the enlarged side 408(FL). As described in connection with the first embodiment, the enlarged side 408 tends to distort the projected image at its ends. Further, it is necessary to perform image recognition based on the two expanded images of the enlarged side surface 408(F) and the enlarged side surface 408(FL), which increases the processing load. In other words, the vicinity of the boundary line 414 tends to be a blind spot in image processing. In the first embodiment (polyhedral projection method), a large enlarged parallel plane 168 is set, and the external object 412 is multiple-projected onto a plurality of enlarged parallel planes 168 to cover blind spots. Since the polyhedral projection method of the first embodiment has more overlapping parts than the rotating polyhedral projection method of the fourth embodiment, the size of the captured image (expanded image) to be processed tends to be large.

In contrast, in the truncated pentagonal pyramid 402(t2) of the fourth embodiment (rotated polyhedron projection method), the external object 412 appears in the center of the enlarged side surface 408(F). Therefore, the image of the external object 412 can be preferably recognized from the projected image of the truncated pentagonal pyramid 402(t2) without preparing a large enlarged parallel plane 168 to cover the blind spot. In other words, even if the external object 412 cannot be projected properly on the pentagonal frustum 402(t1), the external object 412 can be projected suitably on the next pentagonal frustum 402(t2). In the example shown in FIG. 26, even if the external object 412 cannot be image-recognized in the developed image (projected image) for the pentagonal frustum 402 (t1), the external object 412 cannot be recognized in the developed image (projected image) for the pentagonal frustum 402 (t2). Object 412 can be image-recognized.

FIG. 27 is a schematic diagram for explaining the rotation of the enlarged zenith plane 410. FIG.
The robot 100 in this embodiment is shorter than humans. Since there are more opportunities for the user to look into the robot 100 from above, there are more opportunities for the omnidirectional camera 134 to acquire the user's face image on the enlarged zenith plane 410 (zenith plane 406).

A general image recognition program is not good at face recognition processing in an image in which a person is reflected upside down. Since the user often looks into the robot 100 from the rear upper part, the user is likely to appear upside down even on the enlarged zenith plane 410 . In FIG. 27, the user is shown upside down on the enlarged zenith plane 410(t1). As described above, the image conversion unit 154 rotates the pentagonal frustum 402 by 36 degrees. Therefore, even if the user is upside down on the enlarged zenith plane 410(t1), the user is photographed in the correct vertical position on the enlarged zenith plane 410(t6).

From the pentagonal frustum 402(t1) to the pentagonal frustum 402(t10), ten enlarged zenith planes 410 can be acquired at ten different angles. The image conversion unit 154 executes image recognition processing for each of the ten enlarged zenith planes 410, and can recognize a face when the enlarged zenith plane 410 (t6) that is vertically correct is targeted. Rotating the truncated pentagonal pyramid 402 is also effective in obtaining a subject image that is properly vertically positioned on the enlarged zenith plane 410 . In other words, instead of rotating the developed image (projected image) that has already been acquired for image recognition, the enlarged zenith plane 410 is partially rotated to acquire a large number of enlarged zenith planes 410 at various angles, If image recognition processing is performed on each of them, one of them satisfies a suitable condition and the subject can be correctly recognized by the image recognition processing.

FIG. 28 is a schematic diagram showing how a plurality of enlarged side surfaces 408 are spliced together.
In the first embodiment (polyhedral projection method), since enlarged parallel planes 168 are created, one object is often projected onto a plurality of enlarged parallel planes 168 at the same time. In other words, the plurality of enlarged parallel planes 168 have many overlapping portions. In contrast, in the fourth embodiment (rotational polyhedral projection method), blind spots are covered by the rotation of the truncated pentagonal pyramid 402, so there is no need to prepare a large enlarged side surface 408 and enlarged zenith surface 410. Few.

Therefore, in the fourth embodiment, the image conversion unit 154 includes an enlarged zenith surface 410 (T) above the robot 100, an enlarged side surface 408 (F) corresponding to the front, and an enlarged side surface 408 (FL) and an enlarged side surface 408 (F) corresponding to the left and right sides of the robot 100. FR) can be stitched together to form a composite image 420 . Synthetic image 420 is closer to what the robot 100 actually sees, because each projected image has less overlap.

The recognition unit 156 may substantially perform image recognition on three developed images of the enlarged side surface 408 (RL), the enlarged side surface 408 (RR), and the composite image 420 . In the case of the rotational polyhedral projection method, the overlapped portion of each captured image is small, so it is easy to create a seamless composite image 420 . Since even six developed images can be processed as three developed images, the processing cost associated with image recognition processing can be suppressed.

The rotated polyhedral projection method shown as the fourth embodiment can cover the blind spot (boundary line 414) of the truncated pentagonal pyramid 402 by partially rotating the truncated pentagonal pyramid 402 without creating a large enlarged parallel plane 168. Since it is not necessary to create a large enlarged parallel plane 168, in other words, since there are few overlapped portions in a plurality of captured images, the processing cost associated with image recognition can be reduced. In addition, a relatively seamless composite image 420, such as that shown in FIG. 28, can be created due to the small amount of overlap. Reducing the number of developed images to be subjected to image recognition processing by creating a large composite image 420 also contributes to the reduction of processing costs.
In addition, by rotating the enlarged zenith plane 410, an object present above the robot 100, particularly a person, can be imaged in an orientation suitable for image recognition.

In the fourth embodiment, enlarged side 408 has been described as being a square developed surface that includes side 404 . Alternatively, the celestial image 170 may be projected onto the side surface 404 of the trapezoid or the zenith surface 406 of the regular pentagon. The projection target need not be the truncated pentagonal pyramid 402, and may be a pentagonal pyramid or a pentagonal prism. Polygons used in the rotational polyhedral projection method may be polygons having multiple faces, and may be rhombic octahedrons. A polygon that lacks a part of the surface such as the bottom surface as in the fourth embodiment may also be used. Each time a developed image group is created at a predetermined image conversion rate, the image conversion unit 154 rotates the projection destination polyhedron by a predetermined angle with respect to the position of the polyhedron when it was projected immediately before, and converts it into a developed image. Just do it.

As a result of studies by the present inventors, the distortion of the developed image (projected image) increases when the number of side surfaces is four or less. Six or more sides may be prepared, but since it has been found that sufficient image recognition accuracy can be obtained with five sides, five sides are considered to be the best at this stage from the viewpoint of the balance between image recognition accuracy and processing cost.

Although the developed surface has been described as a square surface, it may be formed as a rectangular surface. Also, in the fourth embodiment, the truncated pentagonal pyramid 402 is partially rotated in units of 36 degrees, but the angle of rotation is arbitrary. However, it is desirable to rotate at an angle that is a divisor of the central angle. For example, a pentagon has a central angle of 72 degrees, so 36 degrees (1/2), 24 degrees (1/3), 18 degrees (1/4), 12 degrees (1/6), etc., "72 degrees" It is desirable to partially rotate by an angle that is a divisor of . The rotation direction of the truncated pentagonal pyramid 402 is also arbitrary, and may be vertical rotation instead of horizontal rotation.

The truncated pentagonal pyramid 402 may be rotated by 36 degrees, or the relative angle to the center point O may be repeatedly changed between 0 degrees and 36 degrees. In this case, the pentagonal truncated pyramid 402 vibrates rotationally. Even in such a vibration method, it is possible to cover blind spots in image processing in the developed image. Even in the case of the vibration method, the enlarged zenith surface 410 may be rotated.

The angle of partial rotation of the truncated pentagonal pyramid 402 need not be constant. For example, when the truncated pentagonal pyramid 402 is partially rotated at a constant angle at a constant image conversion rate, if there is a person moving around the robot 100 at the same angular velocity, that person will always be in the same area of the developed image. be positioned. If the area is located at the center of the expanded image, it can always be recognized, but if it is positioned at the boundary, the effect of partial rotation cannot be obtained. Therefore, by varying the angle of partial rotation, for example, by randomly changing the angular velocity in the range of 30 degrees to 42 degrees with the median value of 36 degrees, the external object moving at a constant speed can be moved to the boundary of the unfolded image. , that is, the state of being continuously positioned in a blind spot in image processing may be avoided.

Claims (14)

an image acquisition unit that acquires a celestial image as a captured image of the celestial sphere centered on the camera;
an image conversion unit that obtains a plurality of projected images by projecting the pixels of the celestial image onto a plurality of planes that face the camera at different angles;
By periodically executing image recognition processing for each of the plurality of projection images, an external
and a recognition unit that recognizes images of objects,
When the recognition unit detects the external object from the first projection image at a first time point,
, with respect to the first projection image at a second time point that is the processing timing subsequent to the first time point
1. An image processing apparatus characterized in that an image recognition process for a projected image is executed with priority over other projected images . A recognition unit that detects a single external object projected on both the first projection image and the second projection image by image recognition processing of each of the first projection image and the second projection image. 2. The image processing apparatus according to claim 1, characterized by: 3. The image processing apparatus according to claim 1, wherein the plurality of planes form a polyhedron covering the camera. an image acquisition unit that acquires a celestial image as a captured image of the celestial sphere centered on the camera;
Pixels of the celestial image for a plurality of planes facing the camera at different angles
an image conversion unit configured to acquire a plurality of projection images by projecting the image conversion unit, wherein the image conversion unit further changes the relative angles of the plurality of planes with respect to the camera in time series to obtain a plurality of relative projection images; An image processing apparatus that acquires the plurality of projected images corresponding to respective angles. an image acquisition unit that acquires a celestial image as a captured image of the celestial sphere centered on the camera;
an image conversion unit that acquires a plurality of projection images by projecting pixels of the celestial image onto a plurality of cylindrical images surrounding the camera at different angles with respect to the camera. image processing device. A function to acquire a celestial image as a captured image of the celestial surface centered on the camera;
For a plurality of planes facing the camera at different angles,
a function of obtaining a plurality of projected images by projecting the pixels of the celestial image;
By periodically executing image recognition processing for each of the plurality of projection images, an external
Let the computer demonstrate the function of recognizing images of objects,
When the recognizing function detects the external object from the first projection image at a first time point
, with respect to the first projection image at a second time point that is the processing timing subsequent to the first time point
This is a function that gives priority to the image recognition processing that
An image processing program characterized by: A function to acquire a celestial image as a captured image of the celestial surface centered on the camera;
and a function of obtaining a plurality of projection images by projecting the pixels of the celestial image onto a plurality of cylindrical images surrounding the camera at different angles with respect to the camera. An image processing program characterized by: a camera capable of simultaneously imaging substantially the entire area above the robot;
an image conversion unit that acquires from the camera a celestial image, which is a captured image of the celestial sphere covering above the robot, and converts the celestial image into a polyhedral image;
a recognition unit that recognizes an external object from the polyhedral image,
The image conversion unit sets an image whose imaging plane is a polyhedron that covers the upper part of the robot as the polyhedron image, and projects the pixels of the celestial image onto one of the surfaces of the polyhedron, thereby transforming the celestial image into the polyhedron. convert to image,
The image conversion unit expands the plurality of faces of the polyhedron in parallel to obtain a plurality of images.
, and multiple-projects pixels of the celestial image onto two or more enlarged parallel planes . 9. The robot according to claim 8 , wherein the polyhedron includes a plane perpendicular to an axis connecting the center of the celestial sphere and the zenith. 9. The robot according to claim 8 , wherein the image conversion unit simultaneously executes projection processing for each of a plurality of pixels included in the celestial image. a camera capable of simultaneously imaging substantially the entire area above the robot;
an image conversion unit that acquires from the camera a celestial image, which is a captured image of the celestial sphere covering above the robot, and converts the celestial image into a multiple cylindrical image;
a recognition unit that recognizes an external object from the multiple cylindrical image,
The image conversion unit forms a cylindrical image by projecting the pixels of the celestial image onto a side surface of a cylinder surrounding the robot, and multiplexes a set of a plurality of cylindrical images corresponding to a plurality of relative angles between the cylinder and the celestial sphere. A robot characterized by converting said celestial image into said multiple cylindrical image by setting it as a cylindrical image. 12. The robot according to claim 11 , wherein said image conversion unit further sets a set of a plurality of partial images extracted from the center of each of said plurality of cylindrical images as said multiple cylindrical image. A function to acquire a celestial image, which is a captured image of the celestial sphere covering the upper part of the robot;
A function of converting the celestial image into the polyhedral image by setting an image with the polyhedron covering the upper part of the robot as an imaging surface as a polyhedral image and projecting the pixels of the celestial sphere image onto one of the surfaces of the polyhedron; a function of image recognition of an external object from the polyhedral image;
causing a computer to exhibit a function of selecting the motion of the robot in response to the external object ;
The image recognition function is performed by expanding the plurality of faces of the polyhedron in parallel.
A robot action control program characterized by a function of setting an enlarged parallel plane and multiple-projecting the pixels of the celestial sphere image onto two or more enlarged parallel planes . A function of acquiring a celestial image, which is a captured image of the celestial sphere covering the upper part of the robot, from a camera capable of simultaneously capturing substantially the entire area above the robot ;
Forming a cylindrical image by projecting the pixels of the celestial image onto the side of a cylinder surrounding the robot, and setting a set of a plurality of cylindrical images corresponding to a plurality of relative angles between the cylinder and the celestial sphere as a multiple cylindrical image. a function of converting the celestial image into the multiple cylindrical image by
a function of image recognition of an external object from the multiple cylindrical image;
A behavior control program for a robot, characterized by causing a computer to exhibit a function of selecting a motion of the robot corresponding to the external object.

Images