JPWO2019240208A1 - Robots and their control methods, as well as programs - Google Patents

Abstract

The robot moves by a moving mechanism, a photographing unit that captures the surrounding space, a marker recognition unit that recognizes a predetermined marker included in the captured image captured by the photographing unit, and a moving mechanism based on the recognized markers. It is provided with a movement control unit for controlling.

Description

The present invention relates to a robot, a control method thereof, and a program.

Conventionally, there is a robot that takes an image with a camera while autonomously moving inside the house, recognizes the indoor space from the captured image, sets a movement route based on the recognized space, and moves indoors. The robot movement route is set by the user creating in advance a map that defines the route on which the robot moves. The robot can move a route determined based on the created map (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2016-103277

However, in the autonomous behavior type robot, since the robot can move autonomously in the space, for example, the robot may enter a range that the user does not want to enter or a dangerous range when the robot moves.

In addition to autonomous movement, there are also cases where a robot is moved to a predetermined location to perform a predetermined action.

The present invention has been made in view of the above circumstances, and one object of the present invention is to provide a robot, a control method thereof, and a program capable of controlling the movement of the robot by a user in one embodiment.

(1) In order to solve the above problems, the robot of the embodiment has a moving mechanism, a photographing unit that photographs the surrounding space, and a marker that recognizes a predetermined marker included in the captured image captured by the photographing unit. It includes a recognition unit and a movement control unit that controls movement by the movement mechanism based on the recognized marker.

(2) Further, in the robot of the embodiment, the movement control unit prohibits the approach by the movement based on the recognized marker.

(3) Further, in the robot of the embodiment, the movement control unit limits the speed of the movement based on the recognized marker.

(4) Further, in the robot of the embodiment, the movement control unit controls the movement based on the recognized installation position of the marker.

(5) Further, in the robot of the embodiment, the movement control unit sets a limited range based on the installation position and limits the movement in the limited range.

(6) Further, in the robot of the embodiment, the movement control unit sets a predetermined range as the limitation range on the back side of the installation position or around the installation position.

(7) Further, in the robot of the embodiment, when there are a plurality of the recognized markers, the movement control unit limits the movement based on the recognized plurality of the installed positions.

(8) Further, in the robot of the embodiment, the movement control unit limits the movement based on a line segment connecting the recognized first marker installation position and the recognized second marker installation position.

(9) Further, in the robot of the embodiment, the movement control unit controls the movement based on the recognized type of the marker.

(10) Further, in the robot of the embodiment, the movement control unit controls the movement based on the recorded marker.

(11) Further, in the robot of the embodiment, when the marker is not recognized in the captured image, the movement control unit controls the movement based on the recorded marker.

(12) Further, in the robot of the embodiment, a spatial data generation unit that generates spatial data that recognizes the space based on the captured image captured by the imaging unit, and a spatial data generation unit that generates the spatial data based on the generated spatial data. It further includes a visualization data generation unit that generates visualization data that visualizes spatial elements included in the space, and a visualization data providing unit that provides the generated visualization data to a user terminal.

(13) Further, the robot of the embodiment further includes a designation acquisition unit that acquires the designation of the area included in the provided visualization data from the user terminal, and the spatial data generation unit is the acquired designation. The space is re-recognized based on the photographed image re-photographed in the region.

(14) Further, the robot of the embodiment further includes a state information acquisition unit that acquires state information indicating the state of the movement destination in the movement, and the movement control unit further controls the movement based on the state information. ..

(15) Further, in the robot of the embodiment, a marker information storage unit that stores the position of the marker, a first event detection unit that detects the first event, a second event detection unit that detects the second event, and the like. It further includes an action execution unit that executes an action, and when the first event is detected, it moves to the vicinity of the position of the marker, and when the second event is detected, the marker and the first event And perform the action corresponding to at least one of the second events.

(16) In order to solve the above problem, the robot control method of the embodiment includes a shooting step of shooting the surrounding space and a marker recognition step of recognizing a predetermined marker included in the shot image shot in the shooting step. And a movement control step that controls the movement by the movement mechanism based on the recognized marker.

(17) In order to solve the above problem, the robot control program of the embodiment recognizes on the computer a shooting function for shooting the surrounding space and a predetermined marker included in the shot image shot by the shooting function. A marker recognition function and a movement control function for controlling movement by a movement mechanism based on the recognized marker are realized.

According to one embodiment, it is possible to provide a robot, a control method thereof, and a program in which a user can control the movement of the robot.

It is a block diagram which shows an example of the software structure of the autonomous action type robot in Embodiment 1. FIG. It is a block diagram which shows an example of the hardware composition of the autonomous action type robot in Embodiment 1. FIG. It is a flowchart which shows an example of the operation of the autonomous action type robot control program in Embodiment 1. FIG. It is a flowchart which shows another example of the operation of the autonomous action type robot control program in Embodiment 1. FIG. It is a figure which shows the setting method of the entry prohibition line in Embodiment 1. FIG. It is a figure which shows an example of the display of the user terminal in Embodiment 1. FIG. It is a figure which shows an example of the display of the user terminal in Embodiment 1. FIG. It is a block diagram which shows an example of the module structure of the robot in Embodiment 2. It is a block diagram which shows an example of the module structure of the data providing apparatus in Embodiment 2. It is a block diagram which shows an example of the data structure of the event information storage part in Embodiment 2. It is a block diagram which shows an example of the data structure of the marker information storage part in Embodiment 2. FIG. 12A is a flowchart showing a processing procedure in the marker registration phase of the second embodiment. FIG. 12B is a flowchart showing a processing procedure in the action phase of the second embodiment. FIG. 13A is a flowchart showing a processing procedure in the marker registration phase of the first embodiment. FIG. 13B is a flowchart showing a processing procedure in the action phase of the first embodiment. FIG. 14A is a flowchart showing a processing procedure in the marker registration phase of the second embodiment. FIG. 14B is a flowchart showing a processing procedure in the action phase of the second embodiment. FIG. 15A is a flowchart showing a processing procedure in the marker registration phase of the third embodiment. FIG. 15B is a flowchart showing a processing procedure in the action phase of the third embodiment. FIG. 16A is a flowchart showing a processing procedure in the marker registration phase of the fourth embodiment. FIG. 16B is a flowchart showing a processing procedure in the action phase of the fourth embodiment. FIG. 17A is a flowchart showing a processing procedure in the marker registration phase of the fifth embodiment. FIG. 17B is a flowchart showing a processing procedure in the action phase of the fifth embodiment. FIG. 18A is a flowchart showing a processing procedure in the marker registration phase of the sixth embodiment. FIG. 18B is a flowchart showing a processing procedure in the action phase of the sixth embodiment.

Hereinafter, the autonomous action robot, the data providing device, and the data providing program according to the embodiment of the present invention will be described in detail with reference to the drawings.

[Embodiment 1]
First, the software configuration of the autonomous action type robot 1 will be described with reference to FIG. FIG. 1 is a block diagram showing an example of a software configuration of the autonomous action type robot 1 in the embodiment.

In FIG. 1, the autonomous action type robot 1 has a data providing device 10 and a robot 2. The data providing device 10 and the robot 2 are connected by communication to function as an autonomous action type robot 1. The robot 2 is a mobile robot having each functional unit of a photographing unit 21, a marker recognition unit 22, a movement control unit 23, a state information acquisition unit 24, and a movement mechanism 29. The data providing device 10 has each functional unit of the first communication control unit 11, the point cloud data generation unit 12, the spatial data generation unit 13, the visualization data generation unit 14, the imaging target recognition unit 15, and the second communication control unit 16. .. The first communication control unit 11 has each functional unit of the captured image acquisition unit 111, the spatial data provision unit 112, and the instruction unit 113. The second communication control unit 16 has each functional unit of the visualization data providing unit 161 and the designated acquisition unit 162. Each of the above-mentioned functional units of the data providing device 10 of the autonomous action type robot 1 in the present embodiment will be described as a functional module realized by a data providing program (software) that controls the data providing device 10. Further, each functional unit of the marker recognition unit 22, the movement control unit 23, and the state information acquisition unit 24 of the robot 2 is described as a functional module realized by a program for controlling the robot 2 in the autonomous action type robot 1. To do.

The data providing device 10 is a device capable of executing a part of the functions of the autonomous action type robot 1. For example, the data providing device 10 is installed at a place physically close to the robot 2 and communicates with the robot 2 to communicate with the robot 2. It is an edge server that distributes the processing load. In the present embodiment, the case where the autonomous action type robot 1 is composed of the data providing device 10 and the robot 2 will be described, but the function of the data providing device 10 is included in the function of the robot 2. May be good. Further, the robot 2 is a robot that can move based on spatial data, and is one aspect of a robot in which a moving range is determined based on spatial data. The data providing device 10 may be composed of one housing or a plurality of housings.

The first communication control unit 11 controls the communication function with the robot 2. The communication method with the robot 2 is arbitrary, and for example, wireless LAN (Local Area Network), Bluetooth (registered trademark), short-range wireless communication such as infrared communication, or wired communication can be used. Each function of the captured image acquisition unit 111, the spatial data provision unit 112, and the instruction unit 113 possessed by the first communication control unit 11 communicates with the robot 2 by using the communication function controlled by the first communication control unit 11.

The captured image acquisition unit 111 acquires a captured image captured by the photographing unit 21 of the robot 2. The photographing unit 21 is provided on the robot 2 and can change the photographing range as the robot 2 moves. Here, the photographing unit 21, the marker recognition unit 22, the movement control unit 23, the state information acquisition unit 24, and the movement mechanism 29 of the robot 2 will be described.

The photographing unit 21 can be composed of one or a plurality of cameras. For example, when the shooting unit 21 is a stereo camera composed of two cameras, the shooting unit 21 can three-dimensionally shoot a spatial element to be shot from different shooting angles. The photographing unit 21 is a video camera using an image sensor such as a CCD (Charge-Coupled Device) sensor or a CMOS (Complementary Metal Oxide Sensor) sensor, for example. The shape of the spatial element can be measured by photographing the spatial element with two cameras (stereo cameras). Further, the photographing unit 21 may be a camera using ToF (Time of Flight) technology. In the ToF camera, the shape of the spatial element can be measured by irradiating the spatial element with modulated infrared light and measuring the distance to the spatial element. Further, the photographing unit 21 may be a camera using a structured light. A structured light is a light that projects light in a striped or grid pattern onto a spatial element. The photographing unit 21 can measure the shape of the spatial element from the distortion of the projected pattern by photographing the spatial element from an angle different from that of the structured light. The photographing unit 21 may be any one or a combination of two or more of these cameras.

Further, the photographing unit 21 is attached to the robot 2 and moves according to the movement of the robot 2. However, the photographing unit 21 may be installed separately from the robot 2.

The captured image captured by the photographing unit 21 is provided to the captured image acquisition unit 111 in the communication method corresponding to the first communication control unit 11. The captured captured image is temporarily stored in the storage unit of the robot 2, and the captured image acquisition unit 111 acquires the captured image temporarily stored in real time or at a predetermined communication interval.

The marker recognition unit 22 recognizes a predetermined marker included in the captured image captured by the photographing unit 21. The marker is a spatial element that indicates a restriction on the movement of the robot 2. A marker is a shape, pattern or color of an article that can be recognized from a photographed image, characters or figures attached to the article, or a combination thereof. By installing the marker at a position where the user restricts the movement of the robot 2, when the robot 2 photographs the space with the photographing unit 21, the image is taken together with furniture and the like. The marker may be a flat article or a three-dimensional article. The marker is, for example, a sticker or paper on which a two-dimensional code or a specific color combination or shape is printed. Further, the marker may be a figurine or a rug having a specific color or shape. In this way, by using the printed matter or the things around us as the marker, the user does not need to secure the power supply of the marker or secure the installation place. In addition, the movement of the robot can be restricted by the user's will without damaging the atmosphere of the room. In addition, since the user can also visually recognize the marker, the movement restriction range can be intuitively grasped, and the restriction range can be easily changed. Markers are installed by the user, for example by affixing them to walls or furniture, or by placing them on the floor. The marker recognition unit 17 can recognize that the movement of the robot 2 is restricted by recognizing the image of the marker included in the captured image.

Here, when the marker is flat, it can be attached to a wall or furniture, so that it can be installed in a small space. When the marker is flat, if the flat surface of the marker is photographed from the horizontal direction (when the photographing angle is small), the marker in the photographed image is distorted, which makes recognition difficult. On the other hand, if the plane of the marker is photographed from the vertical direction (when the photographing angle is large), the marker can be easily recognized. Therefore, for example, when the marker is attached to the corridor, the robot 2 can be prevented from recognizing the marker because the shooting angle is small at a position far from the marker. When the robot moves in the corridor and approaches the marker, the shooting angle increases, so the marker is recognized. Therefore, in a flat marker, the installation position of the marker (described later) can be brought close to the position where the robot can recognize the marker, so that the robot can accurately grasp the installation position of the marker. .. Further, when the marker is three-dimensional, it can be easily installed in the center of the room or the like. When the marker is three-dimensional, the marker can be recognized from various shooting angles. Therefore, by installing the three-dimensional marker, it is possible to make the robot 2 located at a position far from the marker installation position recognize the marker.

The marker recognition unit 22 stores the visual features of the marker in advance. For example, the marker recognition unit 22 stores in advance a two-dimensional code or a three-dimensional object to be recognized as a marker. The marker recognition unit 22 may recognize an object registered in advance by the user as a marker. For example, when the user registers a flowerpot taken by the camera of the user terminal 3 as a marker, the flowerpot installed in the corridor or the like can be recognized as a marker. Therefore, the user can install a marker that does not give a sense of discomfort at the place where the marker is installed. The marker recognition unit 22 may recognize a spatial element other than an object as a marker. For example, the marker recognition unit 22 may recognize the user's gesture such as crossing the arm in front of the body as a marker. The marker recognition unit 22 recognizes the position where the user makes a gesture as the installation position of the marker.

The marker recognition unit 22 recognizes a position where the marker is attached or installed (hereinafter, referred to as “installation position”). The installation position is a position in the space where the marker is installed in the spatial data. The installation position can be recognized, for example, based on the spatial data recognized by the robot 2, based on the distance between the current position of the robot 2 and the photographed marker. For example, when the size of the marker is known in advance, the marker recognition unit 22 calculates the distance between the robot 2 and the marker from the size of the marker image included in the captured image, and determines the current position and shooting direction of the robot 2 ( For example, the installation position of the marker can be recognized based on the orientation by an azimuth meter (not shown). Further, the installation position may be recognized from the relative position from the spatial element to the marker whose position in the space is already known. For example, when the position of the door is already known, the marker recognition unit 22 may recognize the installation position from the relative position between the marker and the door. Further, when the captured image is captured by the depth camera, the installation position can be recognized based on the shooting depth of the marker photographed by the depth camera.

Further, the marker recognition unit 22 may recognize a plurality of markers included in the captured image. For example, when it is desired to set a range for restricting movement with a straight line, the user can install a marker composed of a pair of two markers including a first marker and a second marker. The marker recognition unit 22 recognizes the position of the line segment (straight line or curve) connecting the start point and the end point by recognizing the installation position (start point) of the first marker and the installation position (end point) of the second marker. Good. The marker recognition unit 22 can recognize the position of the line segment in the spatial data by mapping the positions of the first marker and the second marker to the spatial data. The user can easily set the line segment that restricts the movement by installing the marker at a predetermined position or the like. It may be possible to install three or more markers. For example, when there are three or more markers, the marker recognition unit 22 can recognize a polygonal line or a polygon (area) based on the installation position of each marker.

The movement control unit 23 limits the movement based on the installation position of the marker recognized by the marker recognition unit 22. The movement control unit 23 has a limit range setting unit 231 that sets a limit range that limits movement according to the installed position of the recognized marker. The movement control unit 23 limits the movement of the robot 2 with respect to the limit range set in the limit range setting unit 231. The marker installation position is a point, line, surface, or space set based on the installation position of one or more markers. The limit range setting unit 231 can set the limit range by recognizing the installation position of the marker as, for example, coordinate data in spatial data. The limit range setting unit 231 may set a limit range based on the installation position and limit the movement in the limit range. For example, the restriction range setting unit 231 sets a line segment that divides a space element such as a corridor, or a circular or spherical area in the space centered on the marker as a restriction range that restricts movement based on the installation position of one marker. Can be set. That is, the limit range setting unit 231 sets the limit range by arranging a geometrically determined range such as a rectangle, a circle, or a straight line in the space based on the installation position of the marker. For example, if the range is set in a circular shape, the limiting range setting unit 231 may set a circular range having a predetermined radius as the limiting range centered on the installation position of the marker. Further, if the range is set in a rectangular shape, the limited range setting unit 231 may determine the rectangular limited range by arranging the marker installation position so as to be in the center of one side of the rectangle. The limiting range is, for example, about 1 to 3 m from the marker, which is narrower than the range in which the marker recognition unit 22 can recognize the marker. The limiting range may be predetermined for each marker, or may be arbitrarily adjusted by the user by using an application described later.

Further, the limit range setting unit 231 may set a line, a surface, or a space set by a plurality of markers as a limit range. For example, the limit range setting unit 231 sets a predetermined range as a limit range on the back side of the marker installation position or around the installation position, based on the position of the robot 2 when the marker recognition unit 22 recognizes the marker. You may. When the restriction range setting unit 231 sets the restriction range linearly, the movement control unit 23 restricts the movement of the robot 2 so as not to cross the line. In this way, the limit range setting unit 231 may set the limit range based on a predetermined rule based on the installation position of the marker.

Further, the limiting range setting unit 231 may recognize the spatial feature around the marker and set the limiting range according to the spatial feature. In other words, the limit range setting unit 231 may recognize the floor plan and set the limit range according to the floor plan. For example, the restriction range setting unit 231 may set the passage as the restriction range if the marker is around the entrance of the passage (within a predetermined range). Further, when the marker is arranged in the center of the room (a predetermined distance away from the wall), the restriction range setting unit 231 may set a circular range centered on the marker as the restriction range. .. Further, the limit range setting unit 231 may set a predetermined range from the wall as the limit range if the marker is attached to the wall and there is no door in the vicinity.

A type may be set for the marker. For example, a marker that restricts movement only when the marker is visible (called a "temporary marker") and a marker that stores the position of the marker and permanently restricts movement even if the marker is not visible (called a "permanent marker"). ) And is set as the marker type. When the permanent marker is visually recognized, the robot 2 stores the position of the marker in a storage unit (not shown), and even if the marker disappears from that location, the movement is restricted based on the stored marker position. .. Further, when the temporary marker is visually recognized, the robot 2 does not memorize the position of the temporary marker, so that the restriction range is released when the temporary marker is removed.

The marker recognition unit 22 recognizes the set marker type. The types of markers can be preliminarily classified by, for example, the shape, pattern, color, character or figure of the marker, or a combination thereof. Further, the types of markers may be classified according to the number of markers installed, the marker installation method (for example, installation in which the vertical direction of the markers is changed), or the like. When the marker includes a two-dimensional code, information for specifying the type of the marker is written in the information of the two-dimensional code. In this case, the marker recognition unit 22 can identify whether it is a temporary marker or a permanent marker by reading the two-dimensional code. Further, identification information for identifying the marker (referred to as "marker identification information") may be written in the two-dimensional code. In this case, the marker recognition unit 22 reads the marker identification information from the two-dimensional code and refers to a table prepared in advance to specify the type of marker associated with the marker identification information.

That is, the marker recognition unit 22 may be configured to read the information incidental to the marker itself when the information incidental to the marker itself is included as in the two-dimensional code. Further, the marker recognition unit 22 may be configured to read the marker identification information from the marker and read the incidental information by referring to the table using the marker identification information as a key. In the present embodiment, the marker recognition unit 22 has a marker information storage unit (not shown) that holds the incidental information of the marker in association with the marker identification information, and the marker can be referred to by referring to the marker information storage unit. The case where it is configured so that the incidental information for each can be acquired is illustrated. The marker recognition unit 22 may read the marker identification information from the two-dimensional code, or may acquire the marker identification information by identifying the marker by general object recognition.

In this way, by configuring the information attached to the marker to be managed, not only the restriction range is set around the marker and the movement of the robot 2 is restricted, but also various conditions are met according to the user's wishes. The action of the robot 2 can be restricted (referred to as "action restriction"). For example, when the robot 2 does not want to enter the dressing room during the time when the bathroom is used, the marker is associated with the time zone when entry is prohibited. If the robot 2 does not want to enter the kitchen while using the kitchen, if there is a person (when a person is detected), a condition for prohibiting the entry is associated with the marker. Also, even if the robot is allowed to enter instead of being prohibited from entering, the action must be suppressed and quiet, no noise must be made, or the robot must run slowly within the restricted range. Instructions for restricting the robot's behavior, such as not being, may be associated as incidental information. That is, the incidental information may include information that specifies the type of marker or information that defines the behavior of the robot within the limited range. The information that defines the behavior is information for restricting the behavior of the robot, and if movement within the restricted range is prohibited, in addition to prohibiting it, information that specifies the prohibited time zone, etc. is included. May be. If the incidental information is conditionally permitted to move within the restricted range, the incidental information may include information for specifying the condition (referred to as "behavioral condition") in addition to the permission. ..

When the marker is not recognized by the marker recognition unit 22, the movement control unit 23 may limit the movement based on the stored installation position. For example, when the command by the marker sets a permanent marker that sets a permanent limit, the movement control unit 23 moves based on the marker even if the marker is removed and cannot be recognized from the captured image. Permanently limit. The marker set in the restriction range setting unit 231 may be edited, for example, by an instruction from the user terminal 3, such as erasing, changing the position, or changing the command. For example, the user terminal 3 may have an application program (hereinafter, referred to as “application”) (hereinafter, referred to as “appliqué”) (hereinafter, referred to as “appliqué”), which is not shown and can edit the marker. For example, the application may display the marker on the display screen of the user terminal 3 described above so as to be selectable, and edit the marker selected by the user. The app may also allow the marker to be changed to a permanent marker if the marker is a temporary marker. As a result, the user can release the restriction range by removing the installed temporary marker, and even after removing the installed marker by changing the temporary marker to a permanent marker in the application. It is possible to maintain the limited range. The application may have a registration function for registering a spatial element photographed by the camera of the user terminal 3 described above as a marker. In addition, the application may have a function of adjusting the restriction range or setting or changing the content of the behavior restriction described above. The application may have a function of connecting to the marker information storage unit of the robot 2 and referencing and updating the action restrictions and action conditions for each marker.

The movement restriction of the robot 2 set by the marker can coexist with the setting of the restriction range by the state information described later. For example, the entry prohibited area in the corridor may be set by installing a marker, and the dressing room may be prohibited from entering by the state information. Further, the area where the movement is restricted may be set by the marker, and the content of the restriction in the area (for example, the condition such as the time when the entry is restricted) may be set by the state information.

The state information acquisition unit 24 acquires state information indicating the state of the movement destination in the movement. The state information is information for restricting the movement of the robot 2 according to the state of the movement destination detected by the robot 2. The state of the destination is, for example, the presence / absence of a person, the presence / absence of a pet, the temperature or humidity of the room, the locked state of the door, the lighting state of the lighting, etc., and the state of time, day of the week, weather, etc. May include. The state information is, for example, information for limiting the moving speed in the area (range) when a person is detected in the moving range. In addition, the status information prohibits entry in the area on a predetermined day or time, prohibits movement through the door when the door is locked, or shoots in an area where the light is on. It may be prohibited. The state information can be provided together with the spatial data.

The data providing device 10 will be described again. The spatial data providing unit 112 provides the robot 2 with the spatial data generated by the spatial data generating unit 13. The spatial data is the data of the spatial elements recognized by the robot in the space where the robot 2 exists. The robot 2 can move within the range defined in the spatial data. That is, the spatial data functions as a map for determining the movable range in the robot 2. The robot 2 is provided with spatial data from the spatial data providing unit 112. For example, the spatial data can include position data of spatial elements such as walls, furniture, electric appliances, and steps on which the robot 2 cannot move. Based on the provided spatial data, the robot 2 can determine whether or not it is a place where it can move. Further, the robot 2 may be able to recognize whether or not the spatial data includes an ungenerated range. Whether or not an ungenerated range is included can be determined by, for example, whether or not a part of the spatial data includes a space having no spatial element.

The instruction unit 113 instructs the robot 2 to take a picture based on the spatial data generated by the spatial data generation unit 13. Since the spatial data generation unit 13 creates spatial data based on the captured image acquired by the captured image acquisition unit 111, for example, when creating indoor spatial data, spatial data is not created for a portion that has not been captured. May include the part of. Further, if the captured image is unclear or the like, noise may be included in the created spatial data, and an inaccurate portion may be included in the spatial data. When the spatial data has an ungenerated portion, the instruction unit 113 may give a shooting instruction for the ungenerated portion. Further, when the spatial data includes an inaccurate portion, the instruction unit 113 may give a shooting instruction for the inaccurate portion. The instruction unit 113 may voluntarily instruct shooting based on the spatial data. The instruction unit 113 may instruct shooting based on an explicit instruction from a user who has confirmed the visualization data (described later) generated based on the spatial data. The user can specify the area included in the visualization data and instruct the robot 2 to take a picture, thereby recognizing the space and generating the spatial data.

The point cloud data generation unit 12 generates three-dimensional point cloud data of spatial elements based on the photographed image acquired by the photographed image acquisition unit 111. The point cloud data generation unit 12 generates point cloud data by converting spatial elements included in the captured image into a set of three-dimensional points in a predetermined space. As described above, the spatial elements are room walls, steps, doors, furniture placed in the room, home appliances, luggage, foliage plants, and the like. Since the point cloud data generation unit 12 generates the point cloud data based on the captured image of the spatial element, the point cloud data represents the shape of the surface of the captured spatial element. The captured image is generated by the imaging unit 21 of the robot 2 photographing at a predetermined imaging position at a predetermined imaging angle. Therefore, when the robot 2 photographs a spatial element such as furniture from the front position, point cloud data cannot be generated for the shape of the back side of the furniture that has not been photographed, and the robot 2 moves to the back side of the furniture. Even if there is a possible space, the robot 2 cannot recognize it. On the other hand, when the robot 2 moves and photographs the furniture from the photographing position on the side surface, point cloud data can be generated for the shape of the back side of the space element such as the furniture, so that the space can be correctly grasped.

The spatial data generation unit 13 generates spatial data that determines the movable range of the robot 2 based on the point cloud data of the spatial elements generated by the point cloud data generation unit 12. Since the spatial data is generated based on the point cloud data in the space, the spatial elements included in the spatial data also have three-dimensional coordinate information. The coordinate information may include information on the position, length (including height), area, or volume of the point. The robot 2 can determine the movable range based on the position information of the spatial element included in the generated spatial data. For example, when the robot 2 has a moving mechanism 29 that horizontally moves on the floor surface, the robot 2 has a step from the floor surface, which is a spatial element in the spatial data, at a predetermined height or more (for example, 1 cm or more). In some cases, it can be determined that movement is impossible. On the other hand, in the spatial data, when the table top plate or the bed, which is a spatial element, has a predetermined height from the floor surface, the robot 2 has a height from the floor surface equal to or higher than the predetermined height (for example, 60 cm or more). ) Is judged as a movable range in consideration of the clearance with its own height. Further, the robot 2 determines in the spatial data that the gap between the wall and the furniture, which is a spatial element, is a predetermined width or more (for example, 40 cm or more) as a movable range in consideration of the clearance with its own width. To do.

The spatial data generation unit 13 may set attribute information for a predetermined area in the space. The attribute information is information that defines the movement conditions of the robot 2 for a predetermined area. The movement condition is, for example, a condition that defines a clearance with a space element that the robot 2 can move. For example, when the normal movement condition in which the robot 2 can move is a clearance of 30 cm or more, attribute information can be set in which the clearance for a predetermined area is 5 cm or more. Further, as the movement condition set in the attribute information, information that restricts the movement of the robot may be set. The movement restriction is, for example, a restriction of the movement speed, a prohibition of entry, or the like. For example, in an area where the clearance is small or an area where a person exists, the attribute information in which the moving speed of the robot 2 is reduced may be set. Further, the movement conditions set in the attribute information may be determined by the flooring material of the area. For example, the attribute information may set a change in the operation (running speed, running means, etc.) of the moving mechanism 29 when the floor is a cushion floor, a flow long, a tatami mat, or a carpet. Further, in the attribute information, the above conditions may be set at a charging spot where the robot 2 can move and charge, a step where movement is restricted due to the unstable posture of the robot 2, or the edge of the carpet. .. The area in which the attribute information is set may be grasped by the user by changing the display method in the visualization data described later.

The spatial data generation unit 13 performs, for example, a Hough transform on the point cloud data generated by the point cloud data generation unit 12, extracts a figure such as a straight line or a curve common to the point cloud data, and uses the extracted figure. Spatial data is generated by the contours of the represented spatial elements. The Hough transform is a coordinate transformation method that extracts the figure that passes through the feature points most when the point cloud data is used as the feature points. Since the point cloud data expresses the shape of a spatial element such as furniture placed in a room in a point cloud, the user can determine what the spatial element represented by the point cloud data is (for example,). It may be difficult to recognize the table, chairs, walls, etc.). Since the spatial data generation unit 13 can express the outline of furniture or the like by Hough transforming the point cloud data, it is possible for the user to easily identify the spatial element. The spatial data generation unit 13 converts the point cloud data generated by the point cloud data generation unit 12 into a basic shape of a spatial element (for example, a table, a chair, a wall, etc.) recognized in image recognition, and creates a space. Data may be generated. By recognizing that a spatial element such as a table is a table by image recognition, the shape of the table can be determined from some point cloud data of the spatial element (for example, point cloud data when the table is viewed from the front). It can be predicted accurately. The spatial data generation unit 13 can generate spatial data that accurately grasps the spatial elements by combining the point cloud data and the image recognition.

The spatial data generation unit 13 generates spatial data based on the point cloud data included in a predetermined range from the moved position of the robot 2. The predetermined range from the moved position of the robot 2 includes the position where the robot 2 actually moved, and may be, for example, a range at a distance of 30 cm or the like from the moved position of the robot 2. Since the point cloud data is generated based on the captured image captured by the photographing unit 21 of the robot 2, the captured image may include a spatial element located at a position away from the robot 2. When the space element is separated from the photographing unit 21, there may be a portion that has not been photographed, or there may be a range in which the robot 2 cannot actually move due to the presence of an obstacle that has not been photographed. Further, when a spatial element located far from the photographing unit 21 such as a corridor is included in the captured image, the spatial element extracted at the feature point may be distorted. Further, when the shooting distance is large, the spatial element included in the shot image becomes small, so that the accuracy of the point cloud data may be low. The spatial data generation unit 13 may generate spatial data that does not include spatial elements with low accuracy or distorted spatial elements by ignoring feature points that are far apart. The spatial data generation unit 13 deletes the point cloud data outside the predetermined range from the moved position of the robot 2 to generate the spatial data, thereby preventing the occurrence of a jumping place where the data does not actually exist. , It is possible to generate spatial data with high data accuracy without including the range in which the robot 2 cannot move. In addition, it is possible to prevent drawing of an excursion shape in the visualization data generated from the spatial data, and it is possible to improve the visibility.

When the marker is recognized by the marker recognition unit 22, the spatial data generation unit 13 sets a limit range for the generated spatial data. By setting the limit range for the spatial data, the limit range can be visualized as a part of the visualization data. Further, the spatial data generation unit 13 sets the state information for the spatial data when the state information is acquired by the state information acquisition unit 24. By setting the state information for the spatial data, the state information can be a part of the visualization data.

The visualization data generation unit 14 generates visualization data that visualizes spatial elements included in the space so that a person can intuitively discriminate them, based on the spatial data generated by the spatial data generation unit 13.

Generally, a robot has various sensors such as a camera and a microphone, and recognizes the surrounding situation by comprehensively judging the information obtained from these sensors. In order for the robot to move, it is necessary to recognize various objects existing in the space and determine the movement route from the spatial data, but the movement route may not be appropriate because the object cannot be recognized correctly. Due to misrecognition, for example, even if a person thinks that there is a sufficiently large space, the robot may recognize that there is an obstacle and can move only in a narrow range. When there is a cognitive discrepancy between a person and a robot in this way, the robot acts contrary to the person's expectations, and the person feels stress. In order to reduce the discrepancy between the recognition of the human and the robot, the self-sustaining action robot in the present embodiment visualizes the spatial data of its own recognition state and provides it to the human, and again for the part pointed out by the human. Recognition processing can be performed.

Spatial data is data including spatial elements recognized by the autonomous action robot 1, whereas visualization data is for the user to visually recognize the spatial elements recognized by the autonomous action robot 1. It is data. Spatial data may include misrecognized spatial elements. By visualizing the spatial data, it becomes easier for a person to confirm the recognition state (presence or absence of erroneous recognition, etc.) of the spatial element in the autonomous action type robot 1.

The visualization data is data that can be displayed on the display device. The visualization data is a so-called floor plan, and the space element recognized as a table, chair, sofa, etc. is included in the area surrounded by the space element recognized as a wall. The visualization data generation unit 14 generates the shape of furniture or the like formed in the figure extracted by the Hough transform as visualization data represented by, for example, RGB data. The spatial data generation unit 13 generates visualization data in which the drawing method of the plane is changed based on the direction of the plane in the three dimensions of the spatial element. The direction of the plane in the three dimensions of the spatial element is, for example, the normal direction of the plane formed by the figure generated in the point cloud data by Hough transforming the point cloud data generated in the point cloud data generation unit 12. Is. The visualization data generation unit 14 generates visualization data in which the drawing method of the plane is changed according to the normal direction. The drawing method is, for example, a hue to be given to a plane, a color attribute such as lightness or saturation, a pattern to be given to a plane, a texture, or the like. For example, when the normal of the plane is the vertical direction (the plane is the horizontal direction), the visualization data generation unit 14 increases the brightness of the plane and draws it in a bright color. On the other hand, when the normal of the plane is the horizontal direction (the plane is the vertical direction), the visualization data generation unit 14 lowers the brightness of the plane and draws it in a dark color. By changing the drawing method of the plane, the shape of the furniture or the like can be expressed three-dimensionally, and the user can easily confirm the shape of the furniture or the like. Further, the visualization data may include coordinate information (referred to as "visualization coordinate information") in the visualization data associated with the coordinate information of each spatial element included in the spatial data. Since the visualized coordinate information is associated with the coordinate information, the points in the visualized coordinate information correspond to the points in the actual space, and the faces in the visualized coordinate information correspond to the surfaces in the actual space. Therefore, when the user specifies the position of a certain point in the visualization data, the position of the point in the actual room corresponding to the point can be specified. Further, a conversion function for converting the coordinate system may be prepared so that the coordinate system in the visualization data and the coordinate system in the spatial data can be mutually converted. Of course, the coordinate system in the visualization data and the coordinate system in the actual space may be mutually convertible.

The visualization data generation unit 14 generates visualization data as three-dimensional (Dimensions) data. Further, the visualization data generation unit 14 may generate visualization data as planar (2D) data. By generating the visualization data in 3D, it is possible for the user to easily confirm the shape of the furniture or the like. The visualization data generation unit 14 may generate the visualization data in 3D when sufficient data for generating the visualization data in 3D is generated in the spatial data generation unit 13. The visualization data generation unit 14 may generate visualization data in 3D according to a 3D viewpoint position (viewpoint height, viewpoint elevation / depression angle, etc.) designated by the user. By making it possible to specify the viewpoint position, it is possible for the user to easily confirm the shape of the furniture or the like. Further, the visualization data generation unit 14 may generate visualization data in which the wall or ceiling of the room is colored only on the back wall and the front wall or ceiling is transparent (not colored). By making the wall on the front side transparent, it is possible for the user to easily check the shape of furniture or the like arranged at the tip (indoor) of the wall on the front side.

The visualization data generation unit 14 generates visualization data to which color attributes are added according to the captured image acquired by the captured image acquisition unit 111. For example, when the captured image contains wood grain furniture and the visualization data generation unit 14 detects the color of the wood grain (for example, brown), the visualization data generation unit 14 imparts a color close to the detected color to the extracted furniture figure. Generate visualization data. By assigning a color attribute according to the captured image, it is possible for the user to easily confirm the type of furniture or the like.

The visualization data generation unit 14 generates visualization data in which the drawing method of a fixed fixed object and a moving moving object is changed. The fixed object is, for example, a wall of a room, a step, fixed furniture, or the like. The moving object is, for example, a chair, a trash can, furniture with casters, and the like. Further, the moving object may include, for example, a temporary object temporarily placed on the floor such as a baggage or a bag. The drawing method is, for example, a hue to be given to a plane, a color attribute such as lightness or saturation, a pattern to be given to a plane, a texture, or the like.

The distinction between fixed, moving or temporary objects can be identified by how long they have been in the area. For example, the spatial data generation unit 13 identifies the classification of a fixed object, a moving object, or a temporary object by the spatial element based on the time-dependent change of the point cloud data generated by the point cloud data generation unit 12, and generates the spatial data. Generate. The spatial data generation unit 13 determines that the spatial element is a fixed object when the spatial element does not change from the difference between the spatial data generated at the first time and the spatial data generated at the second time, for example. to decide. Further, the spatial data generation unit 13 may determine from the difference in the spatial data that the spatial element is a moving object when the position of the spatial element is changed. Further, the spatial data generation unit 13 may determine from the difference of the spatial data that the spatial element is a primary object when the spatial element disappears or appears. The visualization data generation unit 14 changes the drawing method based on the division identified by the spatial data generation unit 13. The change of the drawing method is, for example, color coding, addition of hatching, addition of a predetermined mark, or the like. For example, the spatial data generation unit 13 may display a fixed object in black, a moving object in blue, or a temporary object in yellow. The spatial data generation unit 13 identifies the classification of a fixed object, a moving object, or a temporary object and generates spatial data. The visualization data generation unit 14 may generate visualization data in which the drawing method is changed based on the division identified by the spatial data generation unit 13. Further, the spatial data generation unit 13 may generate visualization data in which the drawing method of the spatial element recognized by the image recognition is changed.

The visualization data generation unit 14 can generate visualization data in a plurality of divided areas. For example, the visualization data generation unit 14 generates visualization data for each room partitioned by walls such as a living room, a bedroom, a dining room, and a corridor. By generating visualization data for each room, for example, it becomes possible to generate spatial data or visualization data separately for each room, and it becomes easy to generate spatial data and the like. Further, it is possible to create spatial data and the like only for the area where the robot 2 may move. The visualization data providing unit 161 provides visualization data in which the user can select an area. The visualization data providing unit 161 may expand the visualization data of the area selected by the user, or provide detailed visualization data of the area selected by the user, for example.

The shooting target recognition unit 15 recognizes spatial elements based on the shot image acquired by the shooting image acquisition unit. Spatial element recognition can be performed, for example, by using an image recognition engine that determines what the spatial element is based on the image recognition results accumulated in machine learning. Image recognition of a spatial element can be recognized, for example, in the shape, color, pattern of the spatial element, characters or figures attached to the spatial element, and the like. The shooting target recognition unit 15 may enable image recognition of spatial elements by using, for example, an image recognition service provided in a cloud server (not shown). The visualization data generation unit 14 generates visualization data in which the drawing method is changed according to the spatial element recognized by the imaging target recognition unit 15. For example, when the image-recognized spatial element is a sofa, the visualization data generation unit 14 generates visualization data in which a texture having the texture of cloth is added to the spatial element. Further, when the image-recognized spatial element is a wall, the visualization data generation unit 14 may generate visualization data to which a wallpaper color attribute (for example, white) is added. By performing such a visualization process, the user can intuitively grasp the recognition state of the space in the robot 2.

The second communication control unit 16 controls communication with the user terminal 3 owned by the user. The user terminal 3 is, for example, a smartphone, a tablet PC, a notebook PC, a desktop PC, or the like. The communication method with the user terminal 3 is arbitrary, and for example, wireless LAN, Bluetooth (registered trademark), short-range wireless communication such as infrared communication, or wired communication can be used. Each function of the visualization data providing unit 161 and the designation acquisition unit 162 of the second communication control unit 16 communicates with the user terminal 3 by using the communication function controlled by the second communication control unit 16.

The visualization data providing unit 161 provides the visualization data generated by the visualization data generation unit 14 to the user terminal 3. The visualization data providing unit 161 is, for example, a Web server, and provides visualization data as a Web page to the browser of the user terminal 3. The visualization data providing unit 161 may provide the visualization data to a plurality of user terminals 3. By visually recognizing the visualization data displayed on the user terminal 3, the user can confirm the range in which the robot 2 can move as a 2D or 3D display. In the visualization data, the shape of furniture or the like is drawn by a predetermined drawing method. By operating the user terminal 3, the user can, for example, switch between 2D display and 3D display, zoom in or out of visualization data, or move the viewpoint in 3D display.

The user can visually check the visualization data displayed on the user terminal 3 and confirm the generation state of the spatial data and the attribute information of the area. The user can specify an area in which spatial data is not generated from the visualization data and instruct the creation of spatial data. In addition, the user visually recognizes the visualization data displayed on the user terminal 3, and if there is an area where the spatial data seems to be inaccurate, such as the shape of the spatial element such as furniture is unnatural, the area is considered to be inaccurate. A region can be specified to instruct the regeneration of spatial data. As described above, since the visualization coordinate information in the visualization data is associated with the coordinate information in the spatial data, the area in the visualization data designated to be regenerated by the user can be uniquely specified as the area in the spatial data. .. It is associated with the coordinate information of. The regenerated spatial data is provided by the visualization data providing unit 161 after the visualization data is regenerated in the visualization data generation unit 14. In addition, even in the regenerated visualization data, the generation state of the spatial data may not change, for example, the spatial element is erroneously recognized. In that case, the user may instruct the generation of spatial data by changing the operation parameters of the robot 2. The operation parameters include, for example, shooting conditions (exposure amount or shutter speed, etc.) in the shooting unit 21 of the robot 2, sensitivity of a sensor (not shown), clearance conditions when allowing the robot 2 to move, and the like. The operation parameters may be included in the spatial data as, for example, area attribute information.

The visualization data generation unit 14 generates visualization data including the display of a button instructing the creation (including "re-creation") of spatial data, for example. The user terminal 3 can transmit an instruction to create spatial data to the autonomous action type robot 1 by the user operating the displayed button. The spatial data creation instruction transmitted from the user terminal 3 is acquired by the designated acquisition unit 162.

The designation acquisition unit 162 acquires an instruction to create spatial data in the area designated by the user based on the visualization data provided by the visualization data providing unit 161. The designated acquisition unit 162 may acquire an instruction to set (including change) the attribute information of the area. Further, the designation acquisition unit 162 acquires the position of the area and the direction when the robot approaches the area, that is, the direction to be photographed. The acquisition of the creation instruction can be executed, for example, in the operation of the Web page provided by the visualization data providing unit 161. As a result, the user can grasp how the robot 2 recognizes the space and instruct the robot 2 to redo the recognition process according to the recognition state.

The instruction unit 113 instructs the robot 2 to take a picture in the area in which the spatial data is instructed to be created. The instruction unit 113 may instruct the imaging of the marker installed in the area. The shooting in the area instructed to create the spatial data may include, for example, shooting conditions such as the coordinate position of the robot 2 (shooting unit 21), the shooting direction of the shooting unit 21, and the resolution. When the spatial data instructed to be created relates to an ungenerated area, the spatial data generation unit 13 adds the newly created spatial data to the existing spatial data, and the spatial data generation unit 13 is created. If the instructed spatial data is related to re-creation, generate spatial data that is an update of the existing spatial data. Further, when the captured image contains a marker, spatial data including the recognized marker may be generated.

As described above, in FIG. 1, the case where the autonomous action type robot 1 is composed of the data providing device 10 and the robot 2 has been described, but the function of the data providing device 10 is included in the function of the robot 2. It may be a thing. For example, the robot 2 may include all the functions of the data providing device 10. The data providing device 10 may temporarily replace the function when the processing capacity of the robot 2 is insufficient, for example.

Further, in the present embodiment, the “acquisition” may be one in which the acquiring entity actively acquires, or one in which the acquiring entity passively acquires. For example, the designated acquisition unit 162 may be acquired by receiving an instruction for creating spatial data transmitted by the user from the user terminal 3, and may be stored in a storage area (not shown) not shown by the user. It may be acquired by reading the instruction for creating the created spatial data from the storage area.

Further, the first communication control unit 11, the point group data generation unit 12, the spatial data generation unit 13, the visualization data generation unit 14, the image capture target recognition unit 15, the second communication control unit 16, and the captured image included in the data providing device 10. Each functional unit of the acquisition unit 111, the spatial data provision unit 112, the instruction unit 113, the visualization data provision unit 161 and the designated acquisition unit 162 shows an example of the function of the autonomous action type robot 1 in the present embodiment. The functions of the autonomous action type robot 1 are not limited. For example, the autonomous action type robot 1 does not have to have all the functional parts of the data providing device 10, but may have some of the functional parts. Further, the autonomous action type robot 1 may have a functional unit other than the above. Further, each functional unit of the marker recognition unit 22, the movement control unit 23, the restriction range setting unit 231 and the state information acquisition unit 24 of the robot 2 shows an example of the functions of the autonomous action type robot 1 in the present embodiment. This does not limit the functions of the autonomous action type robot 1. For example, the autonomous action type robot 1 does not have to have all the functional parts of the robot 2, and may have some of the functional parts.

Further, as described above, each of the above-mentioned functional parts of the autonomous action type robot 1 has been described as being realized by software. However, at least one or more of the above-mentioned functions of the autonomous action type robot 1 may be realized by hardware.

Further, any of the above-mentioned functions possessed by the autonomous action type robot 1 may be executed by dividing one function into a plurality of functions. Further, any two or more of the above-mentioned functions possessed by the autonomous action type robot 1 may be integrated into one function. That is, FIG. 1 shows the functions of the autonomous action robot 1 as functional blocks, and does not show, for example, that each function is composed of a separate program file.

Further, the autonomous action type robot 1 may be a device realized by one housing or a system realized by a plurality of devices connected via a network or the like. For example, the autonomous behavior type robot 1 may realize a part or all of its functions by a virtual device such as a cloud service provided by a cloud computing system. That is, the autonomous action type robot 1 may realize at least one or more of the above-mentioned functions in another device. Further, the autonomous action type robot 1 may be a general-purpose computer such as a tablet PC, or may be a dedicated device having limited functions.

Further, the autonomous action type robot 1 may realize a part or all of its functions in the robot 2 or the user terminal 3.

Next, the hardware configuration of the autonomous action type robot 1 (control unit of the robot 2) will be described with reference to FIG. FIG. 2 is a block diagram showing an example of the hardware configuration of the autonomous action type robot 1 in the embodiment.

The autonomous action type robot 1 has a CPU (Central Processing Unit) 101, a RAM (Random Access Memory) 102, a ROM (Read Only Memory) 103, a touch panel 104, a communication I / F (Interface) 105, a sensor 106, and a clock 107. .. The autonomous action type robot 1 is a device that executes the autonomous action type robot control program described with reference to FIG.

The CPU 101 controls the autonomous action robot 1 by executing the autonomous action robot control program stored in the RAM 102 or the ROM 103. The autonomous action type robot control program is acquired from, for example, a recording medium on which the autonomous action type robot control program is recorded, a program distribution server via a network, or the like, installed in the ROM 103, read from the CPU 101, and executed.

The touch panel 104 has an operation input function and a display function (operation display function). The touch panel 104 enables the user of the autonomous action robot 1 to input operations using a fingertip, a touch pen, or the like. The case where the autonomous action type robot 1 in the present embodiment uses the touch panel 104 having the operation display function will be described, but the autonomous action type robot 1 separates the display device having the display function and the operation input device having the operation input function. It may have. In that case, the display screen of the touch panel 104 can be performed as the display screen of the display device, and the operation of the touch panel 104 can be performed as the operation of the operation input device. The touch panel 104 may be realized in various forms such as a head mount type, a glasses type, and a wristwatch type display.

The communication I / F 105 is an I / F for communication. The communication I / F 105 executes short-range wireless communication such as wireless LAN, wired LAN, and infrared rays. Although only the communication I / F 105 is shown as the communication I / F in FIG. 2, the autonomous action type robot 1 may have each communication I / F in a plurality of communication methods. The communication I / F 105 may communicate with a control unit that controls the photographing unit 21 (not shown) or a control unit that controls the moving mechanism 29.

The sensor 106 is hardware such as a camera of the photographing unit 21, a TOF or a thermo camera, and hardware such as a microphone, a thermometer, a luminometer, or a proximity sensor. The data acquired by these hardware is stored in the RAM 102 and processed by the CPU 101.

The clock 107 is an internal clock for acquiring time information. The time information acquired by the clock 107 is used, for example, to confirm the time zone in which entry is prohibited.

Next, the operation related to the provision of visualization data of the robot control program will be described with reference to FIG. FIG. 3 is a flowchart showing an example of the operation of the robot control program in the embodiment. In the following description of the flowchart, it is assumed that the execution subject of the operation is the autonomous action type robot 1, but each operation is executed in each function of the above-mentioned autonomous action type robot 1.

In FIG. 3, the autonomous action type robot 1 determines whether or not the captured image has been acquired (step S11). The determination of whether or not the captured image has been acquired can be determined by whether or not the captured image acquisition unit 111 has acquired the captured image from the robot 2. The determination as to whether or not the captured image has been acquired is determined in the processing unit of the captured image. For example, when the captured image is a moving image, the moving image is continuously transmitted from the robot 2, so that the number of frames or the amount of data of the acquired moving image is a predetermined value for determining whether or not the captured image is acquired. It can be done depending on whether or not it has reached. The captured image may be acquired mainly by the mobile robot to transmit the captured image, or by the captured image acquisition unit 111 to take the captured image from the mobile robot. When it is determined that the captured image has not been acquired (step S11: NO), the autonomous action type robot 1 repeats the process of step S11 and waits for the captured image to be acquired.

On the other hand, when it is determined that the captured image has been acquired (step S12: NO), the autonomous action type robot 1 generates point cloud data (step S12). The point cloud data is generated by, for example, the point cloud data generation unit 12 detecting a point in the captured image in which the change in brightness is large as a feature point and giving three-dimensional coordinates to the detected feature point. Can be executed. The feature points may be detected, for example, by performing differential processing on the captured image to detect a portion having a large gradation change. Further, the addition of coordinates to the feature points may be executed by detecting the same feature points shot from different shooting angles. The determination of whether or not the captured image is acquired in step S11 can be determined by whether or not the captured image captured from a plurality of directions is acquired.

After executing the process of step S12, the autonomous action type robot 1 generates spatial data (step S13). Spatial data generation can be executed by the spatial data generation unit 13 by, for example, Hough transforming the point cloud data. The details of step S13 will be described with reference to FIG.

After executing the process of step S13, the autonomous action type robot 1 provides the generated spatial data to the robot 2 (step S14). As shown in FIG. 3, the spatial data may be provided to the robot 2 sequentially each time the spatial data is generated, or may be provided asynchronously with the processes shown in steps S11 to S18. Good. The robot 2 provided with the spatial data can grasp the movable range based on the spatial data.

After executing the process of step S14, the autonomous action type robot 1 determines whether or not to recognize the spatial element (step S15). The determination of whether or not to recognize the spatial element can be executed, for example, by setting whether or not to recognize the spatial element in the photographing target recognition unit 15. Even if it is determined that the spatial element is recognized, if the recognition fails, it may be determined that the spatial element is not recognized.

When it is determined that the spatial element is recognized (step S15: YES), the autonomous action type robot 1 generates the first visualization data (step S16). The generation of the first visualization data can be executed in the visualization data generation unit 14. The first visualization data is visualization data generated after the imaging target recognition unit 15 recognizes the spatial element. For example, when the imaging target recognition unit 15 determines that the spatial element is a table, the visualization data generation unit 14 can display the table even if the upper surface of the table is not photographed and does not have the point cloud data. Visualization data can be generated assuming that the top surface is flat. Further, when it is determined that the spatial element is a wall, the visualization data generation unit 14 can generate visualization data assuming that the unphotographed portion is also a plane.

When it is determined that the spatial element is not recognized (step S15: NO), the autonomous action type robot 1 generates the second visualization data (step S17). The generation of the second visualization data can be executed in the visualization data generation unit 14. The second visualization data is visualization data that the photographing target recognition unit 15 does not recognize the spatial element, that is, is generated based on the point cloud data and the spatial data generated from the captured image. The autonomous action type robot 1 can reduce the processing load by not performing the spatial element recognition processing.

After executing the process of step S16 or the process of step S17, the autonomous behavior type robot 1 provides visualization data (step S18). The provision of the visualization data is executed by the visualization data providing unit 161 providing the visualization data generated by the visualization data generation unit 14 to the user terminal 3. The autonomous action type robot 1 may generate and provide visualization data in response to a request from, for example, a user terminal 3. After executing the process of step S18, the autonomous action type robot 1 ends the operation shown in the flowchart.

Next, the operation related to the spatial data generation of the robot control program will be described with reference to FIG. FIG. 4 is a flowchart showing another example of the operation of the robot control program in the embodiment.

In FIG. 4, the autonomous behavior type robot 1 generates spatial data (step S121). Spatial data generation can be executed by the spatial data generation unit 13 by, for example, Hough transforming the point cloud data. After executing step S131, the autonomous action type robot 1 determines whether or not the marker has been recognized (step S122). Whether or not the marker is recognized can be determined by whether or not the marker recognition unit 22 recognizes the marker image in the captured image captured by the photographing unit 21. The robot 2 can notify the data providing device 10 of the recognition result of the marker.

When it is determined that the marker is recognized (step S122: YES), the autonomous action type robot 1 sets a restriction range in which movement is restricted to the spatial data generated in step S121 (step S123).

When it is determined that the marker is not recognized (step S122: NO), the autonomous action type robot 1 determines whether or not the state information has been acquired (step S124). Whether or not the state information has been acquired can be determined by whether or not the state information has been acquired by the state information acquisition unit 24. When it is determined that the state information has been acquired (step S124: YES), the autonomous action type robot 1 sets the acquired state information in correspondence with the spatial data (step S125). The set state information is provided by the visualization data providing unit 161 in correspondence with the visualization data.

On the other hand, when it is determined that the state information has not been acquired (step S124: NO), after executing the process of step S123 or after executing the process of step S125, the autonomous action type robot 1 takes the step shown in the flowchart. The operation of generating the provided data in S12 is terminated.

The processing in each step in the operation of the robot control program (robot control method) described in the present embodiment does not limit the execution order.

Next, with reference to FIG. 5, a method of setting an entry prohibition line by installing a pair of markers will be described. FIG. 5 is a diagram showing a method of setting an entry prohibition line in the embodiment.

<Installation of a line prohibiting entry into the passage by a pair of markers>
In FIG. 5, it is assumed that there is a passage (doorway) in which the robot 2 can move between the wall 1 and the wall 2 and between the wall 1 and the wall 3. The user installs the marker 1a as the first marker on the wall 1 side and further installs the marker 1b as the second marker on the wall 2 side between the wall 1 and the wall 2. The marker 1a and the marker 1b are recognized by the marker recognition unit 22 as a pair of markers. The restriction range setting unit 231 sets the line connecting the marker 1a and the marker 1b with a straight line as the entry prohibition line 1. By setting the entry prohibition line 1, the user can restrict the robot 2 from moving beyond the entry prohibition line 1.

<Installation of a line prohibiting entry into the passage by a single marker>
In the passage between the wall 1 and the wall 3, the user installs the marker 2 on the wall 1 side near the passage. The marker 2 is recognized by the marker recognition unit 22 as a single marker. The limit range setting unit 231 confirms whether or not there is a passage near the marker 2. When there is a passage, the restriction range setting unit 231 sets a straight line on the passage as the entry prohibition line 2 from the installation position of the marker 2 and the position of the passage. Since the restriction range setting unit 231 can confirm whether or not there is a passage in the vicinity of the marker 2, the user can prohibit the robot from entering the passage even with a single marker.

<Installation of the first no-entry area with a single marker>
The user attaches the marker 3 to the wall 3. The marker 3 is recognized by the marker recognition unit 22 as a single marker. The limit range setting unit 231 confirms whether or not there is a passage near the marker 3. When there is no passage, the restriction range setting unit 231 sets a predetermined range (for example, a semicircle centered on the installation position of the marker 3) from the installation position of the marker 2 as the first entry prohibition area.

<Installation of a second no-entry area with a single marker>
The user installs the marker 4 in the center of the room. The marker 4 is, for example, a three-dimensional marker. The marker 4 is recognized by the marker recognition unit 22 as a single marker. The restriction range setting unit 231 sets a predetermined range around the marker 4 (for example, a circle centered on the installation position of the marker 4) as a second entry prohibition area.

Next, the setting of the limited range in the user terminal 3 provided by the autonomous action type robot 1 will be described with reference to FIGS. 6 to 7. 6 to 7 are diagrams showing an example of the display of the user terminal 3 in the embodiment. 6 to 7 are display examples in which a Web page provided as visualization data by the visualization data providing unit 161 is displayed on a touch panel of a smartphone illustrated as a user terminal 3.

In FIG. 6, the user terminal 3 displays the 2D display data generated by the visualization data generation unit 14 based on the image taken by the robot 2 in the living room. The visualization data generation unit 14 rasterizes the line segment (straight line or curve) extracted by Hough transforming the point cloud data, and draws the boundary line of the spatial element such as a wall or furniture in 2D. The visualization data generation unit 14 displays the entry prohibition line 36 based on the pair of marker images recognized by the marker recognition unit 22 or the state information acquired by the state information acquisition unit 24. The entry prohibition line 36 can be set by designating the start point and the end point. The start point and end point can be set by installing a pair of markers or by setting from the user terminal 3. A no-entry mark is displayed at the center of the no-entry line 36. When the entry prohibition mark is pressed, the delete button 37 is displayed. By pressing the delete button 37, the entry prohibition line 36 once set can be deleted. The house-shaped icon h shown in the figure represents the home position where the robot 2 returns home for charging.

In FIG. 6, the area on the right side of the entry prohibition line 36 is an area where spatial data has not been created. The user can set, confirm, or delete the entry prohibition line 36 from the visualization data displayed on the user terminal 3. That is, the autonomous action type robot 1 can generate a visualized map in which the range in which the robot 2 can move is defined from the image taken by the photographing unit 21, and the user terminal 3 can enter the no-entry area where the robot 2 cannot enter. Allows you to set from. The user terminal 3 may be able to set conditions for restricting the movement of the robot 2. For example, the user terminal 3 enables the user to set a time zone during which the robot 2 is prohibited from entering, the content of movement restrictions regarding the presence or absence of a person, the lighting state when the movement is restricted, and the like. May be good. For example, the user terminal 3 sets conditions so as to prohibit entry into the kitchen during the morning and evening hours when a person prepares meals, and prohibit entry into the study when the lights are off. You may be able to do it.

In FIG. 7, the user terminal 3 displays the 2D visualization data generated by the visualization data generation unit 14 based on the image taken in the living room taken by the robot 2 as in FIG. Visualization data of the Western-style room is displayed on the right side of the entry prohibition line 36, and it can be confirmed that the robot 2 is prohibited from entering the Western-style room 38 by the entry prohibition line 36. For example, there is a case where it is desired to set the Western-style room 38 to be prohibited from entering after the spatial data of the Western-style room 38 is generated based on the captured image of the Western-style room 38. The user can set the Western-style room 38 as an entry-prohibited area by installing a marker for prohibiting the entry of the pair at the entrance of the Western-style room 38, or by setting the entry-prohibited line 36 from the user terminal 3. It will be possible. Further, since the spatial data has already been generated in the Western-style room 38, the user can confirm that the robot 2 can be moved to the Western-style room 38 by deleting the entry prohibition line 36. The user can enlarge or reduce the display by pinching in or out of the touch panel of the user terminal 3.

As a modification of the first embodiment, for example, the visualization data providing unit 161 of FIG. 1 may provide the image as the source of the visualization data to the user terminal 3. For example, the user may specify a part of the floor plan displayed on the user terminal 3 so that the captured image is displayed. That is, the image used for the identification is accumulated for each spatial element, and when the user specifies the spatial element, the image associated with the spatial element is provided. As a result, when the user cannot determine the recognition state from the visualization data, the user can determine the recognition state using an image.

As yet another modification, as a method of specifying the restricted range of entry prohibition described with reference to FIG. 6, the operation of sliding the fingertip on the touch panel screen of the user terminal 3 in the manner of drawing a circle with the fingertip may be sufficient. , Specify by moving your fingertip so that it surrounds the restricted range. In conjunction with this operation, the locus of the fingertip is drawn on the screen, and the locus of the fingertip is visualized so as to be superimposed on the visualization data.

Further, when the marker is flat, the marker may not be recognized depending on the shooting angle of the marker as described above. For example, in the case of a road sign, it is installed so as to be substantially perpendicular to the traveling direction so that the driver can easily see the traveling of the vehicle. However, a marker may be attached to the wall in the path in which the robot 2 moves, and the marker may be overlooked depending on the shooting direction of the camera. For example, since the marker 2 shown in FIG. 5 is attached to the wall 1, if the robot 2 moves toward the marker 3 along the wall 3, it may enter the passage before checking the marker. .. Therefore, as a modification of the present embodiment, when a space (for example, a passage provided on a wall) that the robot 2 can enter is found, it is positive whether or not a marker is installed near the entrance of the space. Make sure to perform the confirmation operation. For example, if the entrance to the space is a passageway between walls, the marker may be affixed to the wall near the entrance to the passageway. The robot 2 moves the marker that may be installed on the wall at the entrance of the aisle to a position where it is easy to see, for example, the position in front of the aisle, and overlooks the marker by performing an active confirmation operation to photograph the wall. Can be prevented. The marker positive confirmation operation may be performed when an area with different environmental conditions is found or when the marker is invaded into the area. The environmental conditions are, for example, the illuminance, temperature, humidity, noise level, etc. of the space in which the robot 2 moves, and the changes in the environmental conditions may include changes in spatial elements such as the color of the wall.

Further, the restriction range setting unit 231 may set the restriction range based on the feature points in the vicinity where the marker is installed, instead of the spatial data of the installation position where the marker is installed. Peripheral feature points are, for example, spatial elements such as cables and steps arranged on the floor. By learning the feature points where the markers are installed, it is possible to obtain a learning effect of restricting the movement even in a place having other similar feature points where the markers are not installed.

Further, when there are a plurality of robots, the content of the movement restriction may be set differently for each robot. For example, when the same marker is recognized, the no-entry area restricted by the robot may be set differently. By setting different restrictions for each robot, it is possible to set restrictions according to the purpose of the robot (for example, a cleaning robot, an alarm clock, etc.).

Further, the robot may learn the content of the limit once set by the marker. For example, the robot may learn that the no-entry area has been set by the temporary marker, so that the access frequency may be reduced even after the marker is removed. In order to realize this, the marker recognition unit 22 stores the positions of the temporary marker and the permanent marker in association with the information for specifying the marker type. In this way, by memorizing the position of the temporary marker, the robot behaves to hesitate to enter the area where the temporary marker is installed, or reduces the frequency of entry into the area. Can be executed.

[Embodiment 2]
In the first embodiment, an example in which the marker is used to make the autonomous action robot 1 recognize the place where entry is prohibited may be used, but the marker may be used to make the autonomous action robot 1 recognize an arbitrary place. That is, the marker may be installed at an arbitrary place in the house or facility so that the autonomous action robot 1 recognizes the position of the place where the marker is installed.

Residential includes any area, such as a front door, a children's room or a bedroom. Facilities include any area such as reception counters, rest areas and emergency exits. A marker that can identify the area type (for example, the entrance type or reception counter type described later) associated with these areas is used. The marker may be any shape, pattern, color, characters or figures attached to the marker, or a combination thereof, as long as it has a feature that allows the area type to be identified by image recognition. .. For example, a graphic code obtained by graphicizing the area type code by a general-purpose conversion method (bar code method, two-dimensional bar code method, etc.) may be used as a marker. In this case, the marker recognition unit 22 can read the area type code from the graphic code by the conversion method. Alternatively, a uniquely designed figure may be used as a marker. In this case, the marker recognition unit 22 specifies the type of the figure included in the captured image when the shape of the figure is a predetermined shape, and specifies the area type corresponding to the specified type of the figure. You may. It is assumed that the data providing device 10 or the robot 2 stores data for associating a graphic type with an area type. That is, the marker in the second embodiment can specify any area type.

If such a marker is installed in an area identified by the area type, the autonomous action robot 1 can recognize that the installation location of the marker corresponds to the area identified by the area type. For example, if a front door type marker is installed at the entrance, the autonomous action robot 1 can recognize that the place where the front door type marker is installed is the front door.

In the second embodiment, the autonomous action type robot 1 stores marker information for associating a predetermined event with an area type. When the autonomous action type robot 1 detects a predetermined event, the robot 2 moves to a place (referred to as a marker installation place) where a marker of the area type corresponding to the predetermined event is installed. A predetermined event that triggers the robot 2 to move to the marker installation location is called a first event. The first event may be detected by the robot 2 or may be detected by the data providing device 10.

Further, when the autonomous action type robot 1 detects another predetermined event after the robot 2 moves to the marker installation location, the robot 2 executes a predetermined action. The event that triggers the robot 2 to execute a predetermined action is called a second event. The second event may be detected by the robot 2 or may be detected by the data providing device 10. The action to be executed corresponds to at least one of a first event, an area type and a second event. In the second embodiment, in the autonomous action type robot 1, event information for associating at least one of the first event, the area type, and the second event with the action is stored. The event information may be stored in the robot 2.

FIG. 8 is a block diagram showing an example of the module configuration of the robot 2 in the second embodiment. FIG. 8 also shows the functional parts related to Examples 1 to 6 described later. The robot 2 has each functional unit of a marker recognition unit 22, a position measurement unit 25, a movement control unit 23, a communication control unit 26, a first event detection unit 210, a second event detection unit 220, and an action execution unit 230. Each of the above-mentioned functional units of the robot 2 in the second embodiment will be described as being a functional module realized by a program that controls the robot 2.

The marker recognition unit 22 recognizes the marker included in the captured image and specifies the area type indicated by the marker. The position measuring unit 25 measures the current position and direction of the robot 2. The position measuring unit 25 may measure the current position and direction based on the captured image, or may measure the current position based on the radio wave received from the wireless communication device installed at the predetermined position. The method for measuring the current position may be a conventional technique. The position measuring unit 25 may use SLAM (Simultaneous Localization and Mapping) technology that simultaneously estimates the self-position and creates an environmental map. The movement control unit 23 controls the movement of the robot 2. The movement control unit 23 sets a route to the destination, drives the movement mechanism 29, follows the route, and moves itself to the destination. The communication control unit 26 communicates with the data providing device 10.

The first event detection unit 210 detects the first event. The first event detection unit 210 may detect the first event based on the result of recognition processing such as voice recognition and image recognition. That is, the first event detection unit 210 may detect the first event when it is determined that the sound input by the microphone included in the robot 2 includes a characteristic that is presumed to correspond to a predetermined voice. For example, the first event detection unit 210 records a predetermined voice as a sample in advance, analyzes the voice, and extracts characteristic data such as frequency distribution, intonation, and period in which the volume increases. Then, the first event detection unit 210 performs the same analysis on the sound input by the microphone, and when the characteristic data such as the frequency distribution, the intonation and the period in which the volume increases match or are similar to the case of the sample, the first event detection unit 210 performs the same analysis. It may be presumed that the sound input by the microphone corresponds to the predetermined voice.

Further, the first event detection unit 210 may detect the first event when it is presumed that an arbitrary person, a predetermined person, or a predetermined object is captured in the image captured by the photographing unit 21. The first event detection unit 210, for example, photographs an arbitrary person, a predetermined person, or a predetermined object as a sample in advance with the photographing unit 21, analyzes the photographed image, and arranges the size, shape, and parts of the subject. The feature data of is extracted. Then, when the first event detection unit 210 analyzes the image captured by the photographing unit 21 and determines that the sample contains a subject having common or similar feature data such as size, shape, and arrangement of parts. , It may be presumed that an arbitrary person, a predetermined person, or a predetermined object is shown in the captured image.

The first event detection unit 210 may detect the first event based on the measurement results of various sensors such as a temperature sensor, a contact sensor, and an acceleration sensor. The first event detection unit 210 determines that the measured value of the sensor falls below a predetermined lower limit value, the measured value of the sensor falls within a predetermined range, the measured value of the sensor deviates from the predetermined range, or the sensor The first event may be detected when the measured value of the above exceeds a predetermined upper limit value. The first event detection unit 210 may detect the first event when, for example, a temperature sensor measures a temperature corresponding to a person's body temperature. The first event detection unit 210 may detect the first event when, for example, a contact sensor measures a contact corresponding to a human touch. The first event detection unit 210 may detect the first event when the change in acceleration corresponding to a large shaking such as an impact of a traffic accident or an earthquake is measured by an acceleration sensor.

The first event detection unit 210 may detect the first event based on the communication state in the communication process. The first event detection unit 210 indicates a communication state when, for example, a characteristic value indicating a communication state such as a radio wave strength in wireless communication, a communication time with a predetermined communication partner, or a transmission amount per a predetermined time falls below a lower limit value. The first event is detected when the characteristic value falls within a predetermined range, when the characteristic value indicating the communication state deviates from the predetermined range, or when the characteristic value indicating the communication state exceeds the predetermined upper limit value. You may.

The first event detection unit 210 may detect the first event based on the data received from the data providing device 10, the user terminal 3, or another external device. The first event detection unit 210 may detect the first event when, for example, a predetermined notification is received from the data providing device 10 or another external device. The first event detection unit 210 may detect the first event when a predetermined request by the user is received from the user terminal 3.

The voice recognition unit 211, the radio wave state detection unit 213, the patrol event detection unit 215, and the wake-up event detection unit 217 are examples of the first event detection unit 210. The voice recognition unit 211 will be described in the second embodiment (application example of babysitter). The radio wave state detection unit 213 will be described in the fourth embodiment (application example of call support). The patrol event detection unit 215 will be described in the fifth embodiment (application example of security). The wake-up event detection unit 217 will be described in Example 6 (application example of the alarm clock).

The second event detection unit 220 detects the second event. The second event detection unit 220 may detect the second event based on the result of recognition processing such as voice recognition and image recognition, as in the case of the first event detection unit 210. The second event detection unit 220 may detect the second event based on the measurement results of various sensors such as a temperature sensor, a contact sensor, and an acceleration sensor, as in the case of the first event detection unit 210. The second event detection unit 220 may detect the second event based on the communication state in the communication processing, as in the case of the first event detection unit 210. Similar to the case of the first event detection unit 210, the second event detection unit 220 may detect the second event based on the data received from the data providing device 10, the user terminal 3, or another external device. Good.

The user recognition unit 221, the call request reception unit 223, the person recognition unit 225, and the body position recognition unit 227 are examples of the second event detection unit 220. The user recognition unit 221 will be described in the first embodiment (application example of welcoming). The call request receiving unit 223 will be described in the fourth embodiment (application example of the call support). The human recognition unit 225 will be described in the fifth embodiment (application example of security). The body position recognition unit 227 will be described in Example 6 (application example of the alarm clock).

The action execution unit 230 executes an action triggered by the second event. The action execution unit 230 may execute an action that involves the movement of the robot 2 itself. The action execution unit 230 may execute an action involving input processing such as image, voice, or communication in the robot 2. The action execution unit 230 may execute an action that involves output processing such as image, voice, or communication in the robot 2.

The attitude control unit 231, the voice output unit 232, the remote control unit 233, the message output unit 235, and the telephone communication unit 237 are examples of the action execution unit 230. Further, the movement control unit 23 may function as the action execution unit 230. When the movement control unit 23 functions as the action execution unit 230, the movement control unit 23 controls the movement mechanism 29 to perform an action related to the movement of the robot 2. The attitude control unit 231 performs an action related to the posture of the robot 2. If the robot 2 has a shape imitating a person or a virtual character and the neck and arms can be moved by an actuator, the attitude control unit 231 drives the actuator to cause the robot 2 to take various poses. Or you may have them perform various gestures. If the robot 2 has a shape that imitates a four-legged animal and each leg can be moved by an actuator provided at a joint portion, the attitude control unit 231 drives the actuator to make various robots 2. You may make them pose or make various gestures.

The remote control unit 233 will be described in Example 2 (application example of babysitter) and Example 6 (application example of alarm clock). The message output unit 235 will be described in the third embodiment (application example of customer service). The telephone communication unit 237 will be described in the fourth embodiment (application example of call support).

FIG. 9 is a block diagram showing an example of the module configuration of the data providing device 10 according to the second embodiment. FIG. 9 also shows the functional parts related to Examples 1 to 6 described later. The robot 2 has each functional unit of a first communication control unit 11, a second communication control unit 16, a marker registration unit 110, a first event detection unit 120, a second event detection unit 130, and an action selection unit 140. Each of the above-mentioned functional units of the data providing device 10 in the second embodiment will be described as being a functional module realized by a program that controls the data providing device 10.

The first communication control unit 11 controls wireless communication with the robot 2. The second communication control unit 16 controls wireless communication or wired communication with the user terminal 3 or another external device. The marker registration unit 110 registers marker information including the position and orientation of the marker in the marker information storage unit 153, which will be described later.

The first event detection unit 120 detects the first event. The first event detection unit 120 may detect the first event based on the result of recognition processing such as voice recognition and image recognition, as in the case of the first event detection unit 210 of the robot 2. The first event detection unit 120 detects the first event based on the measurement results of various sensors such as the temperature sensor, contact sensor, and acceleration sensor of the robot 2, as in the case of the first event detection unit 210 of the robot 2. You may. The first event detection unit 120 may detect the first event based on the communication state in the communication processing, as in the case of the first event detection unit 210 of the robot 2. The first event detection unit 120 may detect the first event based on the data received from the user terminal 3 or another external device, as in the case of the first event detection unit 210 of the robot 2. The first event detection unit 120 may detect the first event based on the data received from the robot 2.

The homecoming event detection unit 121 and the visitor event detection unit 123 are examples of the first event detection unit 120. The homecoming event detection unit 121 will be described in the first embodiment (application example of welcoming). The visitor event detection unit 123 will be described in the third embodiment (application example of customer service).

The second event detection unit 130 detects the second event. The second event detection unit 130 may detect the second event based on the result of recognition processing such as voice recognition and image recognition, as in the case of the first event detection unit 210 of the robot 2. The second event detection unit 130 detects the second event based on the measurement results of various sensors such as the temperature sensor, contact sensor, and acceleration sensor of the robot 2, as in the case of the first event detection unit 210 of the robot 2. You may. The second event detection unit 130 may detect the second event based on the communication state in the communication processing, as in the case of the first event detection unit 210 of the robot 2. The second event detection unit 130 may detect the second event based on the data received from the user terminal 3 or another external device, as in the case of the first event detection unit 210 of the robot 2. The second event detection unit 130 may detect the second event based on the data received from the robot 2.

The vacant room event detection unit 131 is an example of the second event detection unit 130. The vacant room event detection unit 131 will be described in the third embodiment (application example of customer service). The action selection unit 140 selects an action corresponding to at least one of the first event, the area type, and the second event. The following is an example of mainly selecting an action corresponding to a combination of the first event and the second event.

The robot 2 further has an event information storage unit 151 and a marker information storage unit 153. The event information storage unit 151 stores event information including the area type, the second event, and the action corresponding to the first event. The event information storage unit 151 will be described later in relation to FIG. The marker information storage unit 153 stores marker information that associates the area type with the position and orientation of the marker. The marker information storage unit 153 will be described later in relation to FIG.

FIG. 10 is a block diagram showing an example of the data structure of the event information storage unit 151 according to the second embodiment. Each record in FIG. 10 defines that when the first event is detected, the robot 2 moves to a place where a marker of the area type corresponding to the first event is installed. Further, each record of FIG. 10 defines that the robot 2 executes an action corresponding to, for example, a combination of the first event and the second event when the second event is detected. Details of each record will be described in Examples 1 to 6. The event information may be set as a default, or may be set by the user in the application of the user terminal 3.

FIG. 11 is a block diagram showing an example of the data structure of the marker information storage unit 153 in the second embodiment. In each record of FIG. 11, the position and orientation of the marker of the area type are set in association with the area type.

In advance, the user attaches an area type marker that identifies an area near the position to an arbitrary position of the house or facility. Then, in the marker registration phase, when the robot 1 detects an unknown marker, the data providing device 10 registers the marker information.

FIG. 12A is a flowchart showing a processing procedure in the marker registration phase of the second embodiment. For example, an unknown marker may be detected when the robot 2 is moving autonomously (S21). Specifically, the marker recognition unit 22 of the robot 2 recognizes a marker included in the image captured by the photographing unit 21 and specifies the area type indicated by the marker. The marker recognition unit 22 stores markers that have been detected in the past, and can determine undetected markers by comparing the stored markers with the detected markers.

When the marker recognition unit 22 determines that the undetected marker has been recognized, the marker recognition unit 22 specifies the relative positional relationship and direction between the robot 2 and the marker by image recognition. The marker recognition unit 22 obtains the distance between the robot 2 and the marker according to the size of the marker included in the captured image. Further, the marker recognition unit 22 determines the direction of the marker with respect to the robot 2 according to how the marker included in the captured image is distorted. Then, the marker recognition unit 22 obtains the position and direction of the marker with reference to the current position and direction of the robot 2 measured by the position measurement unit 25. The communication control unit 26 transmits marker information including the area type and the position and orientation of the marker of the area type to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the marker information, the marker registration unit 110 registers the position and orientation of the marker of the area type in the marker information storage unit 153 in association with the area type (step). S22).

When the marker registration phase is completed, in the action phase, the robot 2 moves to the marker installation location triggered by the first event, and the robot 2 performs a predetermined action triggered by the second event.

FIG. 12B is a flowchart showing a processing procedure in the action phase of the second embodiment. When the first event detection unit 210 of the robot 2 or the first event detection unit 120 of the data providing device 10 detects the first event (S23), the action selection unit 140 of the data providing device 10 marks the mark corresponding to the first event. The type is specified, and the robot 2 is instructed to move to the position of the mark of the mark type. The movement control unit 23 of the robot 2 controls the movement mechanism 29 by setting a route to the mark position according to the movement instruction. By the operation of the moving mechanism 29, the robot 2 moves to the vicinity of the mark (S24).

When the second event detection unit 220 of the robot 2 or the second event detection unit 130 of the data providing device 10 detects the second event, the action selection unit 140 of the data providing device 10 may, for example, combine the first event and the second event. The action corresponding to is selected, and the selected action is instructed to the robot 2. Then, the robot 2 executes the instructed action (S26). Hereinafter, Examples 1 to 6 relating to the second embodiment will be described.

[Example 1]
An application example of welcoming by the autonomous action type robot 1 will be described. In the application example of welcoming, when the user returns home, the robot 2 heads to the entrance to greet the user. If an area type marker that identifies the entrance as an area is installed at the entrance, the autonomous action robot 1 can recognize that the installation location of the marker is the entrance. The area type that identifies the entrance is called the entrance type.

The timing at which the user returns home can be determined by the position measured by the GPS device (Global Positioning System) of the user terminal 3 held by the user approaching the home. For example, when the distance between the position of the user terminal 3 and the entrance (or the data providing device 10) is shorter than the reference length, it is determined that it is the timing when the user returns home. An event for determining when a user returns home is called a user return event. That is, the user return home event corresponds to the first event in the application example of welcoming.

When the robot 2 arrives at the entrance and recognizes the user by the image taken by the photographing unit 21, the robot 2 takes an action in response to the user returning home. An event in which the robot 2 recognizes a user is called a user recognition event. In addition, the action that reacts to the user's return home is called the welcome action. The user recognition event is an opportunity to perform a welcome action. The user recognition event corresponds to the second event in the application example of welcoming.

In the welcoming action, for example, the voice output unit 232 outputs voice from the speaker included in the robot 2. The output voice may be a natural language such as "Welcome back" or a non-language such as cheers. The movement control unit 23 may control the movement mechanism 29 as a welcoming action to perform an action of moving the robot 2 back and forth or rotating the robot 2 in small steps. If the robot 2 has a shape imitating a person or a virtual character and the arm can be moved by the actuator, the attitude control unit 231 may drive the actuator to raise or lower the arm. If the robot 2 can move its head with an actuator, the attitude control unit 231 may drive the actuator to shake its head. If the robot 2 has a shape that imitates a four-legged animal and can move each leg with an actuator provided at the joint part, the attitude control unit 231 poses to stand up with only the hind legs as a welcoming action. You may let me take. As a result, it is possible to produce that the robot 2 is pleased with the return of the user. The user will have a sense of familiarity with the robot 2 that is waiting for him and will deepen his attachment.

The homecoming event detection unit 121 of the data providing device 10 shown in FIG. 9 detects the user returning home event when it is determined that the user has approached the home based on the position information of the user terminal 3 as described above. The homecoming event detection unit 121 may detect the user homecoming event when the user terminal 3 receives a notification from the user terminal 3 indicating that the user terminal 3 has communicated with the beacon transmitter installed at the entrance. The return home event detection unit 121 may detect the user return home event when the user is recognized based on the image taken by the camera-equipped intercom or the input voice. In addition, the homecoming event detection unit 121 may detect the user returning home event when receiving the homecoming notice mail from the user terminal 3.

The user recognition unit 221 of the robot 2 shown in FIG. 8 detects the user, for example, by recognizing a face portion included in a captured image or recognizing an input voice. For example, the user recognition unit 221 photographs the user's face as a sample in advance with the photographing unit 21, analyzes the photographed image, and determines the size, shape, and arrangement of parts (parts such as eyes, nose, and mouth). Extract the feature data. Then, the human recognition unit 225 analyzes the image captured by the photographing unit 21, and when it is determined that the feature data such as the size, shape, and arrangement of the parts includes a subject that is common to or similar to the sample, the image is captured. It may be inferred that the user's face is reflected in the image. Further, the user recognition unit 221 records, for example, a sample user's voice in advance, analyzes the user's voice, and extracts characteristic data such as frequency distribution and intonation. Then, the user recognition unit 221 performs the same analysis on the sound input by the microphone, and when the characteristic data such as the frequency distribution and the intonation match or approximate the case of the user's voice sample, the microphone It may be presumed that the sound input in is corresponding to the user's voice.

In the first record in the data structure of the event information storage unit 151 shown in FIG. 10, a marker whose area type is the entrance type is installed when a user return event is detected as the first event in the application example of welcoming. It is stipulated that the robot 2 moves to the place where it is (that is, the entrance). The first record further stipulates that the robot 2 performs a welcoming action when a user recognition event is detected as a second event. A marker whose area type is the entrance type is called an entrance marker.

In the first record in the data structure of the marker information storage unit 153 shown in FIG. 11, the position and orientation of the entrance marker are set in association with the entrance type of the area type with respect to the application example of welcoming.

FIG. 13A is a flowchart showing a processing procedure in the marker registration phase of the first embodiment. For example, when the marker recognition unit 22 detects the entrance marker while the robot 2 is moving autonomously (step S31), the communication control unit 26 includes the entrance type of the area type and the position and orientation of the entrance marker. Information is transmitted to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the entrance marker information, the marker registration unit 110 registers the position and orientation of the entrance marker in the marker information storage unit 153 in association with the entrance type of the area type ( Step S32).

FIG. 13B is a flowchart showing a processing procedure in the action phase of the first embodiment. When the return home event detection unit 121 of the data providing device 10 detects the user return home event (step S33), the return home event detection unit 121 identifies the entrance type of the area type corresponding to the user return event with reference to the event information storage unit 151. The homecoming event detection unit 121 further refers to the marker information storage unit 153 to specify the position and orientation of the entrance marker corresponding to the entrance type of the area type. The first communication control unit 11 transmits a movement instruction to the entrance including the position and direction of the entrance marker to the robot 2. At this time, the homecoming event detection unit 121 notifies the action selection unit 140 of the user homecoming event.

When the communication control unit 26 of the robot 2 receives a movement instruction to the entrance including the position and orientation of the entrance marker, the movement control unit 23 of the robot 2 controls the movement mechanism 29, and the robot 2 moves to the entrance (step). S34). The robot 2 stays in front of the position of the entrance marker for at least a first predetermined time. The first predetermined time corresponds to the upper limit of the assumed interval from the detection of the user return event to the detection of the user recognition event. If the user recognition event is not detected when the first predetermined time elapses, the process in the action phase of the first embodiment may be interrupted.

When the user opens the door and enters, the robot 2 recognizes the user. Specifically, the user recognition unit 221 recognizes the user's face included in the image taken by the shooting unit 21, or recognizes the sound input by the microphone by voice, and detects the user recognition event (). Step S35). When the user recognition unit 221 of the robot 2 detects the user recognition event, the communication control unit 26 transmits the user recognition event to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the user recognition event, the action selection unit 140 refers to the event information storage unit 151 and determines the first event corresponding to the user recognition event of the second event. Identify the user return event. The action selection unit 140 refers to the event information storage unit 151 and selects a welcoming action corresponding to the combination of the user return event of the first event and the user recognition event of the second event. The first communication control unit 11 transmits an instruction of the selected welcoming action to the robot 2.

When the communication control unit 26 of the robot 2 receives the instruction of the welcoming action, the action executing unit 230 of the robot 2 executes the welcoming action (step S36).

[Example 2]
An application example of babysitting by the autonomous action type robot 1 will be described. In the babysitter application example, when the infant starts crying in the children's room, the robot 2 heads to the children's room to comfort the infant. If an area-type marker that identifies the child's room as an area is installed in the child's room, the autonomous action robot 1 can recognize that the marker is installed in the child's room. The area type that identifies the children's room is called the children's room type.

In the application example of the babysitter, when the infant cries, the robot 2 moves to the children's room and is worried about the infant. The infant's crying is detected by analyzing the sound input to the microphone included in the robot 2. That is, the detection of infant crying corresponds to the first event in the babysitter application example. This event is called an infant crying event.

If there are no adults in the children's room, the robot 2 behaves like a toddler. The behavior of comforting an infant is called a healing action. The absence of an adult is detected by recognizing an image taken by the photographing unit 21. The detection of the absence of an adult corresponds to the second event in the babysitter application. This event is called an adult absence event.

In the bagging action, for example, a speaker provided in the robot 2 outputs a sound effective for stopping the crying of an infant (for example, a bagging voice, a laughing voice, or a sound that makes a paper bag rustle). In addition to the output of voice, the crying action may be an action that moves the components of the robot 2, such as the robot 2 tilting its head or raising or lowering its arms, or an image that is effective for stopping crying of infants. It may be an action such as displaying a predetermined image on the display. The Ayashi action may be a remote control that controls a device different from the robot 2, such as starting a TV soon or outputting music to an audio device. If the robot 2 calms the infant by the tampering action, the user who is the parent will be relieved.

The voice recognition unit 211 of the robot 2 shown in FIG. 8 detects an infant crying event when it is determined that the sound input to the microphone is the infant crying. For example, the voice recognition unit 211 records a sample infant cry in advance, analyzes the infant cry, and extracts characteristic data such as a frequency distribution and a period in which the volume increases. Then, the voice recognition unit 211 performs the same analysis on the sound input to the microphone, and when the characteristic data such as the frequency distribution and the period in which the volume increases match or are similar to the case of the infant crying sample, the voice recognition unit 211 performs the same analysis. It may be inferred that the sound input to the microphone corresponds to the infant crying.

The human recognition unit 225 of the robot 2 shown in FIG. 8 detects an adult absence event when it recognizes that an adult is not shown in the image captured by the photographing unit 21. For example, the human recognition unit 225 captures images of a plurality of adults having different genders and body shapes as samples in advance by the photographing unit 21, analyzes the captured images, and extracts characteristic data such as size and shape. Then, when the human recognition unit 225 analyzes the image captured by the photographing unit 21 and determines that the captured image includes a subject whose characteristic data such as size and shape are common to or similar to the sample, an adult can perform the captured image. You may guess that it is in the picture. The remote control unit 233 of the robot 2 shown in FIG. 8 controls a device different from the robot 2, such as transmitting a wireless signal of the remote control to start a nearby TV or outputting music to an audio device. Perform remote control.

In the second record in the data structure of the event information storage unit 151 shown in FIG. 10, a marker whose area type is the children's room type is installed when an infant crying event is detected as the first event in an application example of child protection. It is stipulated that the robot 2 moves to the place where it is (that is, the children's room). The second record further stipulates that the robot 2 takes an action when an adult absence event is detected as the second event. A marker whose area type is a children's room type is called a children's room marker.

In the second record in the data structure of the marker information storage unit 153 shown in FIG. 11, the position and orientation of the children's room marker are set in association with the child's room type of the area type with respect to the application example of babysitter.

FIG. 14A is a flowchart showing a processing procedure in the marker registration phase of the second embodiment. For example, when the marker recognition unit 22 detects the child room marker when the robot 2 is moving autonomously (step S41), the communication control unit 26 determines the position and orientation of the child room type and the child room marker of the area type. The including child's room marker information is transmitted to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the child room marker information, the marker registration unit 110 registers the position and orientation of the entrance marker in the marker information storage unit 153 in association with the entrance type of the area type. (Step S42).

FIG. 14B is a flowchart showing a processing procedure in the action phase of the second embodiment. When the voice recognition unit 211 of the robot 2 detects the infant crying event (step S43), the communication control unit 26 notifies the data providing device 10 of the infant crying event.

When the first communication control unit 11 of the data providing device 10 receives the notification of the infant crying event, the action selection unit 140 refers to the event information storage unit 151 and refers to the child room of the area type associated with the infant crying event. Identify the type. The action selection unit 140 further refers to the marker information storage unit 153 to specify the position and orientation of the child room marker corresponding to the child room type of the area type. The first communication control unit 11 transmits a movement instruction to the child room including the position and orientation of the child room marker to the robot 2.

When the communication control unit 26 of the robot 2 receives a movement instruction to the children's room including the position and orientation of the children's room marker, the movement control unit 23 controls the movement mechanism 29, and the robot 2 moves to the children's room (step). S44). The robot 2 stays in front of the position of the children's room marker for at least a second predetermined time. The second predetermined time corresponds to the upper limit of the estimated time required for the robot 2 to recognize the absence of an adult after entering the children's room. If the adult absence event is not detected after the lapse of the second predetermined time, the process in the action phase of the second embodiment may be interrupted.

When the human recognition unit 225 detects the adult absence event (step S45), the communication control unit 26 transmits the adult absence event to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the adult absence event, the action selection unit 140 refers to the event information storage unit 151 and makes an infant cry of the first event corresponding to the adult absence event of the second event. Identify the event. The action selection unit 140 refers to the event information storage unit 151 and selects a sneak action corresponding to the combination of the infant crying event of the first event and the adult absence event of the second event. The first communication control unit 11 transmits an instruction of the selected tampering action to the robot 2.

When the communication control unit 26 of the robot 2 receives the instruction of the pacing action, the action executing unit 230 of the robot 2 executes the pacing action (step S46).

[Example 3]
An application example of customer service by the autonomous action type robot 1 will be described. An application example of customer service assumes, for example, a store that provides food and drink services in guest rooms. When a customer who uses the store enters the store, the robot 2 heads to the reception counter and guides the customer to the guest room. If an area type marker that identifies the reception counter as an area is installed at the reception counter, the autonomous action robot 1 can recognize that the installation location of the marker is the reception counter. The area type that identifies the reception counter is called the reception counter type.

In the application example of customer service, when a visitor is detected, the robot 2 moves to the reception counter and responds to the customer. Visitors are detected by the fact that the image taken by the camera installed at the entrance of the store shows the customer entering the store. That is, the detection of a visitor corresponds to the first event in the application example of customer service. This event is called a visitor event.

If there is a vacant room, the robot 2 behaves to guide the customer to the vacant room. The behavior of guiding customers to a vacant room is called a guidance action. Vacancy information can be obtained from the guest room management system (not shown). The detection of a vacant room corresponds to the second event in the application example of customer service. This event is called a vacant room event.

In the guidance action, for example, the robot 2 leads the customer to a vacant room. Alternatively, the message output unit 235 may output the guidance message "Please enter room ○" by voice, or may display it on the display device provided in the robot 2. If the robot 2 serves customers, the customer feels a taste that human service does not have. There is also the aspect that costs can be reduced by unmanned operation.

The visitor event detection unit 123 of the data providing device 10 shown in FIG. 9 detects a visitor event when, for example, an image taken by a camera provided at the entrance of the store recognizes a customer entering the store. The vacant room event detection unit 131 of the data providing device 10 shown in FIG. 9 inquires about the status of the guest room to the guest room management system (not shown), and detects the vacant room event if there is a vacant guest room. The movement control unit 23 of the robot 2 shown in FIG. 8 drives the movement mechanism 29 so as to slowly move to the guest room. The robot 2 may control the speed of movement so as to keep the distance to the customer constant while measuring the distance from the customer according to the appearance of the customer photographed by the photographing unit 21 of the robot 2. Further, the message output unit 235 of the data providing device 10 shown in FIG. 9 outputs the guidance message "Please enter room ○" by voice or displays it on the display device provided in the robot 2. Output a guidance message at.

In the third record in the data structure of the event information storage unit 151 shown in FIG. 10, a marker whose area type is the reception counter type is installed when a visitor event is detected as the first event in the application example of customer service. It is stipulated that the robot 2 moves to the place where the robot 2 is located (that is, the reception counter). The third record further stipulates that the robot 2 performs a guidance action when a vacant room event is detected as the second event. A marker whose area type is the reception counter type is called a reception counter marker.

In the third record in the data structure of the marker information storage unit 153 shown in FIG. 11, the position and orientation of the reception counter marker are set in association with the reception counter type of the area type with respect to the application example of customer service.

FIG. 15A is a flowchart showing a processing procedure in the marker registration phase of the third embodiment. For example, when the marker recognition unit 22 detects the reception counter marker while the robot 2 is moving autonomously (step S51), the communication control unit 26 determines the reception counter type of the area type and the position and orientation of the reception counter marker. The reception counter marker information including the above is transmitted to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the reception counter marker information, the marker registration unit 110 assigns the position and orientation of the reception counter marker to the marker information storage unit 153 in association with the reception counter type of the area type. Register (step S52).

FIG. 15B is a flowchart showing a processing procedure in the action phase of the third embodiment. When the visitor event detection unit 123 of the data providing device 10 detects the visitor event (step S53), the visitor event detection unit 123 refers to the event information storage unit 151 to specify the reception counter type of the area type associated with the visitor event. The visitor event detection unit 123 further refers to the marker information storage unit 153 to specify the position and orientation of the reception counter marker corresponding to the reception counter type of the area type. The first communication control unit 11 transmits a movement instruction to the reception counter including the position and orientation of the reception counter marker to the robot 2. At this time, the visitor event detection unit 123 notifies the action selection unit 140 of the visitor event.

When the communication control unit 26 of the robot 2 receives a movement instruction to the reception counter including the position and orientation of the reception counter marker, the movement control unit 23 controls the movement mechanism 29, and the robot 2 moves to the reception counter (step). S54). The robot 2 stays in front of the position of the reception counter marker for at least a third predetermined time. The third predetermined time corresponds to the upper limit of the switching time until the clerk responds to the customer on behalf of the robot 2. If the vacant room event is not detected when the third predetermined time elapses, the process in the action phase of the third embodiment may be interrupted.

When the vacant room event detection unit 131 of the data providing device 10 detects a vacant room event (step S55), the action selection unit 140 refers to the event information storage unit 151 and is a visitor of the first event corresponding to the vacant room event of the second event. Identify the event. The action selection unit 140 selects a customer service action corresponding to the combination of the visitor event of the first event and the vacant room event of the second event with reference to the event information storage unit 151. The first communication control unit 11 transmits the instruction of the selected customer service action to the robot 2.

When the communication control unit 26 of the robot 2 receives the instruction of the customer service action, the action execution unit 230 of the robot 2 executes the customer service action (step S56).

[Example 4]
An application example of call support by the autonomous action type robot 1 will be described. In this example, it is assumed that the robot 2 has a telephone function. Further, it is assumed that some of the houses and facilities in which the robot 2 is placed can easily reach the radio waves of the wireless communication used in the telephone function, and some of them are difficult to reach. The robot 2 moves from a place where the radio wave condition is bad to a place where the radio wave condition is good (hereinafter, referred to as a place where the radio wave condition is good), and starts a call in response to a call request. If a marker of the area type that identifies a place with good radio waves as an area is installed in a place with good radio waves, the autonomous action robot 1 can recognize that the place where the marker is installed is a place with good radio waves. The area type that identifies a place with good radio waves is called a good radio wave type.

In the application example of the call support, when the radio wave condition deteriorates, the robot 2 moves to a place where the radio wave is good so that the wireless communication is not hindered. That is, the detection of radio wave deterioration corresponds to the first event in the application example of the call support. This event is called a radio wave deterioration event.

When the robot 2 receives a call request from the user, the robot 2 starts the call. The process operation to start a call is called a call start action. The reception of a call request is detected by recognition processing such as recognizing a voice such as "Please make a call" or recognizing a gesture or pose for making a call from a captured image. The reception of a call request corresponds to the second event in the application example of the call support. This event is called a call request event.

The radio wave condition detection unit 213 of the robot 2 shown in FIG. 8 monitors the radio wave condition of the wireless communication used in the telephone function, and detects a radio wave deterioration event when the strength of the radio wave becomes equal to or less than the permissible standard. The call request reception unit 223 of the robot 2 shown in FIG. 8 recognizes the utterance of "Please make a call" by voice, and recognizes the gesture or pose of making a call based on the captured image. Accepts call requests from. The call request reception unit 223 may specify a telephone number or a call partner by voice recognition. The telephone communication unit 237 of the robot 2 shown in FIG. 8 controls telephone communication and performs call processing.

The fourth record in the data configuration of the event information storage unit 151 shown in FIG. 10 contains a marker whose area type is demparyoukou type when a radio wave deterioration event is detected as the first event in an application example of call support. It is stipulated that the robot 2 moves to the place where it is installed (that is, the place where the radio wave is good). The fourth record further stipulates that the robot 2 starts a call when a call request event is detected as the second event. A marker whose area type is a good radio wave type is called a good radio wave marker.

In the fourth record in the data structure of the marker information storage unit 153 shown in FIG. 11, the position and direction of the radio wave good marker are set in association with the radio wave good type of the area type for the application example of the call support.

FIG. 16A is a flowchart showing a processing procedure in the marker registration phase of the fourth embodiment. For example, when the marker recognition unit 22 detects a good radio wave marker while the robot 2 is moving autonomously (step S61), the communication control unit 26 determines the position and direction of the good radio wave type and the good radio wave marker for each area type. The included radio wave good marker information is transmitted to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the radio wave good marker information, the marker registration unit 110 assigns the position and direction of the radio wave good marker to the marker information storage unit 153 in association with the radio wave good type of the area type. Register (step S62).

FIG. 16B is a flowchart showing a processing procedure in the action phase of the fourth embodiment. When the radio wave condition detection unit 213 of the robot 2 detects the radio wave deterioration event (step S63), the communication control unit 26 notifies the data providing device 10 of the radio wave deterioration event.

When the first communication control unit 11 of the data providing device 10 receives the notification of the radio wave deterioration event, the action selection unit 140 refers to the event information storage unit 151 and has a good radio wave of the area type associated with the radio wave deterioration event. Identify the type. The action selection unit 140 further refers to the marker information storage unit 153 to specify the position and orientation of the demparyoukou marker corresponding to the demparyoukou type of the area type. The first communication control unit 11 transmits to the robot 2 a movement instruction to a place where the radio wave is good, including the position and direction of the radio wave good marker.

When the communication control unit 26 of the robot 2 receives a movement instruction to a place with good radio waves including the position and direction of the radio wave deterioration marker, the movement control unit 23 controls the movement mechanism 29, and the robot 2 moves to a place with good radio waves. Move (step S64). The robot 2 stays in front of the position of the radio wave good marker for at least a fourth predetermined time. The fourth predetermined time corresponds to the upper limit length of the period in which the user is expected to make a call. If the call request event is not detected when the fourth predetermined time elapses, the process in the action phase of the fourth embodiment may be interrupted.

When the call request receiving unit 223 detects the call request event (step S65), the communication control unit 26 transmits the call request event to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the call request event, the action selection unit 140 refers to the event information storage unit 151 and selects a call start action corresponding to the call request event of the second event. .. The first communication control unit 11 transmits an instruction of the selected call start action to the robot 2. Therefore, the call start action is transmitted regardless of whether or not the radio wave deterioration event has been notified first.

When the communication control unit 26 of the robot 2 receives the instruction of the call start action, the telephone communication 237 of the robot 2 executes the call start action (step S66). When the telephone communication unit 237 calls and enters a call state, a signal converted from the input voice of the microphone included in the robot 2 is transmitted, and the other party voice converted from the received signal is output from the microphone.

[Example 5]
An application example of security by the autonomous action type robot 1 will be described. In the application example of security, the robot 2 patrols on the assumption that the safe is warned in the house or facility where the safe is placed. If an area-type marker that identifies the safe storage area as an area is installed in the safe storage area, the autonomous action robot 1 can recognize that the installation location of the marker is the safe storage area. The area type that identifies the safe storage area is called the safe type.

In the security application example, the robot 2 instructed to look around moves to the vicinity of the safe and behaves as if to grasp the situation. The reception of the patrol instruction may be performed by recognition processing such as recognition of a voice such as "look at the safe" or recognition of a predetermined pose or gesture from the captured image. The data providing device 10 may receive the watching instruction from the application of the user terminal 3 and transfer it to the robot 2, and the robot 2 may receive the watching instruction. When the predetermined time comes, the data providing device 10 may automatically send a patrol instruction to the robot 2, and the robot 2 may receive the watching instruction. In the security application example, the patrol instruction to the robot 2 corresponds to the first event. This event is called a patrol event.

When a person is caught near the safe by recognizing the captured image, the robot 2 executes a warning action. In other words, in the application example of security, the recognition of a person near the safe corresponds to the second event. This event is called a human recognition event. It is assumed that a person near the safe may be a suspicious person. As a warning action, the communication control unit 26 transmits the video (moving image or still image) captured by the robot 2 to the data providing device 10. Regarding this process, the communication control unit 26 is an example of the action execution unit 230. The data providing device 10 may record the received video as evidence. Further, the data providing device 10 may transmit a warning message to the application of the user terminal 3 or transfer the video received from the robot 2 to transmit the data to the application of the user terminal 3. .. As a warning action, the voice output unit 232 of the robot 2 may emit an alarm sound.

The patrol event detection unit 215 of the robot 2 shown in FIG. 8 detects the patrol event when the patrol instruction is received in the above recognition process or when the patrol instruction is received from the data providing device 10. The human recognition unit 225 of the robot 2 shown in FIG. 8 recognizes the figure of a person included in the captured image. The existence of a person may be recognized by recognizing the voice of the person. The human recognition unit 225 may determine that the voice is a human voice when a frequency corresponding to a standard human voice is extracted from the sound input to the microphone.

In the fifth record in the data configuration of the event information storage unit 151 shown in FIG. 10, a marker whose area type is a safe type is installed when a patrol event is detected as the first event in an application example of security. It is stipulated that the robot 2 moves to the place where it is (that is, the safe storage place). The fifth record further stipulates that the robot 2 takes a warning action when a human recognition event is detected as the second event. A marker whose area type is a safe type is called a safe marker.

In the fifth record in the data structure of the marker information storage unit 153 shown in FIG. 11, the position and orientation of the safe marker are set in association with the safe type of the area type with respect to the application example of security.

FIG. 17A is a flowchart showing a processing procedure in the marker registration phase of the fifth embodiment. For example, when the marker recognition unit 22 detects the safe marker when the robot 2 is moving autonomously (step S71), the communication control unit 26 determines the safe type of the area type and the safe marker including the position and orientation of the safe marker. Information is transmitted to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the safe marker information, the marker registration unit 110 registers the position and orientation of the safe marker in the marker information storage unit 153 in association with the safe type of the area type ( Step S72).

FIG. 17B is a flowchart showing a processing procedure in the action phase of the fifth embodiment. When the patrol event detection unit 215 of the robot 2 detects the patrol event (step S73), the communication control unit 26 notifies the data providing device 10 of the patrol event.

When the first communication control unit 11 of the data providing device 10 receives the notification of the patrol event, the action selection unit 140 refers to the event information storage unit 151 and identifies the safe type of the area type associated with the patrol event. To do. The action selection unit 140 further refers to the marker information storage unit 153 to specify the position and orientation of the safe marker corresponding to the safe type of the area type. The first communication control unit 11 transmits a movement instruction to the safe storage area including the position and orientation of the safe marker to the robot 2.

When the communication control unit 26 of the robot 2 receives a movement instruction to the safe storage area including the position and orientation of the safe marker, the movement control unit 23 controls the movement mechanism 29, and the robot 2 moves to the vicinity of the safe (step S74). ). The robot 2 stays in front of the position of the safe marker for at least a fifth predetermined time. The fifth predetermined time corresponds to the upper limit of the estimated time required for the robot 2 to recognize a person after moving to the vicinity of the safe. If the human recognition event is not detected after the lapse of the fifth predetermined time, the process in the action phase of the fifth embodiment may be interrupted.

When the person recognition unit 225 detects the person recognition event (step S75), the communication control unit 26 transmits the person recognition event to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the person recognition event, the action selection unit 140 refers to the event information storage unit 151 and performs a patrol event of the first event corresponding to the person recognition event of the second event. To identify. The action selection unit 140 refers to the event information storage unit 151 and selects a warning action corresponding to the combination of the patrol event of the first event and the person recognition event of the second event. The first communication control unit 11 transmits an instruction of the selected alert action to the robot 2.

When the communication control unit 26 of the robot 2 receives the instruction of the alert action, the action execution unit 230 of the robot 2 executes the alert action (step S76).

[Example 6]
An application example of the alarm clock by the autonomous action type robot 1 will be described. For example, the robot 2 wakes up a user sleeping in the bedroom in the morning. If an area type marker that identifies the bedroom as an area is installed in the bedroom, the autonomous action robot 1 can recognize that the location of the marker is the bedroom. The area type that identifies the bedroom is called the bedroom type.

In the application example of the alarm clock, the robot 2 moves to the bedroom and performs an alarm clock action at the timing of waking up the user. The timing for waking up the user is determined, for example, by the scheduled wake-up time. Alternatively, the user is woken up by voice recognition such as "Please wake up the father (user)" issued by the user's family, or recognition processing such as recognizing a predetermined pose or gesture from the captured image. You may judge the timing. That is, the determination of the timing of waking up the user corresponds to the first event in the application example of the alarm clock. This event is called a wake-up event.

When the user recognizes that the user is lying on the bed in the bedroom (lying position), the robot 2 executes an alarm clock action. When the user is not in the lying position, such as when there is no user on the bed or when the user is already awake, the robot 2 does not perform the wake-up action. That is, detecting a user lying on the bed in the application example of the alarm clock corresponds to the second event. This event is called a user recumbent event. As an alarm clock action, for example, the voice output unit 232 outputs a voice such as an alarm clock or a call from the speaker. As an alarm clock action, the remote control unit 233 may activate the television, output music to the audio device, or output audio to an external device. The remote control unit 233 may turn on the lighting device. Alternatively, the movement control unit 23 may control the movement mechanism 29 so that the robot 2 violently moves around the bed.

The wake-up event detection unit 217 of the robot 2 shown in FIG. 8 detects the wake-up event when the scheduled wake-up time is reached as described above or by the recognition process described above. The posture recognition unit 227 of the robot 2 shown in FIG. 8 recognizes the posture of a person sleeping on a bed in the bedroom (actually, it is regarded as a user). The body position recognition unit 227 detects the user lying position event if the person sleeping on the bed is in the lying position. The body position recognition unit 227, for example, photographs the figure of the user in the lying position as a sample in advance by the photographing unit 21, analyzes the photographed image, and analyzes the size, shape, and parts (head, hand, and parts) of the user in the lying position. Extract feature data such as the placement of (parts such as legs). Then, the body position recognition unit 227 analyzes the image captured by the imaging unit 21, and when it is determined that the feature data such as the size, shape, and arrangement of the parts includes a subject that is common to or similar to the sample, the image is captured. It may be determined that the user in the recumbent position is shown in the image.

In the sixth record in the data structure of the event information storage unit 151 shown in FIG. 10, a marker whose area type is the bedroom type is installed when a wake-up event is detected as the first event in the application example of the alarm clock. It is stipulated that the robot 2 moves to the place where it is (that is, the bedroom). The sixth record further stipulates that the robot 2 performs an alarm clock action when a user recumbent event is detected as a second event. A bedroom type marker is called a bedroom marker.

In the sixth record in the data structure of the marker information storage unit 153 shown in FIG. 11, the position and orientation of the bedroom marker are set in association with the bedroom type of the area type with respect to the application example of the alarm clock.

FIG. 18A is a flowchart showing a processing procedure in the marker registration phase of the sixth embodiment. For example, when the marker recognition unit 22 detects the bedroom marker when the robot 2 is moving autonomously (step S81), the communication control unit 26 determines the bedroom type of the area type and the bedroom marker including the position and orientation of the bedroom marker. Information is transmitted to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the bedroom marker information, the marker registration unit 110 registers the position and orientation of the bedroom marker in the marker information storage unit 153 in association with the bedroom type of the area type ( Step S82).

FIG. 18B is a flowchart showing a processing procedure in the action phase of the sixth embodiment. When the wake-up event detection unit 217 of the robot 2 detects the wake-up event (step S83), the communication control unit 26 notifies the data providing device 10 of the wake-up event.

When the first communication control unit 11 of the data providing device 10 receives the notification of the wake-up event, the action selection unit 140 refers to the event information storage unit 151 to specify the bedroom type of the area type associated with the wake-up event. To do. The action selection unit 140 further refers to the marker information storage unit 153 to specify the position and orientation of the bedroom marker corresponding to the bedroom type of the area type. The first communication control unit 11 transmits a movement instruction to the bedroom including the position and orientation of the bedroom marker to the robot 2.

When the communication control unit 26 of the robot 2 receives a movement instruction to the bedroom including the position and orientation of the bedroom marker, the movement control unit 23 controls the movement mechanism 29, and the robot 2 moves to the bedroom (step S84). The robot 2 stays in front of the position of the bedroom marker for at least the sixth predetermined time. The sixth predetermined time corresponds to the upper limit of the estimated time required from when the robot 2 enters the bedroom to when the user lying down event is detected. If the user recumbent event is not detected when the sixth predetermined time elapses, the process in the action phase of the sixth embodiment may be interrupted.

When the body position recognition unit 227 detects the user lying position event (step S85), the communication control unit 26 transmits the user lying position event to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the user decubitus event, the action selection unit 140 refers to the event information storage unit 151 and corresponds to the user decubitus event of the second event. Identify the event wakeup event. The action selection unit 140 selects an alarm clock action corresponding to the combination of the wake-up event of the first event and the user recumbent event of the second event with reference to the event information storage unit 151. The first communication control unit 11 transmits an instruction of the selected wake-up action to the robot 2.

When the communication control unit 26 of the robot 2 receives the instruction of the alarm clock action, the action execution unit 230 of the robot 2 executes the alarm clock action (step S86).

In the above-described first to sixth embodiments, the process described to be performed by the first event detection unit 210 of the robot 2 may be performed by the first event detection unit 120 of the data providing device 10. The process described to be performed by the second event detection unit 220 of the robot 2 may be performed by the second event detection unit 130 of the data providing device 10. The process described as being performed by the first event detection unit 120 of the data providing device 10 may be performed by the first event detection unit 210 of the robot 2. The process described as being performed by the second event detection unit 130 of the data providing device 10 may be performed by the second event detection unit 220 of the robot 2.

In the second embodiment described above, the robot 2 is moved to a predetermined place at a predetermined timing, and when a predetermined condition is satisfied, a predetermined action is executed. By installing the marker in a predetermined place, it is convenient because such a series of operations can be easily realized.

[Embodiment 3]
When the robot 2 recognizes the marker, the robot 2 may execute the action instructed by the marker. For that purpose, a marker in which the identifier of the action is graphic is used. For example, a graphic code in which an action identifier is graphicized by a general-purpose conversion method (bar code method, two-dimensional bar code method, etc.) may be used as a marker. In this case, the marker recognition unit 22 can read the action identifier from the graphic code by the conversion method. Alternatively, a uniquely designed figure may be used as a marker. In this case, the marker recognition unit 22 specifies the type of the figure included in the captured image when the shape of the figure is a predetermined shape, and specifies the identifier of the action corresponding to the specified type of the figure. It may be. It is assumed that the marker recognition unit 22 stores data that associates the type of the figure with the identifier of the action. That is, the marker in the third embodiment can identify the identifier of any action.

For example, if a marker is installed at the entrance of the children's room to instruct the movement to the living room as an action, the robot 2 does not enter the children's room when it recognizes the marker in front of the children's room. Immediately move to the living room.

The action directed by the marker may be the search for another predetermined marker. If the action instructed by the marker A is the search for the marker B, the robot 2 searches for the marker B at the stage of recognizing the marker A and starts moving. Further, if the action instructed by the marker B is the search for the marker C, the robot 2 searches for the marker C at the stage of recognizing the marker B and starts moving. If a series of markers instructing the search for the markers in sequence are arranged at points on the route, the robot 2 is made to search the route around the series of markers in order. In this way, it is possible to play an indoor game that imitates orienteering with a child and a robot 2. It is also possible to run a plurality of robots 2 to perform an indoor race.

When performing indoor play such as the above-mentioned indoor game or indoor race, the user instructs the autonomous action robot 1 from the application of the user terminal 3 that the robot 2 first searches for the marker A. May be good. The robot 2 may start searching for a marker by voice-recognizing a user's shout such as "start" or "find" and triggering an event that detects the user's shout. The robot 2 may image-recognize a pose or gesture instructing the user to start from the captured image, and may start searching for a marker triggered by an event in which the pose or gesture is detected. Alternatively, the user sets a start time for the autonomous action robot 1 from the application of the user terminal 3, and the robot 2 starts searching for a marker as a starting point when the start time is reached. May be good.

In the third embodiment described above, by installing the marker at a predetermined place, the action of the robot 2 at that place can be easily instructed.

[Embodiment 4]
The robot 2 may be made to recognize an article that should not be approached by using a marker instructing the prohibition of approach. For example, a marker with a graphic representation of an access prohibition code is installed on fragile items such as ornaments and precision equipment. As described above, a graphic code obtained by graphicizing the access prohibition code by a general-purpose conversion method (bar code method, two-dimensional bar code method, etc.) may be used as a marker. Alternatively, a uniquely designed figure may be used as a marker. In this case, the marker recognition unit 22 stores the type of the figure corresponding to the access prohibition, and determines that the type of the figure specified from the captured image corresponds to the access prohibition. The marker recognition unit 22 of the robot 2 measures the distance from the recognized marker. The movement control unit 23 controls the movement mechanism 29 so as to move the robot 2 in a direction away from the marker when it is determined that the distance from the marker is shorter than the first reference distance. The first reference distance is a distance required for the robot 2 to change the direction and control the movement of the robot 2 so as not to reach the position of the marker. In this way, the risk of the robot 2 colliding with a fragile article can be reduced.

Further, a marker for instructing access prohibition may be installed on an article having a high possibility of moving, such as a balance ball or a vacuum cleaner body, so that the robot 2 recognizes the article having a high possibility of moving. The movement control unit 23 controls the movement mechanism 29 so as to move the robot 2 in a direction away from the marker when it is determined that the distance from the marker is shorter than the second reference distance. The second reference distance is a distance required for the robot 2 to control to increase the distance to the article before the article moves and reaches the position of the robot 2. In this way, the risk of the article moving and colliding with the robot 2 can be reduced.

The type of the article may be identified by the marker, and the approach of the robot 2 may be regulated according to the type of the article. A marker indicating the type of the article may be used, or the article management data for associating the marker ID with the type of the article may be held by the data providing device 10. When a marker indicating the type of the article is used, the type of the article is also detected when the marker recognition unit 22 recognizes the marker. The marker recognition unit 22 stores article management data that associates the marker ID with the type of article. The marker recognition unit 22 detects the marker ID from the marker and identifies the type of the article corresponding to the marker ID with reference to the article management data. The article management data may be set by the user from the application of the user terminal 3.

Further, the marker recognition unit 22 may store the approach control data in which the accessibility of the robot 2 is set for each type of article, and may determine the accessibility of the type of article specified by the marker ID. In the approach control data, access permission is set for items that are difficult to break and are unlikely to move, such as tables and chairs, and fragile items such as ornaments and precision equipment, and balance balls and vacuum cleaners. No access is set for items that are likely to move in this way. Since the vacuum cleaner body always moves forward and never moves backward, the vacuum cleaner body may be allowed to approach the rear. In other words, it is sufficient to prohibit only the approach to the front of the vacuum cleaner body. If the front and back of the vacuum cleaner body can be recognized by the direction of the marker, only the approach to the front of the vacuum cleaner body may be prohibited. If the marker is attached so that the marker includes an arrow and the tip of the arrow points to the front of the vacuum cleaner, the robot 2 can determine the range in which access is prohibited based on the marker. If there is a rule that the point pointed by the arrow is prohibited from approaching, there is no operational problem if the direction of the marker is noted at the stage of pasting the marker. When setting the access prohibition range not limited to the point pointed by the arrow, the access prohibition range based on the arrow may be set in the approach control data, or the same approach from the application of the user terminal 3 may be set. The prohibited range may be set.

In the fourth embodiment described above, by placing a mark on an article that has a risk of collision with the robot 2, such as a fragile article or an article that may move, it is easy to avoid a collision between the robot 2 and the article. ..

[Other variants]

Regarding the second embodiment, the marker recognition unit 22 may store the marker definition information for associating the marker ID with the area type, detect the marker ID from the marker, and specify the area type corresponding to the marker ID. The association between the marker ID and the area type may be set by the user from the application of the user terminal 3.

Regarding the second embodiment, the first event is not limited to the above-mentioned example. The first event is optional. The first event detection unit 210 of the robot 2 and the first event detection unit 120 of the data providing device 10 may detect the first event when recognizing a person other than the user. The first event detection unit 210 of the robot 2 and the first event detection unit 120 of the data providing device 10 may detect the first event when an unknown person who has never been recognized is recognized for the first time. The first event detection unit 210 of the robot 2 and the first event detection unit 120 of the data providing device 10 may detect the first event when a known person who has been recognized in the past is recognized again. The first event detection unit 210 of the robot 2 and the first event detection unit 120 of the data providing device 10 may detect the first event when the same person is repeatedly recognized within a predetermined time. The first event detection unit 210 of the robot 2 and the first event detection unit 120 of the data providing device 10 may detect the first event when recognizing a person who has not been recognized within a predetermined time. Further, when the first event detection unit 210 of the robot 2 and the first event detection unit 120 of the data providing device 10 determine that the positional relationship and orientation between the person and the marker recognized in the captured image satisfy a predetermined condition. The first event may be detected.

Regarding the second embodiment, the second event is not limited to the above-mentioned example. The second event is optional. The second event detection unit 220 of the robot 2 and the second event detection unit 130 of the data providing device 10 may detect the second event when recognizing a person other than the user. The second event detection unit 220 of the robot 2 and the second event detection unit 130 of the data providing device 10 may detect the second event when the unknown person who has never been recognized is recognized for the first time. The second event detection unit 220 of the robot 2 and the second event detection unit 130 of the data providing device 10 may detect the second event when a known person who has been recognized in the past is recognized again. The second event detection unit 220 of the robot 2 and the second event detection unit 130 of the data providing device 10 may detect the second event when the same person is repeatedly recognized within a predetermined time. The second event detection unit 220 of the robot 2 and the second event detection unit 130 of the data providing device 10 may detect the second event when recognizing a person who has not been recognized within a predetermined time. Further, when the second event detection unit 220 of the robot 2 and the second event detection unit 130 of the data providing device 10 determine that the positional relationship and orientation between the person and the marker recognized in the captured image satisfy a predetermined condition. The second event may be detected.

With respect to the second embodiment, the first event and the second event may be a combination of a plurality of stages of events. For example, in the application example of welcoming described in the first embodiment, the homecoming event detection unit 121 of the data providing device 10 detects the event of the first stage when the position of the user terminal 3 approaches the home, and the user terminal. The second stage event may be detected when the 3 communicates with the beacon transmitter installed at the entrance. Then, when the homecoming event detection unit 121 detects both the first stage event and the second stage event, it may be determined that the first event has been detected.

Regarding the second embodiment, the example in which the action selection unit 140 selects the action corresponding to the combination of the first event and the second event has been described, but the action selection unit 140 may select the action corresponding to the second event. Good. The action selection unit 140 may select an action corresponding to the area type. The action selection unit 140 may select an action corresponding to the first event. The action selection unit 140 may select an action corresponding to the combination of the area type and the second event. The action selection unit 140 may select an action corresponding to the combination of the first event and the area type. The action selection unit 140 may select an action corresponding to a combination of the first event, the area type, and the second event.

Regarding the second embodiment, the action execution unit 230 may perform an action targeting a specific person. For example, in the customer service application example described in the third embodiment, the action execution unit 230 may guide a vacant room to a specific customer. Further, the content of the action executed by the action execution unit 230 may be set by the user from the application of the user terminal 3.

Regarding the second embodiment, the first event detection unit 210 of the robot 2 may detect the first event when the communication control unit 26 communicates with another robot 2. The second event detection unit 220 of the robot 2 may detect the second event when the communication control unit 26 communicates with another robot 2. Even if the first event detection unit 210 of the robot 2 and the first event detection unit 120 of the data providing device 10 detect the first event when a predetermined instruction or data is received from the user terminal 3 or another external device. Good. Even if the second event detection unit 220 of the robot 2 and the second event detection unit 130 of the data providing device 10 detect the second event when a predetermined instruction or data is received from the user terminal 3 or another external device. Good. For example, in the customer service application example described in the third embodiment, the visitor event detection unit 123 of the data providing device 10 notifies from the reception tablet terminal that the reception tablet terminal has accepted the guest room conditions such as the number of visitors and smoking cessation. The first event may be detected when it is done.

Regarding the second embodiment, the detection conditions of the first event and the second event may be different for each of the plurality of robots 2. For example, only the specific robot 2 may detect the first event or the second event later than the timing for detecting the first event or the second event. By delaying the detection timing of the first event or the second event, the passive character of the robot 2 can be produced. On the contrary, only the specific robot 2 may detect the first event or the second event earlier than the timing when the first event or the second event is usually detected. If the detection timing of the first event or the second event is advanced, the positive character of the robot 2 can be produced. The contents of the first event and the second event may be set by the user from the application of the user terminal 3.

Regarding the second embodiment, the event information storage unit 151 may set different event information for each robot 2. That is, event information that is applied only to the specific robot 2 may be provided. For example, in a home in which two robots 2 are operated, event information may be set so that only one robot performs a welcoming action and only the other robot performs a babysitter action.

When operating a plurality of robots 2, the marker information recognized by one robot 2 may be notified to the other robot 2. By doing so, the area type, the position and orientation of the marker can be shared quickly.

The robot 2 may consider the feature points and feature shapes detected in the SLAM technology as markers. Further, a device that emits light as a marker may be used. A marker may be installed in the charging station for supplying power to the storage battery included in the robot 2, and the robot 2 may detect the positional relationship and the orientation with the charging station based on the marker. The robot 2 may use the positional relationship and orientation of the charging station detected by the marker when approaching the charging station for automatic connection.

The marker recognition unit 22 may measure the positions of the same marker a plurality of times and obtain an average value for those positions. If the marker recognition unit 22 uses the average value of the marker positions, the influence of an error in measuring the marker position can be reduced. Similarly, the marker recognition unit 22 may measure the orientation of the same marker a plurality of times and obtain an average value for those orientations. If the marker recognition unit 22 uses the average value of the orientations of the markers, the influence of an error in measuring the position of the marker can be reduced.

The application of the user terminal 3 may display the position and orientation of the marker on the output device of the user terminal 3. From the application of the user terminal 3, the user sets the content of the marker (entry prohibited, area type, action identifier or access prohibited, etc.), the position and orientation of the marker, via the input device of the user terminal 3. Alternatively, it may be possible to modify it.

The robot 2 may include a beacon receiver, receive a beacon signal transmitted by a beacon transmitter installed at a predetermined position at the beacon receiver, and specify the ID of the beacon transmitter. The robot 2 may further include a beacon analysis unit, and the beacon analysis unit may identify the position of the beacon transmitter by analyzing the radio wave intensity of the beacon signal. Therefore, the autonomous action type robot 1 may consider the ID of the beacon transmitter as the marker ID and the position of the beacon transmitter as the position of the marker, and apply it to the above-described embodiment.

It should be noted that the program for realizing the function constituting the apparatus described in the present embodiment is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into the computer system and executed. Therefore, the above-mentioned various processes of the present embodiment may be performed. The "computer system" referred to here may include hardware such as an OS and peripheral devices. Further, the "computer system" includes a homepage providing environment (or a display environment) if a WWW system is used. The "computer-readable recording medium" includes a flexible disk, a magneto-optical disk, a ROM, a writable non-volatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, and the like. It refers to the storage device of.

Further, the "computer-readable recording medium" is a volatile memory (for example, DRAM (Dynamic)) inside a computer system that serves as a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. It also includes those that hold the program for a certain period of time, such as Random Access Memory)). Further, the program may be transmitted from a computer system in which this program is stored in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the "transmission medium" for transmitting a program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Further, the above program may be for realizing a part of the above-mentioned functions. Further, it may be a so-called difference file (difference program) that realizes the above-mentioned function in combination with a program already recorded in the computer system.

Although the embodiments of the present invention have been described above with reference to the drawings, the specific configuration is not limited to this embodiment and includes various modifications within a range not deviating from the gist of the present invention. Is done.

Claims (17)

Movement mechanism and
A shooting section that shoots the surrounding space,
A marker recognition unit that recognizes a predetermined marker included in a captured image captured by the photographing unit, and a marker recognition unit.
A robot including a movement control unit that controls movement by the movement mechanism based on the recognized marker. The robot according to claim 1, wherein the movement control unit prohibits entry by the movement based on the recognized marker. The robot according to claim 1 or 2, wherein the movement control unit limits the speed of the movement based on the recognized marker. The robot according to any one of claims 1 to 3, wherein the movement control unit controls the movement based on the recognized installation position of the marker. The robot according to claim 4, wherein the movement control unit sets a limited range based on the installation position and limits the movement in the limited range. The robot according to claim 5, wherein the movement control unit sets a predetermined range as the limiting range on the back side of the installation position or around the installation position. The robot according to any one of claims 4 to 6, wherein the movement control unit restricts the movement based on the recognized plurality of the recognized installation positions when the number of the recognized markers is a plurality. The robot according to claim 7, wherein the movement control unit limits the movement based on a line segment connecting the recognized first marker installation position and the recognized second marker installation position. The robot according to any one of claims 1 to 8, wherein the movement control unit controls the movement based on the recognized type of the marker. The robot according to any one of claims 1 to 9, wherein the movement control unit controls the movement based on a recorded marker. The robot according to claim 10, wherein the movement control unit controls the movement based on the recorded marker when the marker is not recognized in the captured image. A spatial data generation unit that generates spatial data that recognizes the space based on a captured image captured by the imaging unit, and a spatial data generation unit.
A visualization data generation unit that generates visualization data that visualizes spatial elements included in the space based on the generated spatial data.
The robot according to any one of claims 1 to 11, further comprising a visualization data providing unit that provides the generated visualization data to a user terminal. Further provided with a designation acquisition unit for acquiring the designation of the area included in the provided visualization data from the user terminal.
The robot according to claim 12, wherein the space data generation unit re-recognizes the space based on the captured image re-photographed in the acquired region according to the designation. Further, a state information acquisition unit for acquiring state information indicating the state of the movement destination in the movement is provided.
The robot according to any one of claims 1 to 13, wherein the movement control unit further controls the movement based on the state information. A marker information storage unit that stores the position of the marker,
The first event detection unit that detects the first event and
The second event detection unit that detects the second event,
It also has an action execution unit that executes actions,
When the first event is detected, it moves to the vicinity of the position of the marker, and when the second event is detected, it becomes at least one of the marker, the first event, and the second event. The robot according to any one of claims 1 to 14, which executes the corresponding action. Shooting steps to shoot the surrounding space and
A marker recognition step that recognizes a predetermined marker included in a captured image captured in the photographing step, and a marker recognition step.
A robot control method including a movement control step for controlling movement by a movement mechanism based on the recognized marker. On the computer
A shooting function that shoots the surrounding space and
A marker recognition function that recognizes a predetermined marker included in a shot image shot by the shooting function, and a marker recognition function.
A robot control program for realizing a movement control function that controls movement by a movement mechanism based on the recognized marker.

Images