JP2024020582A - Robots, their control methods, and programs - Google Patents

Abstract

The present invention provides a robot, a control method thereof, and a program that allow a user to control the movement of the robot. [Solution] The robot includes a movement mechanism, a photographing unit that photographs the surrounding space, a marker recognition unit that recognizes a predetermined marker included in a photographed image photographed by the photographing unit, and moves based on the recognized marker. and a movement control section that controls movement by the mechanism. [Selection diagram] Figure 1

Description

The present invention relates to a robot, a control method thereof, and a program.

BACKGROUND ART Conventionally, there have been robots that autonomously move inside a house while taking images with a camera, recognize the indoor space from the captured images, set a movement route based on the recognized space, and move indoors. The robot movement route is set by the user creating in advance a map that defines the robot movement route. The robot can move along a route determined based on the created map (see, for example, Patent Document 1).

Japanese Patent Application Publication No. 2016-103277

However, since autonomous robots can move autonomously in space, for example, the robot may enter an area that the user does not want it to enter or an area that is dangerous if it moves.

Furthermore, in addition to autonomous movement, there are times when we want the robot to move to a predetermined location and perform a predetermined action.

The present invention has been made in view of the above circumstances, and in one embodiment, an object thereof is to provide a robot, a control method thereof, and a program that allow a user to control the movement of the robot.

(1) In order to solve the above problems, the robot of the embodiment includes a moving mechanism, a photographing section that photographs the surrounding space, and a marker that recognizes a predetermined marker included in the photographed image photographed by the photographing section. It includes a recognition section and a movement control section that controls movement by the movement mechanism based on the recognized marker.

(2) Furthermore, in the robot of the embodiment, the movement control unit prohibits entry by movement based on the recognized marker.

(3) Furthermore, in the robot of the embodiment, the movement control unit limits the speed of the movement based on the recognized marker.

(4) Furthermore, in the robot of the embodiment, the movement control section controls the movement based on the recognized installation position of the marker.

(5) Furthermore, in the robot of the embodiment, the movement control section sets a restriction range based on the installation position, and limits the movement within the restriction range.

(6) Furthermore, in the robot of the embodiment, the movement control section sets a predetermined range behind the installation position or around the installation position as the limit range.

(7) Furthermore, in the robot of the embodiment, when a plurality of the markers are recognized, the movement control unit limits the movement based on the plurality of recognized installation positions.

(8) Furthermore, in the robot of the embodiment, the movement control unit limits the movement based on a line segment connecting the recognized installation position of the first marker and the recognized installation position of the second marker.

(9) Furthermore, in the robot of the embodiment, the movement control unit controls the movement based on the recognized type of the marker.

(10) Furthermore, in the robot of the embodiment, the movement control section controls the movement based on recorded markers.

(11) Furthermore, in the robot of the embodiment, the movement control unit controls the movement based on the recorded marker when the marker is not recognized in the photographed image.

(12) Further, in the robot of the embodiment, a spatial data generation unit generates spatial data in which the space is recognized based on a photographed image photographed by the photographing unit, and based on the generated spatial data, The apparatus further includes a visualization data generation section that generates visualization data that visualizes spatial elements included in the space, and a visualization data provision section that provides the generated visualization data to a user terminal.

(13) The robot of the embodiment further includes a designation acquisition unit that acquires from the user terminal a designation of a region included in the provided visualization data, and the spatial data generation unit is configured to The space is re-recognized based on the photographed image re-photographed in the area.

(14) The robot of the embodiment further includes a state information acquisition unit that acquires state information indicating a state of a destination in the movement, and the movement control unit controls the movement further based on the state information. .

(15) Further, in the robot of the embodiment, a marker information storage section that stores the position of the marker, a first event detection section that detects the first event, and a second event detection section that detects the second event, and an action execution unit that executes an action, when the first event is detected, the marker is moved to the vicinity of the position of the marker, and when the second event is detected, the marker and the first event are moved to the vicinity of the position of the marker. and performing the action corresponding to at least one of the second events.

(16) In order to solve the above problems, the robot control method of the embodiment includes a photographing step of photographing the surrounding space, and a marker recognition step of recognizing a predetermined marker included in the photographed image photographed in the photographing step. and a movement control step of controlling movement by a movement mechanism based on the recognized marker.

(17) In order to solve the above-mentioned problem, the robot control program of the embodiment provides a computer with a photographing function for photographing the surrounding space and recognizing a predetermined marker included in a photographed image photographed in the photographing function. A marker recognition function and a movement control function for controlling movement by a movement mechanism based on the recognized marker are realized.

According to one embodiment, it is possible to provide a robot, a control method thereof, and a program that allow a user to control the movement of the robot.

1 is a block diagram showing an example of a software configuration of an autonomous robot in Embodiment 1. FIG. 1 is a block diagram showing an example of a hardware configuration of an autonomous robot in Embodiment 1. FIG. 2 is a flowchart illustrating an example of the operation of an autonomous robot control program in Embodiment 1. FIG. 7 is a flowchart showing another example of the operation of the autonomous robot control program in Embodiment 1. FIG. FIG. 3 is a diagram showing a method of setting a no-entry line in the first embodiment. 3 is a diagram showing an example of a display on a user terminal in Embodiment 1. FIG. 3 is a diagram showing an example of a display on a user terminal in Embodiment 1. FIG. 3 is a block diagram showing an example of a module configuration of a robot in Embodiment 2. FIG. FIG. 3 is a block diagram showing an example of a module configuration of a data providing device in Embodiment 2. FIG. 7 is a block diagram illustrating an example of a data configuration of an event information storage unit in Embodiment 2. FIG. 7 is a block diagram illustrating an example of a data configuration of a marker information storage unit in Embodiment 2. FIG. FIG. 12(A) is a flowchart showing the processing procedure in the marker registration phase of the second embodiment. FIG. 12(B) is a flowchart showing the processing procedure in the action phase of the second embodiment. FIG. 13(A) is a flowchart showing the processing procedure in the marker registration phase of the first embodiment. FIG. 13(B) is a flowchart showing the processing procedure in the action phase of the first embodiment. FIG. 14(A) is a flowchart showing the processing procedure in the marker registration phase of the second embodiment. FIG. 14(B) is a flowchart showing the processing procedure in the action phase of the second embodiment. FIG. 15(A) is a flowchart showing the processing procedure in the marker registration phase of the third embodiment. FIG. 15(B) is a flowchart showing the processing procedure in the action phase of the third embodiment. FIG. 16(A) is a flowchart showing the processing procedure in the marker registration phase of the fourth embodiment. FIG. 16(B) is a flowchart showing the processing procedure in the action phase of the fourth embodiment. FIG. 17(A) is a flowchart showing the processing procedure in the marker registration phase of the fifth embodiment. FIG. 17(B) is a flowchart showing the processing procedure in the action phase of the fifth embodiment. FIG. 18(A) is a flowchart showing the processing procedure in the marker registration phase of the sixth embodiment. FIG. 18(B) is a flowchart showing the processing procedure in the action phase of the sixth embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS An autonomous robot, a data providing device, and a data providing program according to embodiments of the present invention will be described in detail below with reference to the drawings.

[Embodiment 1]
First, the software configuration of the autonomous robot 1 will be explained using FIG. 1. FIG. 1 is a block diagram showing an example of the software configuration of an autonomous robot 1 according to an embodiment.

In FIG. 1, an autonomous robot 1 includes a data providing device 10 and a robot 2. The data providing device 10 and the robot 2 are connected through communication and function as an autonomous robot 1. The robot 2 is a mobile robot that includes a photographing section 21 , a marker recognition section 22 , a movement control section 23 , a state information acquisition section 24 , and a movement mechanism 29 . The data providing device 10 includes functional units such as a first communication control unit 11, a point cloud data generation unit 12, a spatial data generation unit 13, a visualization data generation unit 14, an imaging target recognition unit 15, and a second communication control unit 16. . The first communication control unit 11 includes a photographed image acquisition unit 111, a spatial data providing unit 112, and an instruction unit 113. The second communication control unit 16 includes functional units such as a visualization data providing unit 161 and a specification acquisition unit 162. The above-mentioned functional units of the data providing device 10 of the autonomous robot 1 in this embodiment will be described as functional modules realized by a data providing program (software) that controls the data providing device 10. Furthermore, each functional unit of the robot 2, including the marker recognition unit 22, movement control unit 23, and state information acquisition unit 24, will be described as a functional module realized by a program that controls the robot 2 in the autonomous robot 1. do.

The data providing device 10 is a device that can execute some of the functions of the autonomous robot 1, and is, for example, installed in a place physically close to the robot 2, communicates with the robot 2, and communicates with the robot 2. It is an edge server that distributes the processing load. In this embodiment, a case will be described in which the autonomous robot 1 is composed of a data providing device 10 and a robot 2, but the functions of the data providing device 10 are not included in the functions of the robot 2. Good too. Further, the robot 2 is a robot that can move based on spatial data, and is one aspect of the robot whose movement range is determined based on the spatial data. The data providing device 10 may be configured in one housing or may be configured in a plurality of housings.

The first communication control unit 11 controls communication functions with the robot 2. The communication method with the robot 2 is arbitrary, and for example, wireless LAN (Local Area Network), Bluetooth (registered trademark), short-range wireless communication such as infrared communication, or wired communication can be used. Each function of the photographed image acquisition section 111, the spatial data providing section 112, and the instruction section 113 that the first communication control section 11 has communicates with the robot 2 using the communication function controlled by the first communication control section 11.

The photographed image acquisition section 111 acquires a photographed image photographed by the photographing section 21 of the robot 2 . The photographing unit 21 is provided on the robot 2 and can change the photographing range as the robot 2 moves. Here, the photographing section 21, marker recognition section 22, movement control section 23, state information acquisition section 24, and movement mechanism 29 of the robot 2 will be explained.

The photographing unit 21 can be composed of one or more cameras. For example, when the photographing unit 21 is a stereo camera composed of two cameras, the photographing unit 21 can three-dimensionally photograph the spatial element to be photographed from different photographing angles. The photographing unit 21 is, for example, a video camera using an imaging device such as a CCD (Charge-Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. By photographing a spatial element with two cameras (stereo cameras), the shape of the spatial element can be measured. Furthermore, the photographing unit 21 may be a camera using ToF (Time of Flight) technology. In a ToF camera, the shape of a spatial element can be measured by irradiating the spatial element with modulated infrared light and measuring the distance to the spatial element. Further, the photographing unit 21 may be a camera using structured light. Structured lights are lights that project a striped or grid-like pattern of light onto a spatial element. The imaging unit 21 can measure the shape of the spatial element from the distortion of the projected pattern by photographing the spatial element from a different angle from the structured light. The photographing unit 21 may be any one of these cameras or a combination of two or more of these cameras.

Further, the imaging unit 21 is attached to the robot 2 and moves in accordance with the movement of the robot 2. However, the imaging unit 21 may be installed separately from the robot 2.

The photographed image photographed by the photographing section 21 is provided to the photographed image acquisition section 111 using a communication method compatible with the first communication control section 11 . The captured images are temporarily stored in the storage unit of the robot 2, and the captured image acquisition unit 111 acquires the temporarily stored captured images in real time or at predetermined communication intervals.

The marker recognition unit 22 recognizes a predetermined marker included in the photographed image taken by the photographing unit 21. A marker is a spatial element that indicates a restriction on movement of the robot 2. A marker is a shape, pattern, or color of an article that can be recognized from a photographed image, a character or figure attached to an article, or a combination thereof. By placing a marker at a position where the user restricts the movement of the robot 2, when the robot 2 photographs the space with the photographing unit 21, the robot 2 is photographed together with furniture and the like. The marker may be a flat article or a three-dimensional article. The marker is, for example, a sticker or paper on which a two-dimensional code or a specific color combination or shape is printed. Further, the marker may be an ornament or a rug of a specific color or shape. In this way, by using printed matter or objects around us as markers, the user does not need to secure a power source for the marker or secure a place to install it. Furthermore, the movement of the robot can be restricted at the user's will without spoiling the atmosphere of the room. Furthermore, since the user can visually recognize the marker, the user can intuitively understand the movement restriction range and can easily change the movement restriction range. The marker is installed by the user, for example, by pasting it on a wall or furniture, or placing it on the floor. The marker recognition unit 17 can recognize that the movement of the robot 2 is restricted by recognizing the marker image included in the photographed image.

Here, if the marker is flat, it can be attached to a wall or furniture, so it can be installed in a space-saving manner. If the marker is planar, if the plane of the marker is photographed from the horizontal direction (when the photographing angle is small), the marker in the photographed image will be distorted, making recognition difficult. On the other hand, if the plane of the marker is photographed from the vertical direction (when the photographing angle is large), the marker becomes easier to recognize. Therefore, for example, when a marker is pasted in a hallway, the robot 2 can be prevented from recognizing the marker because the photographing angle is small at a position far from the marker. As the robot moves along the hallway and approaches the marker, the imaging angle increases, allowing the marker to be recognized. For this reason, with flat markers, it is possible to bring the marker installation position (described later) closer to the position where the robot can recognize the marker, allowing the robot to accurately grasp the marker installation position. . Furthermore, if the marker is three-dimensional, it can be easily installed in the center of a room. When the marker is three-dimensional, the marker can be recognized from various photographing angles. Therefore, by installing a three-dimensional marker, it is possible to make the robot 2 located far from the marker installation position recognize the marker.

The marker recognition unit 22 stores the visual characteristics of markers in advance. For example, the marker recognition unit 22 stores in advance two-dimensional codes and three-dimensional objects to be recognized as markers. The marker recognition unit 22 may recognize an object registered in advance by the user as a marker. For example, if a user registers a flower pot photographed with the camera of the user terminal 3 as a marker, a flower pot installed in a hallway or the like can be recognized as a marker. Therefore, the user can install an object as a marker that does not feel strange in the place where the marker is installed. Note that the marker recognition unit 22 may recognize spatial elements other than objects as markers. For example, the marker recognition unit 22 may recognize a user's gesture, such as a user crossing his arms in front of his body, as a marker. The marker recognition unit 22 recognizes the position where the user made the gesture as the marker installation position.

The marker recognition unit 22 recognizes a position where a marker is attached or installed (hereinafter referred to as "installation position"). The installation position is the position in the space where the marker in the spatial data is installed. The installation position can be recognized, for example, based on spatial data recognized by the robot 2, based on the distance between the current position of the robot 2 and the photographed marker. For example, if the size of the marker is known in advance, the marker recognition unit 22 calculates the distance between the robot 2 and the marker from the size of the marker image included in the photographed image, and calculates the distance between the robot 2's current position and the photographing direction ( For example, the installation position of the marker can be recognized based on the direction determined by a compass (not shown). Furthermore, the installation position may be recognized from the relative position of the marker from a spatial element whose position in space is already known. For example, if the position of the door is already known, the marker recognition unit 22 may recognize the installation position from the relative position of the marker and the door. Further, if the photographed image is taken with a depth camera, the installation position can be recognized based on the photographing depth of the marker photographed with the depth camera.

Further, the marker recognition unit 22 may recognize a plurality of markers included in the photographed image. For example, if the user wants to set the range in which movement is restricted as a straight line, the user can set the first marker and the second marker.
A marker consisting of a pair of two markers can be installed. By recognizing the installation position (starting point) of the first marker and the installation position (end point) of the second marker, the marker recognition unit 22 can recognize the position of the line segment (straight line or curved line) connecting the starting point and the ending point. good. The marker recognition unit 22 can recognize the position of a line segment in the spatial data by mapping the positions of the first marker and the second marker to the spatial data. By placing a marker at a predetermined position, the user can easily set a line segment that restricts movement. Three or more markers may be installed. For example, when there are three or more markers, the marker recognition unit 22 can recognize a polygonal line or a polygon (area) based on the installation position of each marker.

The movement control unit 23 limits movement based on the marker installation position recognized by the marker recognition unit 22. The movement control unit 23 includes a restriction range setting unit 231 that sets a restriction range for restricting movement according to the recognized installation position of the marker. The movement control section 23 limits the movement of the robot 2 within the limit range set in the limit range setting section 231 . The marker installation position is a point, line, plane, or space that is set based on the installation position of one or more markers. The restricted range setting unit 231 can set the restricted range by recognizing the marker installation position as, for example, coordinate data in spatial data. The restricted range setting unit 231 may set a restricted range based on the installation position and restrict movement within the restricted range. For example, based on the installation position of one marker, the restriction range setting unit 231 sets a line segment that separates spatial elements such as a corridor, or a circular or spherical area in a space centered on the marker as a restriction range that restricts movement. Can be set. That is, the limit range setting unit 231 sets the limit range by arranging a geometrically determined range such as a rectangle, circle, or straight line in space based on the installation position of the marker. For example, if the range is set in a circular shape, the limit range setting unit 231 may set the limit range to be a circular range with a predetermined radius centered on the marker installation position. Further, if the range is set in a rectangular shape, the limit range setting unit 231 may determine the rectangular limit range by arranging the marker installation position at the center of one side of the rectangle. The restricted range is, for example, about 1 to 3 meters from the marker, which is narrower than the range in which the marker recognition unit 22 can recognize the marker. The restriction range may be predetermined for each marker, or may be arbitrarily adjusted by the user using an application to be described later.

Further, the limit range setting unit 231 may set a line, plane, or space set by a plurality of markers as a limit range. For example, the limit range setting unit 231 sets a predetermined range behind or around the marker installation position as the limit range based on the position of the robot 2 when the marker recognition unit 22 recognizes the marker. You may. When the limit range setting section 231 sets a linear limit range, the movement control section 23 limits the movement of the robot 2 so as not to exceed the line. In this way, the restricted range setting unit 231 may set the restricted range based on a predetermined rule with the marker installation position as a reference.

Further, the restricted range setting unit 231 may recognize the spatial characteristics around the marker and set the restricted range according to the spatial characteristics. In other words, the limit range setting unit 231 may recognize the floor plan and set the limit range according to the floor plan. For example, if the marker is located around the entrance of a passage (within a predetermined range), the restriction range setting unit 231 may set the passage as the restriction range. Furthermore, if a marker is placed in the center of the room (at a predetermined distance from the wall), the restriction range setting unit 231 may set a circular range centered on the marker as the restriction range. . Furthermore, if a marker is attached to a wall and there are no doors nearby, the restriction range setting unit 231 may set a predetermined range from the wall as the restriction range.

A type may be set for the marker. For example, there are markers that restrict movement only when the marker is visible (referred to as "temporary markers"), and markers that memorize the marker's position and permanently restrict movement even when the marker is not visible (referred to as "permanent markers"). ) as the marker type. When the permanent marker is visually recognized, the robot 2 stores the marker's position in a storage unit (not shown), and even if the marker disappears from that location, its movement is restricted based on the memorized marker position. . Further, when the temporary marker is visually recognized, the robot 2 does not memorize the position of the temporary marker, so if the temporary marker is removed, the restricted range is canceled.

The marker recognition unit 22 recognizes the type of marker that has been set. The types of markers can be classified in advance by, for example, the shape, pattern, color, text or figure of the marker, or a combination thereof. Further, the types of markers may be classified based on the number of markers installed, the method of installing the markers (for example, installing the markers in different vertical directions), or the like. When the marker includes a two-dimensional code, information specifying the type of marker is written in the two-dimensional code information. In this case, the marker recognition unit 22 can identify whether the marker is a temporary marker or a permanent marker by reading the two-dimensional code. Further, identification information that identifies the marker (referred to as "marker identification information") may be written in the two-dimensional code. In this case, the marker recognition unit 22 reads the marker identification information from the two-dimensional code, refers to a table prepared in advance, and identifies the type of marker associated with the marker identification information.

That is, the marker recognition unit 22 may be configured to read the attached information from the marker itself when the marker itself includes information like a two-dimensional code. Further, the marker recognition unit 22 may be configured to read marker identification information from a marker, and read accompanying information by referring to a table using the marker identification information as a key. In the present embodiment, the marker recognition unit 22 has a marker information storage unit (not shown) that holds additional information of the marker in association with marker identification information, and can identify the marker by referring to the marker information storage unit. This example shows a case where the configuration is such that additional information can be acquired for each item. The marker recognition unit 22 may read the marker identification information from a two-dimensional code, or may acquire the marker identification information by identifying the marker using general object recognition.

In this way, by configuring the information attached to the marker to be manageable, it is possible to not only set a restricted range around the marker and restrict the movement of the robot 2, but also to set various conditions according to the user's wishes. The actions of the robot 2 can be restricted (referred to as "action restrictions"). For example, if you do not want the robot 2 to enter the changing room during times when the bathroom is used, a time period during which entry is prohibited is associated with the marker. Furthermore, if you do not want the robot 2 to enter the kitchen while the kitchen is being used, a condition that prohibits the robot 2 from entering if there is a person (if a person is detected) is associated with the marker. In addition, even if robots are allowed to enter rather than prohibited, they must restrain their actions and be quiet within the restricted range, must not make noise, or must drive slowly. Instructions to restrict the robot's behavior, such as not to do so, may be associated as additional information. That is, the supplementary information may include information specifying the type of marker or information specifying the behavior of the robot within the restricted range. Information regulating behavior is information for restricting the robot's actions, and if movement within the restricted range is prohibited, it also includes information specifying the prohibited time period, etc. You may be If movement within the restricted range is conditionally permitted, the accompanying information may include information specifying the conditions (referred to as "behavior conditions") in addition to permission. .

If the marker recognition unit 22 does not recognize the marker, the movement control unit 23 may restrict movement based on the stored installation position. For example, if the marker command is to set a permanent marker that sets a permanent limit, the movement control unit 23 will control the movement based on the marker even if the marker has been removed and cannot be recognized from the captured image. be permanently restricted. Note that the marker set in the restricted range setting unit 231 may be edited such as deletion, position change, or command change based on an instruction from the user terminal 3, for example. For example, the user terminal 3 may include an unillustrated application program (hereinafter referred to as "application") that can edit markers. For example, the application may display markers in a selectable manner on the display screen of the user terminal 3 described above, and allow the user to edit the selected marker. Additionally, if the marker is a temporary marker, the app may allow the marker to be changed to a permanent marker. As a result, the user can remove the restricted range by removing the installed temporary marker, and by changing the temporary marker to a permanent marker in the app, even after the installed marker is removed. It becomes possible to maintain the restricted range. Note that the application may have a registration function that registers a spatial element photographed by the camera of the user terminal 3 described above as a marker. Further, the application may have a function of adjusting the restriction range or setting or changing the content of the above-mentioned action restriction. The application may have a function of connecting to the marker information storage section of the robot 2, and referring to and updating the action restrictions and action conditions for each marker.

Note that the restriction on the movement of the robot 2 set by the marker can coexist with the setting of a restriction range based on state information, which will be described later. For example, an area where entry to a hallway is prohibited may be set by installing a marker, and entry to a dressing room may be prohibited based on status information. In addition, it may be possible to set an area where movement is restricted using a marker, and to set the content of restrictions in the area (for example, conditions such as time during which entry is restricted) using status information.

The status information acquisition unit 24 acquires status information indicating the status of a destination during movement. The state information is information for restricting the movement of the robot 2 according to the state of the destination detected by the robot 2. The state of the destination includes, for example, the presence or absence of people, the presence or absence of pets, the temperature or humidity of the room, the locked state of the door, the lighting state of the lights, etc., and the state of the time, day of the week, weather, etc. May contain. The status information is, for example, information for restricting the movement speed in the area (range) when a person is detected in the movement area. Status information may also prohibit entry into an area, prohibit movement through a door when it is locked, or prohibit photography in an area where the lights are on, on a given day or time. It may be prohibited. Status information can be provided in conjunction with spatial data.

The data providing device 10 will be explained again. The spatial data providing section 112 provides the robot 2 with the spatial data generated by the spatial data generating section 13 . The spatial data is data of spatial elements recognized by the robot 2 in the space where the robot 2 exists. The robot 2 can move within the range defined by the spatial data. That is, the spatial data functions as a map for determining the movable range of the robot 2. The robot 2 is provided with spatial data from the spatial data providing section 112. For example, the spatial data can include positional data of spatial elements such as walls, furniture, electrical appliances, and steps that the robot 2 cannot move. Based on the provided spatial data, the robot 2 can determine whether or not the location is a place to which it can move. Further, the robot 2 may be configured to be able to recognize whether or not the spatial data includes an ungenerated range. Whether or not an ungenerated range is included can be determined by, for example, whether a part of the spatial data includes a space without a spatial element.

The instruction unit 113 instructs the robot 2 to take an image based on the spatial data generated by the spatial data generation unit 13. Since the spatial data generation unit 13 creates spatial data based on the captured image acquired by the captured image acquisition unit 111, for example, when creating indoor spatial data, spatial data is not created for areas that are not photographed. It may include parts of Furthermore, if the photographed image is unclear, noise may be included in the created spatial data, and the spatial data may include inaccurate portions. If there is an ungenerated portion in the spatial data, the instruction unit 113 may issue an instruction to photograph the ungenerated portion. Further, when the spatial data includes an inaccurate portion, the instruction unit 113 may issue an instruction to photograph the inaccurate portion. The instruction unit 113 may spontaneously instruct photography based on the spatial data. Note that the instruction unit 113 may instruct photography based on an explicit instruction from a user who has confirmed visualization data (described later) generated based on spatial data. The user can recognize the space and generate spatial data by specifying an area included in the visualization data and instructing the robot 2 to take a picture.

The point cloud data generation unit 12 generates three-dimensional point cloud data of a spatial element based on the captured image acquired by the captured image acquisition unit 111. The point cloud data generation unit 12 generates point cloud data by converting spatial elements included in a photographed image into a set of three-dimensional points in a predetermined space. As mentioned above, the spatial elements include the walls of the room, steps, doors, furniture placed in the room, home appliances, luggage, foliage plants, etc. Since the point cloud data generation unit 12 generates point cloud data based on the photographed image of the spatial element, the point cloud data represents the shape of the surface of the photographed spatial element. The photographed image is generated by the photographing unit 21 of the robot 2 photographing at a predetermined photographing position and at a predetermined photographing angle. Therefore, when the robot 2 photographs a spatial element such as furniture from the front position, point cloud data cannot be generated for the shape of the back side of the furniture, which is not photographed, and the robot 2 moves to the back side of the furniture. Even if there is a possible space, the robot 2 cannot recognize it. On the other hand, when the robot 2 moves and photographs furniture from a side photographing position, it is possible to generate point cloud data about the shape of the back side of spatial elements such as furniture, making it possible to accurately grasp the space.

The spatial data generating section 13 generates spatial data that defines the movable range of the robot 2 based on the point cloud data of the spatial elements generated by the point cloud data generating section 12 . Since spatial data is generated based on point group data in space, spatial elements included in the spatial data also have three-dimensional coordinate information. The coordinate information may include information on the position, length (including height), area, or volume of a point. The robot 2 can determine the movable range based on the position information of the spatial elements included in the generated spatial data. For example, if the robot 2 is equipped with a movement mechanism 29 that moves horizontally on the floor, the robot 2 is configured such that the height difference from the floor, which is a spatial element in the spatial data, is at least a predetermined height (for example, 1 cm or more). In some cases, it can be determined that movement is impossible. On the other hand, in the spatial data, if the spatial element such as a table top or a bed has a predetermined height from the floor, the robot 2 will be able to move to ) is determined as a movable range considering the clearance with the user's own height. In addition, in the spatial data, the robot 2 determines an area where the gap between the wall and the furniture, which is a spatial element, is a predetermined width or more (for example, 40 cm or more) as a movable range considering the clearance with the robot's own width. do.

The spatial data generation unit 13 may set attribute information for a predetermined area in the space. Attribute information is information that defines movement conditions for the robot 2 in a predetermined area. The movement condition is, for example, a condition that defines the clearance between the robot 2 and a spatial element in which the robot 2 can move. For example, if the normal movement condition under which the robot 2 can move is that the clearance is 30 cm or more, attribute information can be set such that the clearance for a predetermined area is 5 cm or more. Furthermore, the movement conditions set in the attribute information may include information that limits movement of the robot. Restrictions on movement include, for example, restrictions on movement speed or prohibition of entry. For example, attribute information may be set that reduces the movement speed of the robot 2 in areas where clearance is small or where people are present. Furthermore, the movement conditions set in the attribute information may be determined by the flooring material of the area. For example, the attribute information may set a change in the operation (traveling speed, traveling means, etc.) of the moving mechanism 29 when the floor is a cushion floor, a flowing floor, a tatami mat, or a carpet. In addition, the attribute information may include conditions such as a charging spot where the robot 2 can move to charge, a step or the edge of a carpet where movement is restricted because the robot 2's posture becomes unstable, etc. . Note that the area in which the attribute information is set may be made clear to the user by, for example, changing the display method in the visualization data described later.

The spatial data generation unit 13 performs, for example, Hough transform on the point cloud data generated in the point cloud data generation unit 12, extracts figures such as straight lines and curves that are common in the point cloud data, and uses the extracted figures to Spatial data is generated by the contour of the represented spatial element. Hough transformation is a coordinate transformation method that extracts a figure that passes through the feature points most often when point cloud data is used as feature points. Since point cloud data expresses the shape of spatial elements such as furniture in a room as a point cloud, users are required to determine what spatial elements are represented by point cloud data (for example, It may be difficult to recognize objects such as tables, chairs, walls, etc. The spatial data generation unit 13 can express the contours of furniture, etc. by performing Hough transform on the point cloud data, and therefore can make it easier for the user to identify spatial elements. Note that the spatial data generation unit 13 converts the point cloud data generated in the point cloud data generation unit 12 into basic shapes of spatial elements (for example, tables, chairs, walls, etc.) recognized in image recognition, and May also generate data. When a spatial element such as a table is recognized as a table through image recognition, the shape of the table can be determined from point cloud data of a part of the spatial element (for example, point cloud data when the table is viewed from the front). Can be predicted accurately. By combining point cloud data and image recognition, the spatial data generation unit 13 can generate spatial data in which spatial elements are accurately understood.

The spatial data generation unit 13 generates spatial data based on point group data included in a predetermined range from the position to which the robot 2 has moved. The predetermined range from the position to which the robot 2 has moved includes the position to which the robot 2 has actually moved, and may be, for example, a range at a distance of 30 cm from the position to which the robot 2 has moved. Since the point cloud data is generated based on a photographed image photographed by the photographing unit 21 of the robot 2, the photographed image may include a spatial element located away from the robot 2. If the spatial element is far from the imaging unit 21, there may be a portion that is not photographed or a range in which the robot 2 cannot actually move due to the presence of an obstacle that is not photographed. Furthermore, if a spatial element located far from the photographing unit 21, such as a hallway, is included in the photographed image, the spatial element extracted at the feature point may be distorted. Furthermore, when the photographing distance is long, the spatial elements included in the photographed image become small, so the accuracy of the point cloud data may decrease. The spatial data generation unit 13 may generate spatial data that does not include spatial elements with low accuracy or distorted spatial elements by ignoring feature points that are far apart. The spatial data generation unit 13 generates spatial data by deleting point cloud data outside a predetermined range from the position where the robot 2 has moved, thereby preventing the occurrence of enclaves where no data actually exists. , it is possible to generate spatial data that does not include the range in which the robot 2 cannot move and has high data accuracy. Moreover, it is possible to prevent enclaves from being drawn in visualization data generated from spatial data, and visibility can be improved.

When the marker recognition unit 22 recognizes the marker, the spatial data generation unit 13 sets a restriction range for the generated spatial data. By setting a restricted range for spatial data, it becomes possible to visualize the restricted range as part of the visualization data. Furthermore, when the status information is acquired by the status information acquisition unit 24, the spatial data generation unit 13 sets status information for the spatial data. By setting state information to spatial data, it becomes possible to make the state information a part of visualization data.

The visualization data generation unit 14 generates visualization data based on the spatial data generated by the spatial data generation unit 13 in a manner that allows a person to intuitively identify spatial elements included in the space.

Generally, robots have various sensors such as cameras and microphones, and recognize the surrounding situation by comprehensively judging information obtained from these sensors. In order for a robot to move, it is necessary to recognize various objects existing in space and determine a movement route based on spatial data, but the movement route may not be appropriate because the robot cannot correctly recognize the objects. For example, even if a person thinks there is a large enough space, the robot may think that there are obstacles and that it can only move within a narrow range. When a discrepancy in recognition occurs between humans and robots, the robot may behave in ways that are contrary to human expectations, and humans feel stressed. In order to reduce the discrepancy between recognition between humans and robots, the autonomous behavioral robot in this embodiment visualizes spatial data representing its own recognition state and provides it to humans, and also re-visits points pointed out by humans. Recognition processing can be performed.

Spatial data is data that includes spatial elements that the autonomous robot 1 recognizes, whereas visualization data is data that allows the user to visually recognize the spatial elements that the autonomous robot 1 recognizes. It is data. Spatial data may include spatial elements that are misrecognized. By visualizing the spatial data, it becomes easier for a person to check the recognition state of spatial elements in the autonomous robot 1 (presence or absence of misrecognition, etc.).

Visualized data is data that can be displayed on a display device. The visualization data is a so-called floor plan, and includes spatial elements recognized as tables, chairs, sofas, etc. in an area surrounded by spatial elements recognized as walls. The visualization data generation unit 14 generates the shape of furniture or the like formed in the figure extracted by the Hough transform as visualization data expressed, for example, in RGB data. The spatial data generation unit 13 generates visualization data in which the plane drawing method is changed based on the direction of the three-dimensional plane of the spatial element. The direction of a three-dimensional plane of a spatial element is, for example, the normal direction of a plane formed by a figure generated in the point cloud data by Hough-transforming the point cloud data generated in the point cloud data generation unit 12. It is. The visualization data generation unit 14 generates visualization data in which the plane drawing method is changed according to the normal direction. The drawing method is, for example, a color attribute such as hue, brightness, or saturation applied to a plane, a pattern or texture applied to a plane, or the like. For example, when the normal to the plane is in the vertical direction (the plane is in the horizontal direction), the visualization data generation unit 14 increases the brightness of the plane and draws it in a bright color. On the other hand, when the normal to the plane is in the horizontal direction (the plane is in the vertical direction), the visualization data generation unit 14 lowers the brightness of the plane and draws it in a dark color. By changing the plane drawing method, it becomes possible to represent the shape of furniture, etc. three-dimensionally, and it is possible to make it easier for the user to confirm the shape of the furniture, etc. Furthermore, the visualization data may include coordinate information in the visualization data (referred to as "visualization coordinate information") that is associated with coordinate information of each spatial element included in the spatial data. Since the visualized coordinate information is associated with coordinate information, a point in the visualized coordinate information corresponds to a point in the actual space, and a surface in the visualized coordinate information corresponds to a surface in the actual space. Therefore, when the user specifies the position of a certain point in the visualization data, the corresponding position of the point in the actual room can be specified. Further, a conversion function for converting the coordinate system may be prepared so that the coordinate system in the visualization data and the coordinate system in the spatial data can be mutually converted. Of course, the coordinate system in the visualized data and the coordinate system in the actual space may be mutually convertible.

The visualization data generation unit 14 generates visualization data as three-dimensional (3D (Dimensions)) data. Further, the visualization data generation unit 14 may generate the visualization data as two-dimensional (2D) data. By generating visualization data in 3D, users can easily confirm the shape of furniture, etc. The visualization data generation unit 14 may generate visualization data in 3D when sufficient data is generated in the spatial data generation unit 13 to generate visualization data in 3D. The visualization data generation unit 14 may generate visualization data in 3D based on the 3D viewpoint position (viewpoint height, viewpoint elevation/depression angle, etc.) specified by the user. By making it possible to specify the viewpoint position, it is possible for the user to easily confirm the shape of furniture, etc. Furthermore, the visualization data generation unit 14 may generate visualization data in which only the back wall of the room is colored, and the front wall or ceiling is made transparent (not colored). By making the wall on the near side transparent, it is possible for the user to easily check the shape of furniture, etc. placed beyond the wall on the near side (in the room).

The visualization data generation unit 14 generates visualization data to which color attributes are added according to the captured image acquired by the captured image acquisition unit 111. For example, if the captured image includes woodgrain furniture and the color of the woodgrain (for example, brown) is detected, the visualization data generation unit 14 assigns a color similar to the detected color to the extracted furniture figure. Generate visualization data. By assigning a color attribute according to the photographed image, it is possible for the user to easily confirm the type of furniture or the like.

The visualization data generation unit 14 generates visualization data in which a method for drawing a fixed object and a moving object is changed. Fixed objects include, for example, a wall in a room, a step, fixed furniture, and the like. Examples of moving objects include chairs, trash cans, and furniture with casters. Furthermore, the moving objects may include, for example, temporary objects such as luggage and bags that are temporarily placed on the floor. The drawing method is, for example, a color attribute such as hue, brightness, or saturation applied to a plane, a pattern or texture applied to a plane, or the like.

The categories of fixed objects, moving objects, and temporary objects can be distinguished by the period of time they have existed at a particular location. For example, the spatial data generation unit 13 identifies categories of fixed objects, moving objects, or temporary objects as spatial elements based on changes over time in the point cloud data generated by the point cloud data generation unit 12, and generates spatial data. generate. For example, based on the difference between the spatial data generated at the first time and the spatial data generated at the second time, the spatial data generation unit 13 determines that the spatial element is a fixed object when the spatial element has not changed. to decide. Furthermore, the spatial data generation unit 13 may determine that the spatial element is a moving object when the position of the spatial element has changed based on the difference in the spatial data. Further, the spatial data generation unit 13 may determine that the spatial element is a primary object based on the difference in the spatial data when the spatial element disappears or appears. The visualization data generation unit 14 changes the drawing method based on the classification identified by the spatial data generation unit 13. Changes in the drawing method include, for example, color coding, addition of hatching, or addition of predetermined marks. For example, the spatial data generation unit 13 may display fixed objects in black, moving objects in blue, or temporary objects in yellow. The spatial data generation unit 13 generates spatial data by identifying categories of fixed objects, moving objects, and temporary objects. The visualization data generation unit 14 may generate visualization data in which the drawing method is changed based on the classification identified by the spatial data generation unit 13. Further, the spatial data generation unit 13 may generate visualization data in which the drawing method of spatial elements recognized through image recognition is changed.

The visualization data generation unit 14 can generate visualization data in a plurality of divided areas. For example, the visualization data generation unit 14 generates visualization data for each space partitioned by walls, such as a living room, a bedroom, a dining room, and a hallway, as one room. By generating visualization data for each room, for example, it becomes possible to generate spatial data or visualization data separately for each room, making it easier to generate spatial data and the like. Furthermore, it becomes possible to create spatial data etc. only for areas where the robot 2 may move. The visualization data providing unit 161 provides visualization data that allows the user to select an area. The visualization data providing unit 161 may, for example, enlarge the visualization data of the area selected by the user or provide detailed visualization data of the area selected by the user.

The photographic object recognition section 15 performs image recognition of spatial elements based on the photographed image acquired by the photographed image acquisition section. Recognition of a spatial element can be performed, for example, by using an image recognition engine that determines what a spatial element is based on accumulated image recognition results in machine learning. Image recognition of a spatial element can be performed by recognizing, for example, the shape, color, pattern, or character or figure attached to the spatial element. The photographic object recognition unit 15 may be able to image-recognize spatial elements by using, for example, an image recognition service provided by a cloud server (not shown). The visualization data generation unit 14 generates visualization data in which the drawing method is changed according to the spatial element image-recognized by the imaging target recognition unit 15. For example, if the image-recognized spatial element is a sofa, the visualization data generation unit 14 generates visualization data in which the spatial element is given a texture that has the feel of cloth. Further, when the image-recognized spatial element is a wall, the visualization data generation unit 14 may generate visualization data to which the color attribute (for example, white) of wallpaper is added. By performing such visualization processing, the user can intuitively grasp the spatial recognition state of the robot 2.

The second communication control unit 16 controls communication with the user terminal 3 owned by the user. The user terminal 3 is, for example, a smartphone, a tablet PC, a notebook PC, a desktop PC, or the like. The communication method with the user terminal 3 is arbitrary, and for example, wireless LAN, Bluetooth (registered trademark), short-range wireless communication such as infrared communication, or wired communication can be used. Each function of the visualized data providing unit 161 and the specification acquisition unit 162 that the second communication control unit 16 has communicates with the user terminal 3 using communication functions controlled by the second communication control unit 16.

The visualized data providing unit 161 provides the visualized data generated by the visualized data generating unit 14 to the user terminal 3. The visualized data providing unit 161 is, for example, a web server, and provides visualized data to the browser of the user terminal 3 as a web page. The visualization data providing unit 161 may provide visualization data to a plurality of user terminals 3. By viewing the visualization data displayed on the user terminal 3, the user can confirm the movable range of the robot 2 as a 2D or 3D display. In the visualization data, shapes of furniture and the like are drawn using a predetermined drawing method. By operating the user terminal 3, the user can, for example, switch between 2D display and 3D display, zoom in or out of visualized data, or move the viewpoint in 3D display.

The user can visually check the visualization data displayed on the user terminal 3 and check the generation state of the spatial data and the attribute information of the area. The user can specify an area in the visualized data for which spatial data has not been generated and instruct the creation of spatial data. In addition, the user visually checks the visualization data displayed on the user terminal 3, and if there is an area where the spatial data seems to be inaccurate, such as an unnatural shape of a spatial element such as furniture, You can specify a region and instruct the regeneration of spatial data. As mentioned above, the visualization coordinate information in the visualization data is associated with the coordinate information of the spatial data, so the area in the visualization data that is specified to be regenerated by the user can be uniquely identified as the area in the spatial data. . It is mapped to the coordinate information of The regenerated spatial data is regenerated into visualization data by the visualization data generation unit 14 and provided from the visualization data provision unit 161 . Note that even in the regenerated visualization data, the generation state of the spatial data may not change, such as when spatial elements are misrecognized. In that case, the user may instruct the generation of spatial data by changing the operating parameters of the robot 2. The operating parameters include, for example, the photographing conditions (exposure amount, shutter speed, etc.) in the photographing unit 21 of the robot 2, the sensitivity of a sensor (not shown), the clearance conditions when allowing the robot 2 to move, and the like. The operational parameters may be included in the spatial data, for example, as area attribute information.

The visualization data generation unit 14 generates visualization data including, for example, a button display for instructing creation of spatial data (including "re-creation"). The user terminal 3 can transmit an instruction to create spatial data to the autonomous robot 1 by the user operating a displayed button. The spatial data creation instruction transmitted from the user terminal 3 is acquired by the designation acquisition unit 162.

The designation acquisition unit 162 acquires an instruction to create spatial data of a region designated by the user based on the visualization data provided by the visualization data provision unit 161. The designation acquisition unit 162 may acquire an instruction to set (including change) attribute information of an area. Further, the designation acquisition unit 162 acquires the position of the area and the direction in which the robot approaches the area, that is, the direction in which the image should be photographed. The creation instruction can be obtained, for example, by operating a web page provided by the visualization data providing unit 161. Thereby, the user can understand how the robot 2 recognizes the space and can instruct the robot 2 to redo the recognition process depending on the recognition state.

The instruction unit 113 instructs the robot 2 to take an image in a region for which creation of spatial data has been instructed. The instruction unit 113 may instruct the imaging of a marker placed in the area. Photographing in the area for which creation of spatial data is instructed may include, for example, photographing conditions such as the coordinate position of the robot 2 (photographing unit 21), the photographing direction of the photographing unit 21, and the resolution. If the spatial data that has been instructed to be created relates to an area that has not yet been created, the spatial data generator 13 adds the newly created spatial data to the existing spatial data; If the specified spatial data is related to re-creation, spatial data is generated by updating the existing spatial data. Further, if the photographed image includes a marker, spatial data including the recognized marker may be generated.

As mentioned above, in FIG. 1, the autonomous robot 1 is configured with the data providing device 10 and the robot 2, but the functions of the data providing device 10 are included in the functions of the robot 2. It may be something. For example, the robot 2 may include all the functions of the data providing device 10. The data providing device 10 may temporarily replace the function when the robot 2 lacks processing capacity, for example.

Furthermore, in this embodiment, "acquisition" may be something that is actively acquired by the acquiring entity, or it may be something that is passively acquired by the acquiring entity. For example, the designation acquisition unit 162 may acquire the spatial data by receiving an instruction to create spatial data transmitted from the user terminal 3 by the user, or the designation acquisition unit 162 may acquire the spatial data by receiving the instruction to create the spatial data transmitted by the user from the user terminal 3. The instructions for creating the spatial data may be obtained by reading them from the storage area.

The data providing device 10 also includes a first communication control unit 11, a point cloud data generation unit 12, a spatial data generation unit 13, a visualization data generation unit 14, a photographic target recognition unit 15, a second communication control unit 16, and a photographed image. The functional units of the acquisition unit 111, the spatial data provision unit 112, the instruction unit 113, the visualization data provision unit 161, and the specification acquisition unit 162 are examples of the functions of the autonomous robot 1 in this embodiment. This does not limit the functions that the autonomous robot 1 has. For example, the autonomous robot 1 does not need to have all the functional units that the data providing device 10 has, and may have some of the functional units. Furthermore, the autonomous robot 1 may have functional units other than those described above. Further, each functional unit of the robot 2, including the marker recognition unit 22, the movement control unit 23, the limit range setting unit 231, and the state information acquisition unit 24, represents an example of the functions of the autonomous robot 1 in this embodiment. This does not limit the functions that the autonomous robot 1 has. For example, the autonomous robot 1 does not need to have all the functional parts that the robot 2 has, and may have some of the functional parts.

Further, each of the above-mentioned functional units included in the autonomous robot 1 has been described as being realized by software, as described above. However, at least one or more of the above functions possessed by the autonomous robot 1 may be realized by hardware.

Moreover, any of the above-mentioned functions that the autonomous robot 1 has may be implemented by dividing one function into a plurality of functions. Moreover, any two or more of the above-mentioned functions that the autonomous robot 1 has may be combined into one function and implemented. That is, FIG. 1 expresses the functions of the autonomous robot 1 using functional blocks, and does not indicate that each function is constituted by a separate program file, for example.

Moreover, the autonomous robot 1 may be a device realized by one housing, or a system realized by a plurality of devices connected via a network or the like. For example, the autonomous robot 1 may realize some or all of its functions using a virtual device such as a cloud service provided by a cloud computing system. That is, the autonomous robot 1 may realize at least one or more of the above functions in another device. Further, the autonomous robot 1 may be a general-purpose computer such as a tablet PC, or may be a dedicated device with limited functions.

Further, the autonomous robot 1 may realize some or all of its functions in the robot 2 or the user terminal 3.

Next, the hardware configuration of the autonomous robot 1 (control unit of the robot 2) will be described using FIG. 2. FIG. 2 is a block diagram showing an example of the hardware configuration of the autonomous robot 1 in the embodiment.

The autonomous robot 1 includes a CPU (Central Processing Unit) 101, a RAM (Random Access Memory) 102, a ROM (Read Only Memory) 103, a touch panel 104, a communication I/F (Interface) 105, a sensor 106, and has a clock 107 . The autonomous robot 1 is a device that executes the autonomous robot control program described in FIG.

The CPU 101 controls the autonomous robot 1 by executing an autonomous robot control program stored in the RAM 102 or ROM 103. The autonomous robot control program is acquired from, for example, a recording medium that records the autonomous robot control program, or a program distribution server via a network, installed in the ROM 103, and read and executed by the CPU 101.

The touch panel 104 has an operation input function and a display function (operation display function). The touch panel 104 allows the user of the autonomous robot 1 to input operations using a fingertip, a touch pen, or the like. The autonomous robot 1 in this embodiment uses a touch panel 104 having an operation display function. However, the autonomous robot 1 has a display device having a display function and an operation input device having an operation input function separately. It may be something that you have. In that case, the display screen of the touch panel 104 can be performed as a display screen of a display device, and the operation of the touch panel 104 can be performed as an operation of an operation input device. Note that the touch panel 104 may be realized in various forms such as a head-mounted display, a glasses-type display, a wristwatch-type display, and the like.

Communication I/F 105 is a communication I/F. The communication I/F 105 performs short-range wireless communication such as wireless LAN, wired LAN, and infrared rays, for example. In FIG. 2, only the communication I/F 105 is shown as the communication I/F, but the autonomous robot 1 may have communication I/Fs for each of a plurality of communication methods. The communication I/F 105 may communicate with a control unit that controls the photographing unit 21 or a control unit that controls the moving mechanism 29 (not shown).

The sensor 106 is a camera of the photographing unit 21, hardware such as a TOF or thermo camera, a microphone, a thermometer, an illumination meter, or a proximity sensor. Data acquired by these hardware is stored in RAM 102 and processed by CPU 101.

Clock 107 is an internal clock for acquiring time information. The time information acquired by the clock 107 is used, for example, to confirm the time period during which entry is prohibited.

Next, the operation of the robot control program related to providing visualization data will be described using FIG. 3. FIG. 3 is a flowchart showing an example of the operation of the robot control program in the embodiment. In the explanation of the flowcharts below, it will be assumed that the autonomous robot 1 is the main entity that executes the operations, but each operation is executed by each function of the autonomous robot 1 described above.

In FIG. 3, the autonomous robot 1 determines whether a photographed image has been acquired (step S11). Whether or not a photographed image has been acquired can be determined based on whether or not the photographed image acquisition unit 111 has acquired a photographed image from the robot 2. The determination as to whether or not a photographed image has been acquired is made in units of processing of the photographed image. For example, if the captured image is a video, the video is continuously transmitted from the robot 2, so the determination as to whether or not the captured image has been obtained is based on whether the number of frames or data amount of the captured video is a predetermined value. This can be done depending on whether or not it has been reached. The captured image may be acquired by the mobile robot primarily transmitting the captured image, or by the captured image acquisition unit 111 primarily receiving the captured image from the mobile robot. If it is determined that the photographed image has not been acquired (step S11: NO), the autonomous robot 1 repeats the process of step S11 and waits for the photographed image to be acquired.

On the other hand, if it is determined that a captured image has been acquired (step S12: NO), the autonomous robot 1 generates point cloud data (step S12). The point cloud data is generated by the point cloud data generation unit 12, for example, by detecting points with large changes in brightness in the photographed image as feature points and giving three-dimensional coordinates to the detected feature points. can be executed. To detect feature points, for example, differential processing may be performed on the captured image to detect portions with large gradation changes. Furthermore, assigning coordinates to feature points may be performed by detecting the same feature point photographed from different photographing angles. The determination as to whether or not photographed images have been acquired in step S11 can be made based on whether or not photographed images photographed from a plurality of directions have been acquired.

After executing the process in step S12, the autonomous robot 1 generates spatial data (step S13). The spatial data generation unit 13 can generate the spatial data by, for example, performing Hough transform on the point cloud data. Note that details of step S13 will be explained with reference to FIG.

After executing the process in step S13, the autonomous robot 1 provides the generated spatial data to the robot 2 (step S14). The spatial data may be provided to the robot 2 sequentially each time the spatial data is generated, as shown in FIG. 3, or may be provided asynchronously with the processing shown in steps S11 to S18. good. The robot 2 provided with the spatial data can grasp the movable range based on the spatial data.

After executing the process in step S14, the autonomous robot 1 determines whether or not it recognizes a spatial element (step S15). The determination as to whether or not to recognize a spatial element can be performed, for example, by setting the imaging target recognition unit 15 as to whether or not to recognize a spatial element. Note that even if it is determined that the spatial element is recognized, if recognition fails, it may be determined that the spatial element is not recognized.

If it is determined that the spatial element is recognized (step S15: YES), the autonomous robot 1 generates first visualization data (step S16). Generation of the first visualization data can be executed by the visualization data generation section 14. The first visualization data is visualization data generated after the imaging target recognition unit 15 recognizes a spatial element. For example, when the photographing object recognition unit 15 determines that the spatial element is a table, the visualization data generation unit 14 can detect the table even if the top surface of the table is not photographed and does not have point cloud data. Visualization data can be generated assuming that the top surface is flat. Further, when it is determined that the spatial element is a wall, the visualization data generation unit 14 can generate visualization data by assuming that the portion that is not photographed is also a plane.

If it is determined that the spatial element is not recognized (step S15: NO), the autonomous robot 1 generates second visualization data (step S17). Generation of the second visualization data can be executed by the visualization data generation unit 14. The second visualization data is visualization data that is generated without the photographing target recognition unit 15 recognizing spatial elements, that is, based on point cloud data and spatial data generated from the photographed image. The autonomous robot 1 can reduce the processing load by not performing spatial element recognition processing.

After executing the process of step S16 or the process of step S17, the autonomous robot 1 provides visualization data (step S18). The visualization data is provided by the visualization data providing unit 161 providing the user terminal 3 with the visualization data generated by the visualization data generating unit 14 . The autonomous robot 1 may generate and provide visualization data in response to a request from the user terminal 3, for example. After executing the process of step S18, the autonomous robot 1 ends the operation shown in the flowchart.

Next, operations related to spatial data generation of the robot control program will be explained using FIG. 4. FIG. 4 is a flowchart showing another example of the operation of the robot control program in the embodiment.

In FIG. 4, the autonomous robot 1 generates spatial data (step S121). The spatial data generation unit 13 can generate the spatial data by, for example, performing Hough transform on the point cloud data. After executing step S131, the autonomous robot 1 determines whether the marker has been recognized (step S122). Whether or not the marker has been recognized can be determined by whether the marker recognition section 22 has recognized the marker image in the photographed image photographed by the photographing section 21. The robot 2 can notify the data providing device 10 of the marker recognition results.

If it is determined that the marker has been recognized (step S122: YES), the autonomous robot 1 sets a restriction range within which movement is restricted in the spatial data generated in step S121 (step S123).

If it is determined that the marker is not recognized (step S122: NO), the autonomous robot 1 determines whether state information has been acquired (step S124). Whether or not the status information has been acquired can be determined based on whether the status information is acquired by the status information acquisition unit 24. If it is determined that the state information has been acquired (step S124: YES), the autonomous robot 1 sets the acquired state information in correspondence with the spatial data (step S125). Note that the set status information is provided by the visualization data providing unit 161 in correspondence with the visualization data.

On the other hand, if it is determined that the state information has not been acquired (step S124: NO), after executing the process of step S123 or after executing the process of step S125, the autonomous robot 1 performs the steps shown in the flowchart. The operation of generating provided data in S12 ends.

Note that the execution order of the processing in each step in the operation of the robot control program (robot control method) described in this embodiment is not limited.

Next, a method of setting a no-entry line by installing a pair of markers will be explained using FIG. 5. FIG. 5 is a diagram showing a method of setting a no-entry line in the embodiment.

<Installation of a no-entry line into the aisle using paired markers>
In FIG. 5, it is assumed that passages (entrances and exits) through which the robot 2 can move exist between walls 1 and 2 and between walls 1 and 3. Between walls 1 and 2, the user places a marker 1a on the wall 1 side as a first marker, and further places a marker 1b on the wall 2 side as a second marker. The marker recognition unit 22 recognizes the marker 1a and the marker 1b as a pair of markers. The restricted range setting unit 231 sets a line connecting the marker 1a and the marker 1b with a straight line as the no-entry line 1. By setting the entry prohibition line 1, the user can restrict the robot 2 from moving beyond the entry prohibition line 1.

<Installation of a no-entry line in the aisle using a single marker>
In the passage between the walls 1 and 3, the user places the marker 2 on the wall 1 side near the passage. Marker 2 is recognized by the marker recognition unit 22 as a single marker. The limit range setting unit 231 checks whether there is a passage near the marker 2 or not. If there is a passage, the restriction range setting unit 231 sets a straight line on the passage as a no-entry line 2 based on the installation position of the marker 2 and the position of the passage. Since the restriction range setting unit 231 can confirm whether there is a passage near the marker 2, the user can prohibit the robot from entering the passage even with a single marker.

<Establishment of the first no-entry area using a single marker>
The user pastes the marker 3 on the wall 3. Marker 3 is recognized by the marker recognition unit 22 as a single marker. The limit range setting unit 231 checks whether there is a passage near the marker 3 or not. If there is no passage, the restricted range setting unit 231 sets a predetermined range from the installation position of the marker 2 (for example, a semicircle centered on the installation position of the marker 3) as the first no-entry area.

<Establishment of a second no-entry area using a single marker>
The user places the marker 4 in the center of the room. The marker 4 is, for example, a three-dimensional marker. The marker recognition unit 22 recognizes the marker 4 as a single marker. The restricted range setting unit 231 sets a predetermined range around the marker 4 (for example, a circle centered on the installation position of the marker 4) as a second no-entry area.

Next, the setting of the limit range in the user terminal 3 provided by the autonomous robot 1 will be explained using FIGS. 6 to 7. 6 to 7 are diagrams showing examples of displays on the user terminal 3 in the embodiment. 6 to 7 are display examples in which a web page provided as visualization data from the visualization data providing unit 161 is displayed on a touch panel of a smartphone exemplified as the user terminal 3.

In FIG. 6, the user terminal 3 displays 2D display data generated by the visualization data generation unit 14 based on the captured image of the living room captured by the robot 2. The visualization data generation unit 14 rasterizes line segments (straight lines or curves) extracted by Hough transform of the point cloud data, and draws boundaries of spatial elements such as walls and furniture in 2D. The visualization data generation unit 14 displays the no-entry line 36 based on the pair of marker images recognized by the marker recognition unit 22 or the status information acquired by the status information acquisition unit 24. The no-entry line 36 can be set by specifying a starting point and an ending point. The starting point and ending point can be set by setting a pair of markers or by setting from the user terminal 3. An entry prohibition mark is displayed in the center of the entry prohibition line 36. When the entry prohibition mark is pressed, a delete button 37 is displayed. By pressing the delete button 37, the once set entry prohibition line 36 can be deleted. Note that the illustrated house-shaped icon h represents the home position where the robot 2 returns to its home for charging.

In FIG. 6, the area to the right of the no-entry line 36 is an area where spatial data has not yet been created. The user can set, confirm, or delete the no-entry line 36 from the visualization data displayed on the user terminal 3. That is, the autonomous robot 1 can generate a visualized map that defines the movable range of the robot 2 from the image taken by the photographing unit 21, and also mark the user terminal 3 as a prohibited area where the robot 2 cannot enter. Allows you to configure from Note that the user terminal 3 may be configured to be able to set conditions for restricting the movement of the robot 2. For example, the user terminal 3 allows the user to set the time period during which the robot 2 is prohibited from entering, the content of restrictions on movement based on the presence or absence of people, the state of lighting when restricting movement, etc. Good too. For example, the user terminal 3 sets conditions such as prohibiting people from entering the kitchen during morning and evening hours when people are preparing meals, or prohibiting people from entering the study when the lights are off. It may be possible to do so.

In FIG. 7, the user terminal 3 displays 2D visualization data generated by the visualization data generation unit 14 based on the captured image of the living room captured by the robot 2, as in FIG. Visualization data of the Western-style room is displayed on the right side of the no-entry line 36, and it can be confirmed that the robot 2 is prohibited from entering the Western-style room 38 due to the no-entry line 36. For example, after the spatial data of the Western-style room 38 is generated based on the photographed image of the Western-style room 38, there is a case where it is desired to set the Western-style room 38 to be prohibited from entering. The user can set the Western-style room 38 as a no-entry area by placing a marker at the entrance of the Western-style room 38 to prohibit the pair from entering, or by setting a no-entry line 36 from the user terminal 3. It becomes possible. Furthermore, since spatial data has already been generated for the Western-style room 38, the user can confirm that the robot 2 can be moved to the Western-style room 38 by deleting the no-entry line 36. Note that the user can enlarge or reduce the display by pinching in or pinching out the touch panel of the user terminal 3.

As a modification of the first embodiment, for example, the visualization data providing unit 161 in FIG. 1 may provide the user terminal 3 with the original image along with the visualization data. For example, when the user specifies a part of the floor plan displayed on the user terminal 3, an image taken of that part may be displayed. That is, images used for specifying each spatial element are accumulated, and when a user specifies a spatial element, an image associated with that spatial element is provided. This allows the user to use the image to determine the recognition state when the recognition state cannot be determined from the visualized data.

As yet another modification, the method of specifying the restricted area where entry is prohibited as explained using FIG. 6 may be performed by sliding the fingertip in a circular motion on the touch panel screen of the user terminal 3. , specify by moving your fingertips to enclose the restricted range. In conjunction with this operation, the trajectory of the fingertip is drawn on the screen, and the trajectory of the fingertip is visualized overlapping the visualization data.

Furthermore, if the marker is planar, the marker may not be recognized depending on the photographing angle of the marker as described above. For example, in the case of a road sign, the road sign is installed approximately perpendicular to the direction of travel so that the driver can easily see the road sign. However, markers may be pasted on walls along the route that the robot 2 moves, and the markers may be overlooked depending on the shooting direction of the camera. For example, since the marker 2 shown in FIG. 5 is pasted on the wall 1, if the robot 2 moves toward the marker 3 along the wall 3, it may enter the passageway before checking the marker. . Therefore, as a modification of the present embodiment, when the robot 2 discovers a space into which it can enter (for example, a passage provided on a wall), it actively checks whether or not a marker is installed near the entrance of the space. Perform a confirmation operation. For example, if the entrance to the space is a passageway between walls, the marker may be affixed to the wall near the entrance to the passageway. Robot 2 moves to a position where it is easy to see markers that may be installed on the wall at the entrance of the passageway, for example, to a position in front of the passageway, and performs an active checking operation of photographing the wall, thereby making sure that the marker is not overlooked. This makes it possible to prevent Note that the active marker confirmation operation may be performed when an area with different environmental conditions is discovered or when an area is entered. The environmental conditions are, for example, the illuminance, temperature, humidity, or noise level of the space in which the robot 2 moves, and changes in the environmental conditions may include changes in spatial elements such as the color of the walls.

Further, the restricted range setting unit 231 may set the restricted range based on feature points around the marker, instead of the spatial data of the installation position where the marker is installed. The surrounding feature points are, for example, spatial elements such as cables placed on the floor and steps. By learning the feature points where markers are installed, it is possible to obtain the learning effect of restricting movement even in locations with similar feature points where markers are not installed.

Furthermore, if there are multiple robots, the content of movement restrictions may be set differently for each robot. For example, when the same marker is recognized, the prohibited areas restricted by the robot may be set differently. By setting different restrictions for each robot, it is possible to set restrictions according to the purpose of the robot (for example, a cleaning robot, an alarm clock robot, etc.).

Further, the robot may learn the contents of the restrictions once set by the marker. For example, by learning that a no-entry area has been set by a temporary marker, the robot may reduce the frequency of entry even after the marker is removed. To achieve this, the marker recognition unit 22 stores the positions of temporary markers and permanent markers in association with information specifying the marker type. In this way, by remembering the position of the temporary marker, the robot can take actions such as hesitating to enter the area where the temporary marker was placed or reducing the frequency of entering the area. It becomes possible to execute.

[Embodiment 2]
In the first embodiment, an example is shown in which a marker is used to make the autonomous robot 1 recognize a place where entry is prohibited, but a marker may be used to make the autonomous robot 1 recognize an arbitrary place. That is, a marker may be installed at an arbitrary location in a residence or facility, and the autonomous robot 1 may be made to recognize the position of the location where the marker is installed.

A home can include any area such as the entryway, children's room or bedroom. Facilities include any areas such as reception counters, rest areas and emergency exits. A marker is used that can identify the area type (for example, an entrance type or a reception counter type, which will be described later) associated with these areas. The marker may be of any type as long as it has characteristics that allow the area type to be identified through image recognition, such as shape, pattern, color, characters or figures attached to the marker, or a combination thereof. . For example, a graphic code obtained by converting an area type code into a graphic using a general-purpose conversion method (bar code method, two-dimensional bar code method, etc.) may be used as the marker. In this case, the marker recognition unit 22 can read the area type code from the graphic code using the conversion method. Alternatively, a uniquely designed figure may be used as a marker. In this case, the marker recognition unit 22 specifies the type of figure included in the captured image if the shape is a predetermined shape, and specifies the area type corresponding to the identified type of figure. You can. It is assumed that the data providing device 10 or the robot 2 stores data that associates the type of figure with the type of area. In other words, the marker in the second embodiment can specify any area type.

If such a marker is installed in an area identified by the area type, the autonomous robot 1 can recognize that the marker installation location corresponds to the area identified by the area type. For example, if an entrance-type marker is installed at the entrance, the autonomous robot 1 can recognize that the entrance-type marker is installed at the entrance.

In the second embodiment, the autonomous robot 1 stores marker information that associates a predetermined event with an area type. When the autonomous robot 1 detects a predetermined event, the robot 2 moves to a location where a marker of the area type corresponding to the predetermined event is installed (referred to as a marker installation location). The predetermined event that triggers the robot 2 to move to the marker installation location is referred to as a first event. The first event may be detected by the robot 2 or the data providing device 10.

Furthermore, when the autonomous robot 1 detects another predetermined event after the robot 2 moves to the marker installation location, the robot 2 executes a predetermined action. An event that triggers the robot 2 to perform a predetermined action is referred to as a second event. The second event may be detected by the robot 2 or the data providing device 10. The action to be executed corresponds to at least one of the first event, the area type, and the second event. In the second embodiment, the autonomous robot 1 stores event information that associates an action with at least one of the first event, area type, and second event. The robot 2 may store the event information.

FIG. 8 is a block diagram showing an example of the module configuration of the robot 2 in the second embodiment. FIG. 8 also shows functional units related to Examples 1 to 6, which will be described later. The robot 2 includes a marker recognition section 22, a position measurement section 25, a movement control section 23, a communication control section 26, a first event detection section 210, a second event detection section 220, and an action execution section 230. The above-mentioned functional units of the robot 2 in the second embodiment will be described as functional modules realized by a program that controls the robot 2.

The marker recognition unit 22 recognizes the marker included in the photographed image and specifies the area type indicated by the marker. The position measuring unit 25 measures the current position and direction of the robot 2. The position measuring unit 25 may measure the current position and direction based on a captured image, or may measure the current position based on radio waves received from a wireless communication device installed at a predetermined position. The method of determining the current location may be any conventional technique. The position measurement unit 25 may use SLAM (Simultaneous Localization and Mapping) technology that simultaneously estimates the self-position and creates an environmental map. The movement control unit 23 controls the movement of the robot 2. The movement control unit 23 sets a route to the destination and drives the movement mechanism 29 to move itself to the destination along the route. The communication control unit 26 communicates with the data providing device 10.

The first event detection unit 210 detects a first event. The first event detection unit 210 may detect the first event based on the result of recognition processing such as voice recognition or image recognition. That is, the first event detection unit 210 may detect the first event when it is determined that the sound input by the microphone included in the robot 2 includes characteristics that are presumed to correspond to the predetermined sound. The first event detection unit 210, for example, records a predetermined audio sample in advance, analyzes the audio, and extracts characteristic data such as frequency distribution, intonation, and the period of increase in volume. Then, the first event detection unit 210 performs a similar analysis on the sound input through the microphone, and if the characteristic data such as frequency distribution, intonation, and period of increase in volume match or approximate the sample, It may be assumed that the sound inputted by the microphone corresponds to the predetermined sound.

Further, the first event detection section 210 may detect the first event when it is estimated that an arbitrary person, a predetermined person, or a predetermined object is included in the image photographed by the photographing section 21. The first event detection unit 210 photographs an arbitrary person, a predetermined person, or a predetermined object as a sample in advance using the photographing unit 21, and analyzes the photographed image to determine the size, shape, and arrangement of parts of the subject. Extract the feature data of. Then, the first event detection unit 210 analyzes the image taken by the imaging unit 21 and determines that the sample includes an object that has common or similar characteristic data such as size, shape, and arrangement of parts. , it may be assumed that an arbitrary person, a predetermined person, or a predetermined object appears in the photographed image.

The first event detection unit 210 may detect the first event based on measurement results of various sensors such as a temperature sensor, a contact sensor, or an acceleration sensor. The first event detection unit 210 detects when the measured value of the sensor falls below a predetermined lower limit value, when the measured value of the sensor falls within a predetermined range, when the measured value of the sensor deviates from the predetermined range, or when the measured value of the sensor The first event may be detected when the measured value of exceeds a predetermined upper limit. The first event detection unit 210 may detect the first event when, for example, a temperature sensor measures a temperature corresponding to a person's body temperature. The first event detection unit 210 may detect the first event when a contact corresponding to a human touch is measured by a contact sensor, for example. The first event detection unit 210 may detect the first event when an acceleration sensor measures a change in acceleration corresponding to a large shaking such as an impact caused by a traffic accident or an earthquake.

The first event detection unit 210 may detect the first event based on the communication state in communication processing. The first event detection unit 210 indicates the communication state when a characteristic value indicating the communication state, such as radio wave intensity in wireless communication, communication time with a predetermined communication partner, or transmission amount per predetermined time, falls below a lower limit value. The first event is detected when the characteristic value falls within a predetermined range, when the characteristic value indicating the communication status deviates from the predetermined range, or when the characteristic value indicating the communication status exceeds a predetermined upper limit value. You can.

The first event detection unit 210 may detect the first event based on data received from the data providing device 10, the user terminal 3, or another external device. The first event detection unit 210 may detect the first event, for example, when receiving a predetermined notification from the data providing device 10 or another external device. The first event detection unit 210 may detect the first event when receiving a predetermined request from the user from the user terminal 3.

The voice recognition section 211 , the radio wave condition detection section 213 , the patrol event detection section 215 , and the wake-up event detection section 217 are examples of the first event detection section 210 . The voice recognition unit 211 will be described in a second embodiment (an application example of babysitting). The radio wave condition detection unit 213 will be explained in a fourth embodiment (an application example of call support). The patrol event detection unit 215 will be described in Example 5 (security application example). The wake-up event detection unit 217 will be described in Example 6 (alarm clock application example).

The second event detection unit 220 detects a second event. The second event detection section 220 may detect the second event based on the result of recognition processing such as voice recognition or image recognition, as in the case of the first event detection section 210. The second event detection section 220 may detect the second event based on the measurement results of various sensors such as a temperature sensor, a contact sensor, or an acceleration sensor, as in the case of the first event detection section 210. The second event detection section 220 may detect the second event based on the communication state in the communication process, as in the case of the first event detection section 210. Similarly to the first event detection unit 210, the second event detection unit 220 detects a second event based on data received from the data providing device 10, the user terminal 3, or another external device. good.

The user recognition section 221, the call request reception section 223, the person recognition section 225, and the body position recognition section 227 are examples of the second event detection section 220. The user recognition unit 221 will be described in Example 1 (an application example of greeting). The call request receiving unit 223 will be described in a fourth embodiment (an application example of call support). The person recognition unit 225 will be explained in Example 5 (security application example). The body position recognition unit 227 will be described in Example 6 (alarm clock application example).

The action execution unit 230 executes an action triggered by the second event. The action execution unit 230 may execute an action that involves the movement of the robot 2 itself. The action execution unit 230 may execute an action that involves input processing such as image, voice, or communication in the robot 2. The action execution unit 230 may execute an action that involves output processing of images, sounds, communications, etc. in the robot 2.

Posture control section 231 , voice output section 232 , remote control section 233 , message output section 235 , and telephone communication section 237 are examples of action execution section 230 . Furthermore, the movement control section 23 may function as the action execution section 230. When the movement control unit 23 functions as the action execution unit 230, the movement control unit 23 controls the movement mechanism 29 to perform an action related to movement of the robot 2. The posture control unit 231 performs actions related to the posture of the robot 2. If the robot 2 has a shape imitating a person or a virtual character and can move its neck and arms using actuators, the posture control unit 231 drives the actuators to make the robot 2 take various poses. or make various gestures. If the robot 2 has a shape imitating a four-legged animal and can move each leg using actuators provided at the joints, the posture control unit 231 drives the actuators to cause the robot 2 to perform various movements. You may also have them take a pose or perform various gestures.

The remote control unit 233 will be described in Example 2 (an application example of babysitting) and Example 6 (an application example of alarm clock). The message output unit 235 will be explained in Example 3 (customer service application example). The telephone communication unit 237 will be described in a fourth embodiment (an application example of call support).

FIG. 9 is a block diagram showing an example of the module configuration of the data providing device 10 in the second embodiment. FIG. 9 also shows functional units related to Examples 1 to 6, which will be described later. The robot 2 includes a first communication control section 11 , a second communication control section 16 , a marker registration section 110 , a first event detection section 120 , a second event detection section 130 , and an action selection section 140 . The above-mentioned functional units of the data providing device 10 in the second embodiment will be described as functional modules realized by a program that controls the data providing device 10.

The first communication control unit 11 controls wireless communication with the robot 2. The second communication control unit 16 controls wireless communication or wired communication with the user terminal 3 and other external devices. The marker registration unit 110 registers marker information including the position and orientation of the marker in a marker information storage unit 153, which will be described later.

The first event detection unit 120 detects a first event. The first event detection section 120 may detect the first event based on the result of recognition processing such as voice recognition or image recognition, as in the case of the first event detection section 210 of the robot 2. The first event detection unit 120 detects a first event based on the measurement results of various sensors such as the temperature sensor, contact sensor, or acceleration sensor of the robot 2, as in the case of the first event detection unit 210 of the robot 2. You can. The first event detection section 120 may detect the first event based on the communication state in the communication process, similarly to the case of the first event detection section 210 of the robot 2. The first event detection section 120 may detect the first event based on data received from the user terminal 3 or other external device, as in the case of the first event detection section 210 of the robot 2. The first event detection unit 120 may detect the first event based on data received from the robot 2.

The returning home event detection unit 121 and the visitor event detection unit 123 are examples of the first event detection unit 120. The returning home event detection unit 121 will be described in Example 1 (an application example of meeting and receiving). The visitor event detection unit 123 will be explained in Example 3 (an application example of customer service).

The second event detection unit 130 detects a second event. The second event detection unit 130 may detect the second event based on the result of recognition processing such as voice recognition or image recognition, as in the case of the first event detection unit 210 of the robot 2. The second event detection unit 130 detects a second event based on the measurement results of various sensors such as the temperature sensor, contact sensor, or acceleration sensor of the robot 2, as in the case of the first event detection unit 210 of the robot 2. You can. The second event detection unit 130 may detect the second event based on the communication state in the communication process, as in the case of the first event detection unit 210 of the robot 2. The second event detection unit 130 may detect the second event based on data received from the user terminal 3 or other external device, similarly to the first event detection unit 210 of the robot 2. The second event detection unit 130 may detect the second event based on data received from the robot 2.

The vacant room event detection unit 131 is an example of the second event detection unit 130. The vacant room event detection unit 131 will be explained in the third embodiment (an application example for customer service). The action selection unit 140 selects an action corresponding to at least one of the first event, area type, and second event. Below, an example will be shown in which an action mainly corresponding to a combination of a first event and a second event is selected.

The robot 2 further includes an event information storage section 151 and a marker information storage section 153. The event information storage unit 151 stores event information including the area type, second event, and action corresponding to the first event. The event information storage unit 151 will be described later in relation to FIG. 10. The marker information storage unit 153 stores marker information that associates the area type with the position and orientation of the marker. The marker information storage unit 153 will be described later in relation to FIG. 11.

FIG. 10 is a block diagram showing an example of the data structure of the event information storage unit 151 in the second embodiment. Each record in FIG. 10 specifies that when the first event is detected, the robot 2 moves to a location where a marker of the area type corresponding to the first event is installed. Further, each record in FIG. 10 specifies that, when the second event is detected, the robot 2 executes an action corresponding to the combination of the first event and the second event, for example. Details of each record will be explained in Examples 1 to 6. The event information may be set as a default or may be set by the user using an application on the user terminal 3.

FIG. 11 is a block diagram showing an example of the data structure of the marker information storage unit 153 in the second embodiment. In each record in FIG. 11, the position and orientation of the marker of the area type are set in association with the area type.

The user attaches an area type marker to an arbitrary position of the house or facility in advance to identify the area near that position. Then, in the marker registration phase, when the robot 1 detects an unknown marker, the data providing device 10 registers the marker information.

FIG. 12(A) is a flowchart showing the processing procedure in the marker registration phase of the second embodiment. For example, an unknown marker may be detected while the robot 2 is moving autonomously (S21). Specifically, the marker recognition unit 22 of the robot 2 recognizes the marker included in the image photographed by the photographing unit 21, and specifies the area type indicated by the marker. The marker recognition unit 22 stores markers that have been detected in the past, and can identify undetected markers by comparing the stored markers and the detected markers.

When the marker recognition unit 22 determines that an undetected marker has been recognized, the marker recognition unit 22 identifies the relative positional relationship and direction between the robot 2 and the marker by image recognition. The marker recognition unit 22 determines the distance between the robot 2 and the marker based on the size of the marker included in the photographed image. Furthermore, the marker recognition unit 22 determines the orientation of the marker with respect to the robot 2 based on the way the marker is distorted in the photographed image. Then, the marker recognition unit 22 determines the position and orientation of the marker based on the current position and direction of the robot 2 measured by the position measurement unit 25. The communication control unit 26 transmits marker information including the area type and the position and orientation of the marker of the area type to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the marker information, the marker registration unit 110 registers the position and orientation of the marker of the area type in the marker information storage unit 153 in association with the area type (step S22).

After completing the marker registration phase, in the action phase, the robot 2 moves to the marker installation location using the first event as a trigger, and the robot 2 performs a predetermined action using the second event as a trigger.

FIG. 12(B) is a flowchart showing the processing procedure in the action phase of the second embodiment. When the first event detection unit 210 of the robot 2 or the first event detection unit 120 of the data providing device 10 detects the first event (S23), the action selection unit 140 of the data providing device 10 selects a mark corresponding to the first event. The type is specified and the robot 2 is instructed to move to the position of the mark of the mark type. The movement control unit 23 of the robot 2 controls the movement mechanism 29 by setting a route to the mark position according to movement instructions. The robot 2 moves near the mark by the operation of the moving mechanism 29 (S24).

When the second event detection unit 220 of the robot 2 or the second event detection unit 130 of the data providing device 10 detects the second event, the action selection unit 140 of the data providing device 10 selects, for example, a combination of the first event and the second event. , and instructs the robot 2 to perform the selected action. The robot 2 then executes the instructed action (S26). Examples 1 to 6 related to the second embodiment will be described below.

[Example 1]
An application example of greeting by the autonomous robot 1 will be explained. In an application example of greeting, when the user returns home, the robot 2 heads to the entrance and greets the user. If an area type marker that identifies the entrance as an area is installed at the entrance, the autonomous robot 1 can recognize that the location where the marker is installed is the entrance. The area type that identifies the entrance is called the entrance type.

The timing for the user to return home can be determined by the fact that the position measured by the GPS device (Global Positioning System) of the user terminal 3 held by the user approaches his or her home. For example, when the distance between the position of the user terminal 3 and the entrance (or the data providing device 10) becomes shorter than a reference length, it is determined that it is time for the user to return home. An event for determining when a user returns home is called a user return home event. In other words, the user returning home event corresponds to the first event in the application example of meeting and picking up.

When the robot 2 arrives at the entrance and recognizes the user from the image photographed by the photographing unit 21, it performs an action in response to the user returning home. An event in which the robot 2 recognizes a user is called a user recognition event. Furthermore, an action that responds to the user returning home is called a welcome action. A user recognition event is an opportunity to perform a welcome action. The user recognition event corresponds to the second event in the meet and greet application.

In the greeting action, for example, the audio output unit 232 outputs audio from a speaker included in the robot 2. The output voice may be a natural language such as "welcome back" or a non-verbal voice such as cheers. The movement control unit 23 may control the movement mechanism 29 to perform an action of moving the robot 2 back and forth in small increments or rotating it as a greeting action. If the robot 2 has a shape imitating a person or a virtual character and can move its arms using actuators, the posture control unit 231 may drive the actuators to raise and lower its arms. If the robot 2 can move its head using an actuator, the posture control unit 231 may drive the actuator to shake its head. If the robot 2 has a shape imitating a four-legged animal and can move each leg using actuators provided at its joints, the posture control unit 231 can create a pose in which it stands up with only its hind legs as a greeting action. You may have them take it. This makes it possible to show that the robot 2 is happy for the user to return home. The user begins to feel close to the robot 2 that is waiting for him and deepens his attachment to it.

The return home event detection unit 121 of the data providing device 10 shown in FIG. 9 detects a user return home event when it is determined that the user approaches home based on the position information of the user terminal 3, as described above. The returning home event detection unit 121 may detect a user returning home event when receiving a notification from the user terminal 3 indicating that the user terminal 3 has communicated with a beacon transmitter installed at the entrance. The returning home event detection unit 121 may detect a user returning home event when the user is recognized based on an image photographed by a camera-equipped intercom or an input voice. Further, the return home event detection unit 121 may detect a user return home event when receiving a return home advance notice email from the user terminal 3.

The user recognition unit 221 of the robot 2 shown in FIG. 8 detects a user by, for example, recognizing a facial part included in a photographed image or recognizing an input voice. The user recognition unit 221, for example, photographs a user's face as a sample in advance with the photographing unit 21, analyzes the photographed image, and determines the size, shape, and arrangement of parts (parts such as eyes, nose, mouth, etc.). Extract feature data. Then, the person recognition unit 225 analyzes the image photographed by the photographing unit 21, and if it is determined that the image contains a subject whose characteristic data such as size, shape, and arrangement of parts is the same as or similar to the sample, the photographed image is It may be assumed that the user's face appears in the image. Further, the user recognition unit 221 records the user's voice as a sample in advance, for example, and analyzes the user's voice to extract characteristic data such as frequency distribution and intonation. Then, the user recognition unit 221 performs a similar analysis on the sound input by the microphone, and if the characteristic data such as frequency distribution and intonation match or approximate the user's voice sample, the user recognition unit 221 determines whether the microphone It may be assumed that the input sound corresponds to the user's voice.

In the first record in the data structure of the event information storage unit 151 shown in FIG. 10, when a user returning home event is detected as the first event regarding the application example of greeting, a marker whose area type is the entrance type is installed. It is determined that the robot 2 moves to the location where the robot 2 is located (that is, the entrance). The first record further specifies that the robot 2 performs a welcoming action when a user recognition event is detected as a second event. A marker whose area type is an entrance type is called an entrance marker.

In the first record in the data structure of the marker information storage unit 153 shown in FIG. 11, the position and direction of the entrance marker are set in association with the entrance type of the area type regarding the application example of greeting.

FIG. 13(A) is a flowchart showing the processing procedure in the marker registration phase of the first embodiment. For example, when the marker recognition unit 22 detects an entrance marker while the robot 2 is moving autonomously (step S31), the communication control unit 26 detects an entrance marker including the entrance type of the area type and the position and orientation of the entrance marker. The information is transmitted to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the entrance marker information, the marker registration unit 110 registers the position and orientation of the entrance marker in the marker information storage unit 153 in association with the entrance type of the area type ( Step S32).

FIG. 13(B) is a flowchart showing the processing procedure in the action phase of the first embodiment. When the returning home event detection unit 121 of the data providing device 10 detects the user returning home event (step S33), it refers to the event information storage unit 151 and specifies the entrance type of the area type corresponding to the user returning home event. The returning home event detection unit 121 further refers to the marker information storage unit 153 to identify the position and orientation of the entrance marker corresponding to the entrance type of the area type. The first communication control unit 11 transmits to the robot 2 a movement instruction to the entrance including the position and orientation of the entrance marker. At this time, the return home event detection unit 121 notifies the action selection unit 140 of the user return home event.

When the communication control unit 26 of the robot 2 receives an instruction to move to the entrance including the position and orientation of the entrance marker, the movement control unit 23 of the robot 2 controls the movement mechanism 29, and the robot 2 moves to the entrance (step S34). The robot 2 remains in front of the entrance marker position for at least a first predetermined period of time. The first predetermined time corresponds to the upper limit of the expected interval from when the user returns home event is detected until when the user recognition event is detected. If the user recognition event is not detected after the first predetermined time has elapsed, the processing in the action phase of the first embodiment may be interrupted.

When a user opens the door and enters, the robot 2 recognizes the user. Specifically, the user recognition unit 221 detects a user recognition event by recognizing the user's face included in the image taken by the photographing unit 21 or by voice recognition of the sound input by the microphone ( Step S35). When the user recognition unit 221 of the robot 2 detects a user recognition event, the communication control unit 26
sends a user recognition event to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives a user recognition event, the action selection unit 140 refers to the event information storage unit 151 and selects the first event corresponding to the user recognition event of the second event. Identify user return home events. The action selection unit 140 refers to the event information storage unit 151 and selects a welcome action corresponding to the combination of the first event, the user returning home event, and the second event, the user recognition event. The first communication control unit 11 transmits an instruction for the selected welcome action to the robot 2.

When the communication control unit 26 of the robot 2 receives the instruction for the greeting action, the action execution unit 230 of the robot 2 executes the greeting action (step S36).

[Example 2]
An application example of babysitting using the autonomous robot 1 will be described. In a babysitting application example, when an infant starts crying in a child's room, the robot 2 heads to the child's room and comforts the infant. If a marker of the area type that identifies the child's room as an area is installed in the child's room, the autonomous robot 1 can recognize that the location of the marker is the child's room. The area type for identifying a child's room is called a child's room type.

In the babysitting application example, when an infant cries, the robot 2 moves to the child's room and shows concern for the infant. The infant's cry is detected by analyzing the sound input to the microphone included in the robot 2. In other words, the detection of infant crying corresponds to the first event in the babysitting application example. This event is called an infant crying event.

If there is no adult in the child's room, the robot 2 acts as if it were soothing the infant. The behavior of soothing an infant is called a soothing action. Absence of an adult is detected by recognition of an image photographed by the photographing unit 21. Detection of the absence of an adult corresponds to the second event in the babysitting application. This event is called an adult absence event.

In the cradling action, for example, a speaker included in the robot 2 outputs a sound that is effective for making the baby stop crying (for example, a cradling voice, laughter, the sound of rustling a paper bag). In addition to outputting sounds, the soothing action may also be a behavior that moves the components of the robot 2, such as the robot 2 tilting its head or raising and lowering its arms, or an image that is effective in stopping infants from crying. An action such as displaying a predetermined image on a display may also be used. The cradling action may be a remote control that controls a device different from the robot 2, such as starting up a television or outputting music from an audio device. If the robot 2 calms down the infant through the soothing action, the user, who is the parent, will feel at ease.

The voice recognition unit 211 of the robot 2 shown in FIG. 8 detects an infant crying event when it determines that the sound input to the microphone is an infant's crying. The voice recognition unit 211 records, for example, a sample of an infant's cry in advance, analyzes the infant's cry, and extracts characteristic data such as frequency distribution and a cycle of increasing volume. Then, the voice recognition unit 211 performs a similar analysis on the sound input to the microphone, and if the characteristic data such as the frequency distribution and the period of increase in volume match or approximate the sample of infant crying, It may be assumed that the sound input into the microphone corresponds to the cry of an infant.

The human recognition unit 225 of the robot 2 shown in FIG. 8 detects an adult absence event when it recognizes that an adult is not included in the image taken by the imaging unit 21. For example, the person recognition unit 225 uses the photographing unit 21 to previously photograph images of a plurality of adults with different genders and body types as samples, and analyzes the photographed images to extract feature data such as size and shape. Then, the person recognition unit 225 analyzes the image photographed by the photographing unit 21, and if it is determined that the photographed image contains a subject whose characteristic data such as size and shape is common or similar to the sample, the photographed image contains an adult. You can assume that it is in the photo. The remote control unit 233 of the robot 2 shown in FIG. 8 transmits remote control wireless signals to control devices different from the robot 2, such as starting a nearby television or outputting music to an audio device. remote control.

In the second record in the data structure of the event information storage unit 151 shown in FIG. 10, when an infant crying event is detected as the first event regarding the babysitting application example, a marker whose area type is a children's room type is set. It is determined that the robot 2 will move to the location where the child is playing (that is, the child's room). The second record further specifies that the robot 2 performs a soothing action when an adult absence event is detected as a second event. A marker whose area type is a children's room type is called a children's room marker.

In the second record in the data structure of the marker information storage unit 153 shown in FIG. 11, the position and orientation of the child's room marker is set in association with the child's room type of the area type regarding the babysitting application example.

FIG. 14(A) is a flowchart showing the processing procedure in the marker registration phase of the second embodiment. For example, when the marker recognition unit 22 detects a children's room marker while the robot 2 is moving autonomously (step S41), the communication control unit 26 determines the area type of the children's room type and the position and orientation of the children's room marker. The children's room marker information including the child's room marker information is transmitted to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the children's room marker information, the marker registration unit 110 registers the position and orientation of the entrance marker in the marker information storage unit 153 in association with the entrance type of the area type. (Step S42).

FIG. 14(B) is a flowchart showing the processing procedure in the action phase of the second embodiment. When the voice recognition unit 211 of the robot 2 detects an infant crying event (step S43), the communication control unit 26 notifies the data providing device 10 of the infant crying event.

When the first communication control unit 11 of the data providing device 10 receives a notification of an infant crying event, the action selection unit 140 refers to the event information storage unit 151 and selects a child's room of the area type associated with the infant crying event. Identify the type. The action selection unit 140 further refers to the marker information storage unit 153 to specify the position and orientation of the children's room marker corresponding to the children's room type of the area type. The first communication control unit 11 transmits to the robot 2 an instruction to move to the child's room, including the position and orientation of the child's room marker.

When the communication control unit 26 of the robot 2 receives an instruction to move to the child's room including the position and orientation of the child's room marker, the movement control unit 23 controls the movement mechanism 29, and the robot 2 moves to the child's room (step S44). The robot 2 remains in front of the child's room marker for at least a second predetermined period of time. The second predetermined time corresponds to the upper limit of the estimated time required for the robot 2 to recognize the absence of an adult after entering the child's room. If the adult absence event is not detected after the second predetermined time has elapsed, the processing in the action phase of the second embodiment may be interrupted.

When the person recognition unit 225 detects an adult absence event (step S45), the communication control unit 26 transmits the adult absence event to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the adult absence event, the action selection unit 140 refers to the event information storage unit 151 and selects the first event, the infant crying, corresponding to the second event, the adult absence event. Identify the event. The action selection unit 140 refers to the event information storage unit 151 and selects a soothing action corresponding to the combination of the first event, the infant crying event, and the second event, the adult absence event. The first communication control unit 11 transmits an instruction for the selected cradling action to the robot 2.

When the communication control unit 26 of the robot 2 receives the instruction for the soothing action, the action execution unit 230 of the robot 2 executes the soothing action (step S46).

[Example 3]
An application example of customer service using the autonomous robot 1 will be explained. An example of a customer service application is assumed to be a store that provides food and beverage services in guest rooms, for example. When a customer using the store enters the store, the robot 2 heads to the reception counter and guides the customer to the guest room. If an area type marker that identifies the reception counter as an area is installed at the reception counter, the autonomous robot 1 can recognize that the location where the marker is installed is the reception counter. The area type that identifies the reception counter is called the reception counter type.

In a customer service application example, when a visitor is detected, the robot 2 moves to a reception counter and serves the customer. A visitor is detected by seeing the customer entering the store in an image taken by a camera installed at the entrance of the store. In other words, the detection of a visitor corresponds to the first event in the customer service application example. This event is called a visitor event.

If there is a vacant room, the robot 2 behaves to guide the guest to the vacant room. The behavior of guiding a guest to a vacant room is called a guidance action. Information on vacant rooms is obtained from a guest room management system (not shown). Detection of a vacant room corresponds to the second event in the customer service application. This event is called a vacant room event.

In the guidance action, for example, the robot 2 guides the customer to a vacant room. Alternatively, the message output unit 235 may output the guidance message "Please enter room No. ○" by voice or display it on the display device included in the robot 2. When Robot 2 serves customers, customers feel a special touch that human service does not have. By making it unmanned, costs can also be reduced.

The visitor event detection unit 123 of the data providing device 10 shown in FIG. 9 detects a visitor event, for example, when a customer entering the store is recognized from an image taken with a camera provided at the entrance of the store. The vacant room event detection unit 131 of the data providing device 10 shown in FIG. 9 inquires of the guest room management system (not shown) about the status of guest rooms, and detects a vacant room event if there is a vacant guest room. The movement control unit 23 of the robot 2 shown in FIG. 8 drives the movement mechanism 29 so that the robot 2 moves slowly to the guest room. The robot 2 may measure its distance with the customer based on the image of the customer photographed by the photographing unit 21 of the robot 2, and may control the speed of movement so as to maintain a constant distance from the customer. In addition, the message output unit 235 of the data providing device 10 shown in FIG. Outputs a guidance message.

In the third record in the data structure of the event information storage unit 151 shown in FIG. 10, when a visitor event is detected as the first event regarding the application example of customer service, a marker whose area type is the reception counter type is installed. It is determined that the robot 2 moves to the location where the robot 2 is located (that is, the reception counter). The third record further specifies that the robot 2 performs a guidance action when an empty room event is detected as the second event. A marker whose area type is a reception counter type is called a reception counter marker.

In the third record in the data configuration of the marker information storage unit 153 shown in FIG. 11, the position and orientation of the reception counter marker is set in association with the reception counter type of the area type regarding the customer service application example.

FIG. 15(A) is a flowchart showing the processing procedure in the marker registration phase of the third embodiment. For example, when the marker recognition unit 22 detects a reception counter marker while the robot 2 is moving autonomously (step S51), the communication control unit 26 determines the reception counter type of the area type and the position and orientation of the reception counter marker. The reception counter marker information including the information is sent to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the reception counter marker information, the marker registration unit 110 stores the position and orientation of the reception counter marker in the marker information storage unit 153 in association with the reception counter type of the area type. Register (step S52).

FIG. 15(B) is a flowchart showing the processing procedure in the action phase of the third embodiment. When the visitor event detection unit 123 of the data providing device 10 detects a visitor event (step S53), it refers to the event information storage unit 151 and identifies the reception counter type of the area type associated with the visitor event. The visitor event detection unit 123 further refers to the marker information storage unit 153 to identify the position and orientation of the reception counter marker corresponding to the reception counter type of the area type. The first communication control unit 11 transmits to the robot 2 an instruction to move to the reception counter, including the position and orientation of the reception counter marker. At this time, the visitor event detection section 123 notifies the action selection section 140 of the visitor event.

When the communication control unit 26 of the robot 2 receives an instruction to move to the reception counter including the position and orientation of the reception counter marker, the movement control unit 23 controls the movement mechanism 29, and the robot 2 moves to the reception counter (step S54). The robot 2 remains in front of the position of the reception counter marker for at least a third predetermined period of time. The third predetermined time corresponds to the upper limit of the switching time until the store clerk responds to the customer instead of the robot 2. If no vacant room event is detected after the third predetermined time period has elapsed, the processing in the action phase of the third embodiment may be interrupted.

When the vacant room event detection unit 131 of the data providing device 10 detects a vacant room event (step S55), the action selection unit 140 refers to the event information storage unit 151 to determine whether the visitor of the first event corresponding to the vacant room event of the second event Identify the event. The action selection unit 140 refers to the event information storage unit 151 and selects a customer service action corresponding to the combination of the first event, the visitor event, and the second event, the vacant room event. The first communication control unit 11 transmits an instruction for the selected customer service action to the robot 2.

When the communication control unit 26 of the robot 2 receives the instruction for the customer service action, the action execution unit 230 of the robot 2 executes the customer service action (step S56).

[Example 4]
An application example of call support by the autonomous robot 1 will be described. In this example, it is assumed that the robot 2 has a telephone function. Furthermore, it is assumed that in the residence or facility where the robot 2 is located, there are some places where it is easy to receive radio waves for wireless communication used for telephone functions and some places where it is difficult to reach. The robot 2 moves from a place where the radio wave condition is poor to a place where the radio wave condition is good (hereinafter referred to as a place with good radio wave condition), and starts a call in response to a call request. If a marker of an area type for identifying a place with good radio waves is installed in a place with good radio waves, the autonomous robot 1 can recognize that the place where the marker is installed is a place with good radio waves. The area type that identifies a location with good radio waves is called a good radio wave type.

In a call support application example, when the radio wave condition deteriorates, the robot 2 moves to a place with good radio waves so that there is no problem with wireless communication. In other words, the detection of radio wave deterioration corresponds to the first event in the call support application example. This event is called a radio wave deterioration event.

When the robot 2 receives a call request from the user, it starts the call. The processing action for starting a call is called a call start action. Acceptance of a call request is detected through recognition processing, such as recognizing a voice such as "Please make a call," or recognizing a gesture or pose used to make a call from a captured image. Acceptance of the call request corresponds to the second event in the application example of call support. This event is called a call request event.

The radio wave state detection unit 213 of the robot 2 shown in FIG. 8 monitors the state of radio waves for wireless communication used in the telephone function, and detects a radio wave deterioration event when the strength of the radio waves falls below an acceptable standard. The call request accepting unit 223 of the robot 2 shown in FIG. Accept call requests from. The call request reception unit 223 may identify the telephone number and the other party to call by voice recognition. The telephone communication unit 237 of the robot 2 shown in FIG. 8 controls telephone communication and performs call processing.

In the fourth record in the data structure of the event information storage unit 151 shown in FIG. 10, when a radio wave deterioration event is detected as the first event regarding the application example of call support, a marker whose area type is good radio wave type is displayed. It is determined that the robot 2 moves to a location where it is installed (that is, a location with good radio waves). The fourth record further specifies that the robot 2 initiates a call when a call request event is detected as the second event. A marker whose area type is a good signal type is called a good signal marker.

In the fourth record in the data structure of the marker information storage unit 153 shown in FIG. 11, the position and orientation of the radio wave good marker is set in association with the radio wave good type of the area type regarding the application example of call support.

FIG. 16(A) is a flowchart showing the processing procedure in the marker registration phase of the fourth embodiment. For example, when the marker recognition unit 22 detects a good radio wave marker while the robot 2 is moving autonomously (step S61), the communication control unit 26 detects the good radio wave type of the area type and the position and orientation of the good radio wave marker. The radio wave good marker information including the information is transmitted to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the good radio wave marker information, the marker registration unit 110 stores the position and orientation of the good radio wave marker in the marker information storage unit 153 in association with the good radio wave type of the area type. Register (step S62).

FIG. 16(B) is a flowchart showing the processing procedure in the action phase of the fourth embodiment. When the radio wave condition detection unit 213 of the robot 2 detects a radio wave deterioration event (step S63), the communication control unit 26 notifies the data providing device 10 of the radio wave deterioration event.

When the first communication control unit 11 of the data providing device 10 receives a notification of a radio wave deterioration event, the action selection unit 140 refers to the event information storage unit 151 and selects a good radio wave for the area type associated with the radio wave deterioration event. Identify the type. The action selection unit 140 further refers to the marker information storage unit 153 to identify the position and orientation of the radio wave good marker corresponding to the radio wave good type of the area type. The first communication control unit 11 transmits to the robot 2 an instruction to move to a location with good radio waves, including the position and orientation of the radio wave good marker.

When the communication control unit 26 of the robot 2 receives an instruction to move to a place with good radio waves, including the position and direction of the poor radio wave marker, the movement control unit 23 controls the movement mechanism 29, and the robot 2 moves to a place with good radio waves. Move (step S64). The robot 2 remains in front of the position of the radio wave good marker for at least a fourth predetermined time. The fourth predetermined time corresponds to the upper limit length of the period during which the user is expected to request a call. If a call request event is not detected after the fourth predetermined time has elapsed, the processing in the action phase of the fourth embodiment may be interrupted.

When the call request reception unit 223 detects a call request event (step S65), the communication control unit 26 transmits the call request event to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the call request event, the action selection unit 140 refers to the event information storage unit 151 and selects a call start action corresponding to the second event, the call request event. . The first communication control unit 11 transmits an instruction for the selected call start action to the robot 2. Therefore, the call start action is transmitted regardless of whether or not the radio wave deterioration event has been notified first.

When the communication control unit 26 of the robot 2 receives the instruction for the call start action, the telephone communication 237 of the robot 2 executes the call start action (step S66). When the telephone communication unit 237 makes a call and enters a conversation state, a signal obtained by converting the input voice of the microphone provided in the robot 2 is transmitted, and the other party's voice converted from the received signal is output from the microphone.

[Example 5]
An example of security application using the autonomous robot 1 will be described. In a security application example, the robot 2 patrols a residence or facility where a safe is located to guard against safe breakers. If a marker of the area type that identifies the safe storage area is installed in the safe storage area, the autonomous robot 1 can recognize that the location where the marker is installed is the safe storage area. The area type that identifies the safe storage area is called a safe type.

In a security application example, the robot 2, which is instructed to patrol, moves to the vicinity of a safe and behaves to grasp the situation. The reception of the patrol instruction may be performed, for example, by recognition of a voice such as "Come and check the safe" or recognition processing such as recognition of a predetermined pose or gesture from a photographed image. The data providing device 10 may receive a monitoring instruction from an application on the user terminal 3, transfer it to the robot 2, and the robot 2 may receive the monitoring instruction. At a predetermined time, the data providing device 10 may automatically send a patrol instruction to the robot 2, and the robot 2 may receive the guard instruction. In the security application example, an instruction to the robot 2 to patrol corresponds to the first event. This event is called a patrol event.

When the robot 2 detects a person near the safe by recognizing the photographed image, the robot 2 executes a warning action. In other words, in a security application example, the recognition of a person near a safe corresponds to the second event. This event is called a person recognition event. It is assumed that the person near the safe may be a suspicious person. As a warning action, the communication control unit 26 transmits the image (moving image or still image) of the robot 2 photographed by the photographing unit 21 to the data providing device 10 . Regarding this process, the communication control unit 26 is an example of the action execution unit 230. The data providing device 10 may record the received video as evidence. Further, the data providing device 10 may transmit data to the application on the user terminal 3 by transmitting a warning message to the application on the user terminal 3 or forwarding the video received from the robot 2. . As a warning action, the audio output unit 232 of the robot 2 may emit an alarm sound.

The patrol event detection unit 215 of the robot 2 shown in FIG. 8 detects a patrol event when it receives a patrol instruction in the recognition process described above or when it receives a patrol instruction from the data providing device 10. The human recognition unit 225 of the robot 2 shown in FIG. 8 recognizes the human figure included in the photographed image. The presence of a person may be recognized by voice recognition of the person's speaking voice. The person recognition unit 225 may determine that the sound is a human speaking voice when a frequency corresponding to a standard human voice is extracted from the sound input to the microphone.

The fifth record in the data structure of the event information storage unit 151 shown in FIG. 10 is related to the security application example, and when a patrol event is detected as the first event, a marker whose area type is a safe type is installed. It is determined that the robot 2 moves to the location where the robot 2 is located (that is, the safe storage area). The fifth record further specifies that the robot 2 performs a warning action when a human recognition event is detected as a second event. A marker whose area type is a safe type is called a safe marker.

In the fifth record in the data structure of the marker information storage unit 153 shown in FIG. 11, the position and orientation of the safe marker is set in association with the safe type of the area type regarding the security application example.

FIG. 17(A) is a flowchart showing the processing procedure in the marker registration phase of the fifth embodiment. For example, when the marker recognition unit 22 detects a safe marker while the robot 2 is moving autonomously (step S71), the communication control unit 26 detects the safe marker, which includes the safe type of the area type and the position and orientation of the safe marker. The information is transmitted to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the safe marker information, the marker registration unit 110 registers the position and orientation of the safe marker in the marker information storage unit 153 in association with the safe type of the area type ( Step S72).

FIG. 17(B) is a flowchart showing the processing procedure in the action phase of the fifth embodiment. When the patrolling event detection unit 215 of the robot 2 detects a patrolling event (step S73), the communication control unit 26 notifies the data providing device 10 of the patrolling event.

When the first communication control unit 11 of the data providing device 10 receives a notification of a patrol event, the action selection unit 140 refers to the event information storage unit 151 and specifies the safe type of the area type associated with the patrol event. do. The action selection unit 140 further refers to the marker information storage unit 153 to specify the position and orientation of the safe marker corresponding to the safe type of the area type. The first communication control unit 11 transmits to the robot 2 an instruction to move to the safe storage area, including the position and orientation of the safe marker.

When the communication control unit 26 of the robot 2 receives an instruction to move to the safe storage area including the position and orientation of the safe marker, the movement control unit 23 controls the movement mechanism 29, and the robot 2 moves to the vicinity of the safe (step S74). ). The robot 2 remains in front of the position of the safe marker for at least a fifth predetermined period of time. The fifth predetermined time corresponds to the upper limit of the estimated time required for the robot 2 to recognize a person after moving near the safe. If the human recognition event is not detected after the fifth predetermined time has elapsed, the processing in the action phase of the fifth embodiment may be interrupted.

When the person recognition unit 225 detects a person recognition event (step S75), the communication control unit 26 transmits the person recognition event to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives a person recognition event, the action selection unit 140 refers to the event information storage unit 151 and selects the patrol event of the first event corresponding to the person recognition event of the second event. Identify. The action selection unit 140 refers to the event information storage unit 151 and selects a vigilance action corresponding to the combination of the first event, which is a patrol event, and the second event, which is a person recognition event. The first communication control unit 11 transmits an instruction for the selected warning action to the robot 2.

When the communication control unit 26 of the robot 2 receives the warning action instruction, the action execution unit 230 of the robot 2 executes the warning action (step S76).

[Example 6]
An application example of an alarm clock using the autonomous robot 1 will be explained. For example, the robot 2 wakes up a user who is sleeping in the bedroom in the morning. If an area type marker that identifies the bedroom as an area is installed in the bedroom, the autonomous robot 1 can recognize that the location where the marker is installed is the bedroom. The area type that identifies a bedroom is called a bedroom type.

In the alarm clock application example, the robot 2 moves to the bedroom and performs a wake-up action at the timing to wake up the user. The timing to wake up the user is determined, for example, based on the scheduled wake-up time. Alternatively, the user can be woken up by recognition processing such as recognizing voices such as "Please wake up dad (user)" uttered by the user's family, or recognizing predetermined poses and gestures from captured images. You can judge the timing. In other words, determining the timing to wake up the user corresponds to the first event in the alarm clock application example. This event is called a wake-up event.

When the robot 2 recognizes that the user is lying on the bed in the bedroom (supine position), the robot 2 performs a wake-up action. When the user is not in a lying position, such as when there is no user on the bed or when the user is already awake, the robot 2 does not perform the wake-up action. In other words, detecting a user in a lying position on a bed in an application example of an alarm clock corresponds to the second event. This event is called a user lying down event. As a wake-up action, for example, the audio output unit 232 outputs a voice such as a wake-up sound or a call from a speaker. As a wake-up action, the remote control unit 233 may cause an external device to output audio, such as starting a television or causing an audio device to output music. The remote control unit 233 may turn on lighting equipment. Alternatively, the movement control unit 23 may control the movement mechanism 29 so that the robot 2 moves vigorously around the bed.

The wake-up event detection unit 217 of the robot 2 shown in FIG. 8 detects a wake-up event when the scheduled wake-up time arrives as described above, or by the recognition process described above. The body position recognition unit 227 of the robot 2 shown in FIG. 8 recognizes the body position of a person (actually, considered to be a user) sleeping on a bed in a bedroom. The body position recognition unit 227 detects a user supine position event if the person sleeping on the bed is in the supine position. The body position recognition unit 227, for example, photographs a sample of a user in a supine position using the photographing unit 21 in advance, analyzes the photographed image, and determines the size, shape, and parts (head, hands, etc.) of the user in a supine position. Feature data such as the location of parts such as feet are extracted in advance. Then, the body position recognition unit 227 analyzes the image photographed by the photographing unit 21, and if it is determined that the image contains a subject whose characteristic data such as size, shape, and arrangement of parts is common or similar to the sample, It may be determined that the user in the supine position is included in the image taken.

The sixth record in the data structure of the event information storage unit 151 shown in FIG. 10 indicates that when a wake-up event is detected as the first event regarding the alarm clock application example, a marker whose area type is the bedroom type is installed. It is determined that the robot 2 moves to a place where the robot 2 is present (that is, the bedroom). The sixth record further specifies that the robot 2 performs a wake-up action when a user lying event is detected as a second event. A bedroom type marker is called a bedroom marker.

In the sixth record in the data structure of the marker information storage unit 153 shown in FIG. 11, the position and orientation of the bedroom marker is set in association with the bedroom type of the area type regarding the alarm clock application example.

FIG. 18(A) is a flowchart showing the processing procedure in the marker registration phase of the sixth embodiment. For example, when the marker recognition unit 22 detects a bedroom marker while the robot 2 is moving autonomously (step S81), the communication control unit 26 detects a bedroom marker that includes the bedroom type of the area type and the position and orientation of the bedroom marker. The information is transmitted to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the bedroom marker information, the marker registration unit 110 registers the position and orientation of the bedroom marker in the marker information storage unit 153 in association with the bedroom type of the area type ( Step S82).

FIG. 18(B) is a flowchart showing the processing procedure in the action phase of the sixth embodiment. When the wake-up event detection unit 217 of the robot 2 detects a wake-up event (step S83), the communication control unit 26 notifies the data providing device 10 of the wake-up event.

When the first communication control unit 11 of the data providing device 10 receives the notification of the wake-up event, the action selection unit 140 refers to the event information storage unit 151 and specifies the bedroom type of the area type associated with the wake-up event. do. The action selection unit 140 further refers to the marker information storage unit 153 to specify the position and orientation of the bedroom marker corresponding to the bedroom type of the area type. The first communication control unit 11 transmits to the robot 2 an instruction to move to the bedroom, including the position and orientation of the bedroom marker.

When the communication control unit 26 of the robot 2 receives the instruction to move to the bedroom including the position and orientation of the bedroom marker, the movement control unit 23 controls the movement mechanism 29, and the robot 2 moves to the bedroom (step S84). The robot 2 remains in front of the bedroom marker position for at least a sixth predetermined period of time. The sixth predetermined time corresponds to the upper limit of the estimated time required from when the robot 2 enters the bedroom to when the user's prone position event is detected. When the sixth predetermined period of time has elapsed, if the user lying down event has not been detected, the processing in the action phase of the sixth embodiment may be interrupted.

When the body position recognition unit 227 detects a user supine position event (step S85), the communication control unit 26 transmits the user supine position event to the data providing device 10.

When the first communication control unit 11 of the data providing device 10 receives the user supine event, the action selection unit 140 refers to the event information storage unit 151 and selects the first communication control unit 11 corresponding to the user supine event as the second event. Identify the triggering event for the event. The action selection unit 140 refers to the event information storage unit 151 and selects a wake-up action corresponding to the combination of the first event, which is a wake-up event, and the second event, which is a user lying down event. The first communication control unit 11 transmits an instruction for the selected wake-up action to the robot 2.

When the communication control unit 26 of the robot 2 receives the instruction for the wake-up action, the action execution unit 230 of the robot 2 executes the wake-up action (step S86).

In the first to sixth embodiments described above, the processing described as being performed by the first event detection unit 210 of the robot 2 may be performed by the first event detection unit 120 of the data providing device 10. The processing described as being performed by the second event detection unit 220 of the robot 2 may be performed by the second event detection unit 130 of the data providing device 10. The processing described as being performed by the first event detection unit 120 of the data providing device 10 may be performed by the first event detection unit 210 of the robot 2. The processing described as being performed by the second event detection unit 130 of the data providing device 10 may be performed by the second event detection unit 220 of the robot 2.

In the second embodiment described above, the robot 2 is moved to a predetermined location at a predetermined timing, and further a predetermined action is executed when a predetermined condition is satisfied. It is convenient because such a series of operations can be easily realized by installing the marker at a predetermined location.

[Embodiment 3]
When the robot 2 recognizes a marker, the robot 2 may execute an action instructed by the marker. For this purpose, a marker is used that is a graphic representation of the action identifier. For example, a graphic code obtained by converting an action identifier into a graphic using a general-purpose conversion method (bar code method, two-dimensional bar code method, etc.) may be used as a marker. In this case, the marker recognition unit 22 can read the action identifier from the graphic code using the conversion method. Alternatively, a uniquely designed figure may be used as a marker. In this case, the marker recognition unit 22 specifies the type of figure included in the photographed image when the shape is a predetermined shape, and specifies the identifier of the action corresponding to the specified type of figure. You can also do this. It is assumed that the marker recognition unit 22 stores data that associates the type of figure with the identifier of the action. That is, the marker in Embodiment 3 can specify the identifier of any action.

For example, if you set up a marker at the entrance of a child's room that instructs the robot to move to the living room as an action, when the robot 2 recognizes the marker in front of the child's room, it can move without entering the child's room. Move to the living room immediately.

The action indicated by the marker may be a search for another predetermined marker. If the action instructed by marker A is a search for marker B, the robot 2 searches for marker B and starts moving when it recognizes marker A. Furthermore, if the action instructed by marker B is a search for marker C, the robot 2 searches for marker C and starts moving when marker B is recognized. If a series of markers that instruct the robot to search for markers in sequence are placed at points on the route, the robot 2 will be made to search for a route that goes around the series of markers in order. In this way, it is also possible to play an indoor game simulating orienteering in which the child and the robot 2 compete. Moreover, it is also possible to run a plurality of robots 2 and perform an indoor race.

When playing an indoor play such as the above-mentioned indoor game or indoor race, the user instructs the autonomous robot 1 from the application on the user terminal 3 so that the robot 2 searches for marker A first. Good too. The robot 2 may recognize a user's call such as "Start" or "Search" by voice, and start searching for a marker in response to an event in which the user's call is detected. The robot 2 may image-recognize the pose or gesture instructing the user to start from the photographed image, and start searching for the marker in response to an event in which the pose or gesture is detected. Alternatively, the user may set a start time for the autonomous robot 1 from an application on the user terminal 3, and the robot 2 may start searching for a marker as a starting point upon reaching the start time. Good too.

In the third embodiment described above, by installing a marker at a predetermined location, it is possible to easily instruct the action of the robot 2 at that location.

[Embodiment 4]
The robot 2 may be made to recognize objects that should not be approached by using a marker that instructs the robot 2 to prohibit access. For example, a marker with a graphic prohibition code is placed on fragile items such as ornaments and precision equipment. As described above, a graphic code obtained by converting an access prohibition code into a graphic using a general-purpose conversion method (bar code method, two-dimensional bar code method, etc.) may be used as a marker. Alternatively, a uniquely designed figure may be used as a marker. In this case, the marker recognition unit 22 stores the type of figure that corresponds to the prohibition of approach, and determines that the type of figure specified from the photographed image corresponds to the prohibition of approach. The marker recognition unit 22 of the robot 2 measures the distance from the recognized marker. The movement control unit 23 controls the movement mechanism 29 to move the robot 2 in a direction away from the marker when it is determined that the distance from the marker has become shorter than the first reference distance. The first reference distance is a distance required to control the robot 2 so that it does not change direction and reach the marker position due to movement of the robot 2. In this way, the risk of the robot 2 colliding with fragile items can be reduced.

Alternatively, a marker may be installed to instruct the robot 2 not to approach an object that is likely to move, such as a balance ball or a vacuum cleaner body, so that the robot 2 can recognize the object that is likely to move. The movement control unit 23 controls the movement mechanism 29 to move the robot 2 in a direction away from the marker when it is determined that the distance from the marker has become shorter than the second reference distance. The second reference distance is a distance necessary for controlling the robot 2 to increase the distance from the article before the article moves and reaches the position of the robot 2. In this way, the risk of objects moving and colliding with the robot 2 can be reduced.

The type of article may be identified by the marker, and the approach of the robot 2 may be restricted depending on the type of article. A marker indicating the type of article may be used, or the data providing device 10 may hold article management data that associates the marker ID with the type of article. When using a marker that indicates the type of article, the marker recognition unit 22 also detects the type of article when it recognizes the marker. The marker recognition unit 22 stores article management data that associates marker IDs with article types. The marker recognition unit 22 detects the marker ID from the marker, and refers to the article management data to identify the type of article corresponding to the marker ID. The article management data may be set by the user using an application on the user terminal 3.

Furthermore, the marker recognition unit 22 may store approach control data in which the approachability of the robot 2 is set for each type of article, and determine whether the robot 2 can approach the type of article specified by the marker ID. In the access control data, access permission is set for items that are hard to break and unlikely to move, such as tables and chairs, and for items that are fragile such as decorations and precision equipment, as well as balance balls and vacuum cleaners. Access prohibitions are set for items that are likely to be moved. Since the main body of the vacuum cleaner always moves forward and never moves backward, approaching the rear of the main body of the vacuum cleaner may be permitted. In other words, it is sufficient to prohibit only approaching the front of the vacuum cleaner body. If the front and back of the vacuum cleaner body can be recognized based on the orientation of the marker, only approaching the front of the vacuum cleaner body may be prohibited. If the marker includes an arrow and is pasted so that the tip of the arrow points in front of the vacuum cleaner, the robot 2 can determine the range to which access is prohibited based on the marker. If there is a rule that prohibits people from approaching the area where the arrow points, there will be no operational problems as long as the direction of the marker is taken care of when pasting the marker. If you want to set a prohibited approach range that is not limited to where the arrow points, you may set the prohibited approach range based on the arrow in the approach control data, or you can set a similar approach range from the application on the user terminal 3. A prohibited range may be set.

In the fourth embodiment described above, by placing marks on items that are at risk of collision with the robot 2, such as fragile items or items that may move, collisions between the robot 2 and the items can be easily avoided. .

[Other variations]

Regarding the second embodiment, the marker recognition unit 22 may store marker definition information that associates a marker ID with an area type, detect the marker ID from the marker, and specify the area type corresponding to the marker ID. The association between the marker ID and the area type may be set by the user from an application on the user terminal 3.

Regarding the second embodiment, the first event is not limited to the example described above. The first event is optional. The first event detection unit 210 of the robot 2 and the first event detection unit 120 of the data providing device 10 may detect the first event when recognizing a person other than the user. The first event detection unit 210 of the robot 2 and the first event detection unit 120 of the data providing device 10 may detect the first event when an unknown person who has never been recognized is recognized for the first time. The first event detection unit 210 of the robot 2 and the first event detection unit 120 of the data providing device 10 may detect a first event when a known person who has been recognized in the past is recognized again. The first event detection unit 210 of the robot 2 and the first event detection unit 120 of the data providing device 10 may detect the first event when the same person is repeatedly recognized within a predetermined time. The first event detection unit 210 of the robot 2 and the first event detection unit 120 of the data providing device 10 may detect the first event when they recognize a person who has not been recognized within a predetermined time. Furthermore, the first event detection unit 210 of the robot 2 and the first event detection unit 120 of the data providing device 10 determine that the positional relationship and orientation of the person and the marker recognized in the captured image satisfy predetermined conditions. A first event may be detected.

Regarding the second embodiment, the second event is not limited to the example described above. The second event is optional. The second event detection unit 220 of the robot 2 and the second event detection unit 130 of the data providing device 10 may detect the second event when recognizing a person other than the user. The second event detection unit 220 of the robot 2 and the second event detection unit 130 of the data providing device 10 may detect the second event when the robot recognizes an unknown person who has never been recognized for the first time. The second event detection unit 220 of the robot 2 and the second event detection unit 130 of the data providing device 10 may detect a second event when a known person who has been recognized in the past is recognized anew. The second event detection unit 220 of the robot 2 and the second event detection unit 130 of the data providing device 10 may detect the second event when the same person is repeatedly recognized within a predetermined time. The second event detection unit 220 of the robot 2 and the second event detection unit 130 of the data providing device 10 may detect the second event when they recognize a person who has not been recognized within a predetermined time. Further, the second event detection unit 220 of the robot 2 and the second event detection unit 130 of the data providing device 10 determine that the positional relationship and orientation of the person and the marker recognized in the captured image satisfy predetermined conditions. A second event may also be detected.

Regarding the second embodiment, the first event and the second event may be a combination of events in multiple stages. For example, in the pick-up application example described in Embodiment 1, the returning home event detection unit 121 of the data providing device 10 detects the first stage event when the user terminal 3 approaches the home, and The second stage event may be detected when the mobile terminal 3 communicates with a beacon transmitter installed at the entrance. Then, the returning home event detection unit 121 may determine that the first event has been detected when both the first stage event and the second stage event are detected.

Regarding the second embodiment, an example has been described in which the action selection unit 140 selects an action corresponding to a combination of a first event and a second event. good. The action selection unit 140 may select an action corresponding to the area type. The action selection unit 140 may select an action corresponding to the first event. The action selection unit 140 may select an action corresponding to the combination of the area type and the second event. The action selection unit 140 may select an action corresponding to the combination of the first event and the area type. The action selection unit 140 may select an action corresponding to a combination of the first event, area type, and second event.

Regarding the second embodiment, the action execution unit 230 may perform an action targeting a specific person. For example, in the customer service application example described in the third embodiment, the action execution unit 230 may guide a specific customer to a vacant room. Further, the content of the action executed by the action execution unit 230 may be set by the user from an application on the user terminal 3.

Regarding the second embodiment, the first event detection unit 210 of the robot 2 may detect the first event when the communication control unit 26 communicates with another robot 2. The second event detection section 220 of the robot 2 may detect the second event when the communication control section 26 communicates with another robot 2 . The first event detection unit 210 of the robot 2 and the first event detection unit 120 of the data providing device 10 detect the first event when receiving a predetermined instruction or data from the user terminal 3 or other external device. good. The second event detection unit 220 of the robot 2 and the second event detection unit 130 of the data providing device 10 detect a second event when receiving a predetermined instruction or data from the user terminal 3 or other external device. good. For example, in the customer service application example described in Embodiment 3, the visitor event detection unit 123 of the data providing device 10 notifies the reception tablet terminal that the reception tablet terminal has received the number of visitors and room conditions such as non-smoking. The first event may be detected when the first event occurs.

Regarding the second embodiment, the detection conditions for the first event and the second event may be different for each of the plurality of robots 2. For example, only a specific robot 2 may detect the first event or the second event later than the timing at which the first event or the second event is normally detected. By delaying the detection timing of the first event or the second event, the passive character of the robot 2 can be produced. Conversely, only a specific robot 2 may detect the first event or the second event earlier than the timing at which the first event or the second event is normally detected. If the detection timing of the first event or the second event is made earlier, the aggressive character of the robot 2 can be produced. Note that the contents of the first event and the second event may be set by the user from an application on the user terminal 3.

Regarding the second embodiment, different event information may be set for each robot 2 in the event information storage section 151. In other words, event information applicable only to a specific robot 2 may be provided. For example, in a household where two robots 2 are operated, event information may be set so that only one robot performs a greeting action and only the other robot performs a babysitting action.

When operating a plurality of robots 2, the marker information recognized by one robot 2 may be notified to the other robot 2. In this way, the area type, marker position, and orientation can be quickly shared.

The robot 2 may regard feature points and feature shapes detected by SLAM technology as markers. Furthermore, a device that emits light may be used as a marker. A marker may be installed at a charging station for supplying power to a storage battery included in the robot 2, and the robot 2 may detect the positional relationship and orientation with respect to the charging station based on the marker. The robot 2 may use the positional relationship and orientation of the charging station detected by the marker when approaching the charging station for automatic connection.

The marker recognition unit 22 may measure the position of the same marker multiple times and calculate the average value regarding those positions. If the marker recognition unit 22 uses the average value of the marker positions, the influence of errors in measuring the marker positions can be reduced. Similarly, the marker recognition unit 22 may measure the orientation of the same marker multiple times and calculate the average value regarding those orientations. If the marker recognition unit 22 uses the average value of the orientations of the markers, it is possible to reduce the influence of errors in measuring the positions of the markers.

The application on the user terminal 3 may display the position and orientation of the marker on the output device of the user terminal 3. From the application on the user terminal 3, the user sets the contents of the marker (no entry, area type, action identifier, or no approach, etc.), the position and orientation of the marker, via the input device of the user terminal 3, Alternatively, it may be possible to modify it.

The robot 2 may include a beacon receiver, receive a beacon signal transmitted by a beacon transmitter installed at a predetermined position, and identify the ID of the beacon transmitter. The robot 2 may further include a beacon analysis section, and the beacon analysis section may identify the position of the beacon transmitter by analyzing the radio wave intensity of the beacon signal. Therefore, the autonomous robot 1 may regard the ID of the beacon transmitter as the marker ID, and the position of the beacon transmitter as the position of the marker, and may be applied to the above-described embodiment.

Note that a program for realizing the functions of the device described in this embodiment may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed. Accordingly, the various processes described above in this embodiment may be performed. Note that the "computer system" here may include hardware such as an OS and peripheral devices. Furthermore, the term "computer system" includes the homepage providing environment (or display environment) if a WWW system is used. Furthermore, "computer-readable recording media" refers to flexible disks, magneto-optical disks, ROMs, writable non-volatile memories such as flash memories, portable media such as CD-ROMs, hard disks built into computer systems, etc. storage device.

Furthermore, "computer-readable recording medium" refers to volatile memory (for example, DRAM (Dynamic It also includes those that retain programs for a certain period of time, such as Random Access Memory). Further, the program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in a transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Moreover, the above program may be for realizing a part of the above-mentioned functions. Furthermore, it may be a so-called difference file (difference program) that realizes the above-mentioned functions in combination with a program already recorded in the computer system.

Although the embodiments of the present invention have been described above with reference to the drawings, the specific configuration is not limited to these embodiments, and various modifications may be made without departing from the spirit of the present invention. It will be done.

Claims (17)

a moving mechanism;
A photography department that photographs the surrounding space,
a marker recognition unit that recognizes a predetermined marker included in a photographed image photographed by the photographing unit;
and a movement control unit that controls movement by the movement mechanism based on the recognized marker. The robot according to claim 1, wherein the movement control unit prohibits entry by movement based on the recognized marker. The robot according to claim 1 or 2, wherein the movement control unit limits the speed of the movement based on the recognized marker. The robot according to any one of claims 1 to 3, wherein the movement control unit controls the movement based on the recognized installation position of the marker. The robot according to claim 4, wherein the movement control unit sets a restricted range based on the installation position and limits the movement within the restricted range. The robot according to claim 5, wherein the movement control section sets a predetermined range behind the installation position or around the installation position as the limited range. The robot according to any one of claims 4 to 6, wherein the movement control unit limits the movement based on the plurality of recognized installation positions when a plurality of the markers are recognized. The robot according to claim 7, wherein the movement control unit limits the movement based on a line segment connecting a recognized installation position of the first marker and a recognized installation position of the second marker. The robot according to any one of claims 1 to 8, wherein the movement control unit controls the movement based on the recognized type of the marker. The robot according to any one of claims 1 to 9, wherein the movement control unit controls the movement based on recorded markers. The robot according to claim 10, wherein the movement control unit controls the movement based on the recorded marker when the marker is not recognized in the captured image. a spatial data generation unit that generates spatial data that recognizes the space based on a photographed image photographed by the photographing unit;
a visualization data generation unit that generates visualization data that visualizes spatial elements included in the space based on the generated spatial data;
The robot according to any one of claims 1 to 11, further comprising: a visualization data providing unit that provides the generated visualization data to a user terminal. further comprising a designation acquisition unit that acquires a designation of an area included in the provided visualization data from the user terminal,
The robot according to claim 12, wherein the spatial data generation unit re-recognizes the space based on the photographed image re-photographed in the acquired area related to the designation. further comprising a status information acquisition unit that acquires status information indicating the status of the destination during the movement,
The robot according to any one of claims 1 to 13, wherein the movement control unit controls the movement further based on the state information. a marker information storage unit that stores the position of the marker;
a first event detection unit that detects a first event;
a second event detection unit that detects a second event;
further comprising an action execution unit that executes the action;
When the first event is detected, the marker is moved to the vicinity of the position, and when the second event is detected, the marker is moved to at least one of the marker, the first event, and the second event. 15. A robot according to any one of claims 1 to 14, which performs the corresponding action. a photographing step of photographing the surrounding space;
a marker recognition step of recognizing a predetermined marker included in the photographed image photographed in the photographing step;
A robot control method, comprising: a movement control step of controlling movement by a movement mechanism based on the recognized marker. to the computer,
A shooting function that takes pictures of the surrounding space,
a marker recognition function that recognizes a predetermined marker included in a photographed image photographed in the photographing function;
A robot control program for realizing a movement control function that controls movement by a movement mechanism based on the recognized marker.