WO2019188697A1

WO2019188697A1 - Autonomous action robot, data supply device, and data supply program

Info

Publication number: WO2019188697A1
Application number: PCT/JP2019/011814
Authority: WO
Inventors: 要林; 博教小川; 秀哉南地
Original assignee: ＧｒｏｏｖｅＸ株式会社
Priority date: 2018-03-27
Filing date: 2019-03-20
Publication date: 2019-10-03

Abstract

An autonomous action robot comprises: a movement mechanism; an imaging unit to image the surrounding space; a space data generation unit to generate space data for a recognized space on the basis of the image imaged by the imaging unit; a visualization data generation unit to generate visualization data that visualizes space elements contained in the space on the basis of the generated space data; and a visualization data supply unit to supply the generated visualization data.

Description

Autonomous behavior robot, data providing apparatus, and data providing program

The present invention relates to an autonomous behavior robot, a data providing apparatus, and a data providing program.

Conventionally, there is a robot that takes an image with a camera while moving autonomously inside a house, recognizes the indoor space from the captured image, and sets the movement path based on the recognized space and moves indoors ( For example, see Patent Document 1).

JP 2016-103277 A

The robot recognizes an element related to a space such as a shape or a position (hereinafter referred to as a “space element”) such as a room, a wall or indoor furniture, a home appliance, a houseplant, or a luggage based on the photographed image. When shooting conditions such as a shooting position and a shooting angle are not appropriate, the robot may erroneously recognize a spatial element. Due to misrecognition, the robot moves differently from human speculation.

The present invention has been made in view of the above circumstances, and in one embodiment, an autonomous behavior robot, a data providing apparatus, and a data providing program capable of confirming a spatial element recognized by a robot are provided. One purpose is to provide.

(1) In order to solve the above-described problem, the autonomous behavior robot according to the embodiment includes a moving mechanism, a photographing unit that photographs a surrounding space, and the space based on a photographed image photographed by the photographing unit. A spatial data generation unit that generates recognized spatial data, a visualization data generation unit that generates visualization data that visualizes spatial elements included in the space, based on the generated spatial data, and the generated visualization data A visualization data providing unit for providing

(2) The autonomous behavior robot according to the embodiment further includes an instruction acquisition unit that acquires designation of a region included in the provided visualization data, and the spatial data generation unit relates to the obtained designation Spatial data recognizing the space in the region is regenerated.

(3) In the autonomous behavior robot according to the embodiment, the visualization data generation unit reflects the recognized feature of the spatial element based on the spatial element recognized from the captured image captured by the imaging unit. The visualization data is generated.

(4) In the autonomous behavior robot according to the embodiment, the visualization data generation unit may be configured to display the visualization data in which a color attribute is given to the space element based on color information of a captured image captured by the imaging unit. Generate.

(5) In the autonomous behavior robot according to the embodiment, the visualization data generation unit generates the visualization data in which a fixed spatial element is distinguished from a movable spatial element.

(6) In the autonomous behavior robot according to the embodiment, the visualization data generation unit distinguishes the fixed spatial element and the movable spatial element based on a temporal change of the spatial data. The visualization data is generated.

(7) In the autonomous behavior robot according to the embodiment, the visualization data generation unit generates the visualization data obtained by visualizing the spatial data included in a predetermined range from a position moved by the moving mechanism.

(8) Moreover, in the autonomous behavior type robot of the embodiment, the robot further includes a point cloud data generation unit that generates three-dimensional point cloud data based on a captured image captured by the imaging unit, and the spatial data generation unit includes: The spatial data is generated based on the generated point cloud data.

(9) In the autonomous behavior type robot according to the embodiment, the spatial data generation unit generates the spatial data by specifying an outline of the spatial element in the point cloud data, and the visualization data generation unit includes: The visualization data in which the spatial element is visualized based on the identified outline is generated.

(10) In order to solve the above-described problem, the data providing apparatus according to the embodiment generates visualization data in which the spatial elements included in the space are visualized based on the spatial data obtained by the robot recognizing the space. A visualization data generation unit; and a visualization data provision unit that provides the generated visualization data.

(11) Further, the data providing apparatus according to the embodiment includes a designation obtaining unit that obtains designation of an area included in the provided visualization data, and the space in the area related to the designation obtained with respect to the robot. And an instruction unit for instructing recognition.

(12) In order to solve the above-described problem, the data providing program according to the embodiment is a visualization data obtained by visualizing a spatial element included in the space based on the spatial data obtained by the robot recognizing the space. A visualization data generation function for generating the visualization data and a visualization data provision function for providing the generated visualization data are realized.

According to one embodiment, it is possible to provide an autonomous behavior robot, a data providing apparatus, and a data providing program that allow a person to confirm a spatial element recognized by the robot.

It is a block diagram which shows an example of the software structure of the autonomous behavior type robot in embodiment. It is a block diagram which shows an example of the hardware constitutions of the autonomous behavior type robot in embodiment. It is a flowchart which shows an example of operation | movement of the autonomous behavior type robot control program in embodiment. It is a flowchart which shows another example of operation | movement of the autonomous behavior type robot control program in embodiment. It is a figure which shows an example of the display of the user terminal in embodiment. It is a figure which shows an example of the display of the user terminal in embodiment. It is a figure which shows an example of the display of the user terminal in embodiment. It is a figure which shows an example of the display of the user terminal in embodiment. It is a figure which shows an example of the display of the user terminal in embodiment. It is a figure which shows an example of the display of the user terminal in embodiment.

Hereinafter, an autonomous behavior robot, a data providing apparatus, and a data providing program according to an embodiment of the present invention will be described in detail with reference to the drawings. Generally, a robot has various sensors such as a camera and a microphone, and recognizes surrounding conditions by comprehensively determining information obtained from these sensors. In order for the robot to move, it is necessary to recognize various objects in the space and determine the moving route. However, the moving route may not be appropriate because the object cannot be recognized correctly. Due to misrecognition, for example, even if a person thinks that there is a sufficiently large space, the robot may recognize that there is an obstacle and that only a narrow range can move. When a recognition trap occurs between a person and a robot in this way, the robot will behave against the expectation of the person, and the person will feel stress. Therefore, in order to reduce the discrepancies between human and robot recognition, the autonomous behavior robot of the present invention visualizes its recognition state and provides it to the person, and performs recognition processing again on the point indicated by the person. Can be done.

First, the software configuration of the autonomous behavior type robot 1 will be described with reference to FIG. FIG. 1 is a block diagram illustrating an example of a software configuration of the autonomous behavior robot 1 according to the embodiment.

1, the autonomous behavior type robot 1 includes a data providing device 10 and a robot 2. The data providing device 10 and the robot 2 are connected by communication and function as an autonomous behavior type robot 1 (system for causing the robot 2 to autonomously act). The robot 2 is a mobile robot having a photographing unit 21 and a moving mechanism 29. The data providing apparatus 10 has functions of a first communication control unit 11, a point cloud data generation unit 12, a spatial data generation unit 13, a visualization data generation unit 14, a photographing target recognition unit 15, and a second communication control unit 16. The first communication control unit 11 has functions of a captured image acquisition unit 111, a spatial data providing unit 112, and an instruction unit 113. The second communication control unit 16 has the functions of the visualization data providing unit 161 and the designation acquiring unit 162. Each function of the data providing device 10 of the autonomous behavior robot 1 in the present embodiment will be described as a functional module realized by a data providing program (software) that controls the data providing device 10.

The data providing apparatus 10 is an apparatus that can execute a part of the functions of the autonomous behavior robot 1. For example, the data providing apparatus 10 is installed in a place physically close to the robot 2, communicates with the robot 2, and This is an edge server that distributes the processing load. In the present embodiment, the autonomous behavior robot 1 will be described as being configured by the data providing device 10 and the robot 2, but the function of the data providing device 10 is included in the function of the robot 2. Also good. The robot 2 is a robot that can move based on the spatial data, and is a mode of the robot in which the movement range is determined based on the spatial data. The data providing apparatus 10 may be configured with one casing or may be configured with a plurality of casings.

The first communication control unit 11 controls a communication function with the robot 2. The communication method with the robot 2 is arbitrary, and for example, wireless LAN (Local Area Network), Bluetooth (registered trademark), near field communication such as infrared communication, wired communication, or the like can be used. Each function of the captured image acquisition unit 111, the spatial data providing unit 112, and the instruction unit 113 included in the first communication control unit 11 communicates with the robot 2 using a communication function controlled by the first communication control unit 11.

The captured image acquisition unit 111 acquires a captured image captured by the imaging unit 21 of the robot 2. The imaging unit 21 is provided in the robot 2 and can change the imaging range as the robot 2 moves.

The photographing unit 21 can be composed of one or a plurality of cameras. For example, when the photographing unit 21 is a stereo camera composed of two cameras, the photographing unit 21 can three-dimensionally photograph a spatial element that is a photographing target from different photographing angles. The imaging unit 21 is a video camera using an image sensor such as a CCD (Charge-Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The shape of the spatial element can be measured by photographing the spatial element with two cameras (stereo cameras). Further, the photographing unit 21 may be a camera using ToF (Time of Flight) technology. In the ToF camera, the shape of the spatial element can be measured by irradiating the spatial element with modulated infrared light and measuring the distance to the spatial element. The photographing unit 21 may be a camera using a structured light. A structured light is a light that projects light in a stripe or lattice pattern onto a spatial element. The imaging unit 21 can measure the shape of the spatial element from the distortion of the projected pattern by imaging the spatial element from a different angle from the structured light. The imaging unit 21 may be any one of these cameras or a combination of two or more.

Further, the photographing unit 21 is attached to the robot 2 and moves in accordance with the movement of the robot 2. However, the photographing unit 21 may be installed separately from the robot 2.

The captured image captured by the capturing unit 21 is provided to the captured image acquisition unit 111 in a communication method corresponding to the first communication control unit 11. The captured image is temporarily stored in the storage unit of the robot 2, and the captured image acquisition unit 111 acquires the captured image temporarily stored in real time or at a predetermined communication interval.

The spatial data providing unit 112 provides the spatial data generated by the spatial data generating unit 13 to the robot 2. Spatial data is data obtained by converting spatial elements recognized by the robot in the space where the robot 2 exists. Spatial data is expressed as, for example, a combination of map information indicating a room layout and spatial element information indicating the shape and arrangement of a spatial element (object) such as furniture. The robot 2 can move within a range determined by the spatial data. That is, the spatial data functions as a map for determining the movable range in the robot 2. The robot 2 is provided with spatial data from the spatial data providing unit 112. For example, the spatial data can include position data of spatial elements such as walls, furniture, electrical appliances, and steps that the robot 2 cannot move. The robot 2 can determine whether or not the robot 2 can move based on the provided spatial data. Further, the robot 2 may be able to recognize whether or not an ungenerated range is included in the spatial data. Whether or not an ungenerated range is included can be determined, for example, based on whether or not a space having no spatial element is included in part of the spatial data.

The instruction unit 113 instructs the robot 2 to shoot based on the spatial data generated by the spatial data generation unit 13. Since the spatial data generation unit 13 creates spatial data based on the captured image acquired by the captured image acquisition unit 111, for example, when creating indoor spatial data, spatial data is not generated for a portion that has not been captured. May be included. Further, if the captured image is unclear, noise may be included in the created spatial data, and an inaccurate part may be included in the spatial data. When there is an ungenerated part in the spatial data, the instructing unit 113 may issue an imaging instruction for the ungenerated part. In addition, when the spatial data includes an inaccurate portion, the instructing unit 113 may instruct the imaging for the inaccurate portion. The instruction unit 113 may voluntarily instruct photographing based on the spatial data. Note that the instruction unit 113 may instruct photographing based on an explicit instruction from a user who has confirmed visualization data (described later) generated based on the spatial data. The user can recognize the space and generate the spatial data by designating the area included in the visualization data and instructing the robot 2 to perform photographing.

The point cloud data generation unit 12 generates three-dimensional point cloud data of spatial elements based on the captured image acquired by the captured image acquisition unit 111. The point cloud data generation unit 12 generates point cloud data by converting a spatial element included in the captured image into a three-dimensional set of points in a predetermined space. As described above, the spatial elements are room walls, steps, doors, furniture placed in the room, home appliances, luggage, houseplants, and the like. Since the point cloud data generation unit 12 generates point cloud data based on the captured image of the spatial element, the point cloud data represents the shape of the surface of the captured spatial element. The photographed image is generated by the photographing unit 21 of the robot 2 photographing at a predetermined photographing angle at a predetermined photographing position. Therefore, when the robot 2 images a spatial element such as furniture from the front position, point cloud data cannot be generated for the shape of the back side of the furniture that has not been imaged, and the robot 2 moves to the back side of the furniture. Even if there is a possible space, the robot 2 cannot recognize it. On the other hand, when the robot 2 moves and photographs the furniture from the side photographing position, point cloud data can be generated for the shape of the back side of the space element such as furniture, and the robot 2 grasps the space relatively accurately. It becomes possible to do.

The spatial data generation unit 13 generates spatial data that determines the movable range of the robot 2 based on the point cloud data of the spatial elements generated by the point cloud data generation unit 12. Since the spatial data is generated based on the point cloud data in the space, the spatial element included in the spatial data also has three-dimensional coordinate information. The coordinate information may include point position, length (including height), area, or volume information. The robot 2 can determine a movable range based on the position information of the spatial elements included in the generated spatial data. For example, when the robot 2 has a moving mechanism 29 that horizontally moves on the floor surface, the robot 2 has a level difference from the floor surface that is a spatial element in the spatial data at a predetermined height or higher (for example, 1 cm or higher). In some cases, it can be determined that movement is impossible. On the other hand, in the spatial data, when the table top plate or bed, which is a spatial element, has a predetermined height from the floor surface, the robot 2 has a height from the floor surface of a predetermined height or higher (for example, 60 cm or higher). ) Is determined as a movable range in consideration of the clearance with its own height. In addition, the robot 2 determines, in the spatial data, a range in which a space between the wall and the furniture, which is a spatial element, is a predetermined width or more (for example, 40 cm or more) as a movable range in consideration of a clearance with its own width. To do.

The spatial data generation unit 13 may set attribute information for a predetermined area in the space. The attribute information is information that defines the movement condition of the robot 2 for a predetermined area. The movement condition is, for example, a condition that defines a clearance from a space element that the robot 2 can move. For example, when the normal movement condition in which the robot 2 can move is that the clearance is 30 cm or more, attribute information in which the clearance for a predetermined area is 5 cm or more can be set. In addition, as the movement condition set in the attribute information, information for restricting the movement of the robot may be set. The movement restriction is, for example, movement speed restriction or entry prohibition. For example, attribute information that reduces the moving speed of the robot 2 may be set in an area where the clearance is small or an area where people exist. Further, the movement condition set in the attribute information may be determined by the floor material of the area. For example, the attribute information may be set to change the operation (traveling speed or traveling means) of the moving mechanism 29 when the floor is a cushion floor, flooring, tatami mat, or carpet. In the attribute information, the above conditions may be set at the charging spot where the robot 2 can move and charge, the step where the movement of the robot 2 is unstable and the movement is restricted, or the end of the carpet. . Note that the area in which the attribute information is set may be understood by the user, for example, by changing the display method in the visualization data described later.

The spatial data generation unit 13 performs, for example, a Hough transform on the point cloud data generated by the point cloud data generation unit 12 to extract a common line, curve, or other graphic in the point cloud data. Spatial data is generated by the contour of the spatial element to be expressed. The Hough transform is a coordinate transformation method for extracting a figure that passes through the feature points most when the point cloud data is a feature point. Since the point cloud data expresses the shape of a spatial element such as furniture placed in a room in the point cloud, the user can determine what the spatial element is represented by the point cloud data (for example, Recognition of tables, chairs, walls, etc.) may be difficult. Since the spatial data generation unit 13 can express the outline of furniture or the like by performing the Hough transform on the point cloud data, the user can easily determine the spatial elements. The spatial data generation unit 13 converts the point cloud data generated by the point cloud data generation unit 12 into a basic shape in a spatial element (for example, a table, a chair, a wall, etc.) recognized in the image recognition. Data may be generated. By recognizing that a spatial element such as a table is a table by image recognition, the shape of the table is determined from a part of point cloud data of the spatial element (for example, point cloud data when the table is viewed from the front). It can be predicted accurately. The spatial data generation unit 13 can generate spatial data that accurately grasps the spatial elements by combining point cloud data and image recognition.
The basic shape may be a predetermined shape (for example, a square, a circle, a triangle, a rectangular parallelepiped, a cone, or the like) associated with a spatial element. The spatial data generation unit 13 may generate spatial data by deforming the basic shape by enlarging or reducing the basic shape. In the spatial data, the shape of the spatial element may be expressed by combining plural or plural types of basic shape objects. In addition, the visualization data generation unit 14 to be described later visualizes the feature of the spatial element by synthesizing the texture associated with the spatial element with the spatial data after processing such as the basic shape or deformation. Data may be generated.

The spatial data generation unit 13 generates spatial data based on point cloud data included in a predetermined range from the position where the robot 2 has moved. The predetermined range from the position to which the robot 2 has moved includes the position where the robot 2 has actually moved, and may be, for example, a range at a distance such as 30 cm from the position to which the robot 2 has moved. Since the point cloud data is generated based on the captured image captured by the capturing unit 21 of the robot 2, the captured image may include a spatial element at a position away from the robot 2. When the imaging unit 21 is far away from the space element, there may be a portion where the robot 2 is not moved due to the presence of an uncaptured portion or the presence of an obstacle that is not captured. In addition, when a captured image includes a spatial element that is far from the imaging unit 21 such as a hallway, the spatial element extracted at the feature point may be distorted. In addition, when the shooting distance is large, the spatial elements included in the shot image are small, and the accuracy of the point cloud data may be low. The spatial data generation unit 13 may generate spatial data that does not include a spatial element with low accuracy or a distorted spatial element by ignoring feature points that are greatly separated by a predetermined distance or more. The spatial data generation unit 13 deletes point cloud data outside a predetermined range from the position where the robot 2 has moved to generate spatial data, thereby preventing the occurrence of an enclave where no data actually exists. In addition, it is possible to generate spatial data that does not include a range in which the robot 2 cannot move and has high data accuracy. Further, it is possible to prevent enclave drawing in the visualization data generated from the spatial data, and to improve visibility.

The visualization data generation unit 14 generates visualization data that is visualized based on the spatial data generated by the spatial data generation unit 13 so that a person can intuitively determine the spatial elements included in the space. As described above, the spatial data is data including the spatial elements recognized by the autonomous behavior robot 1, whereas the visualization data is the spatial data recognized by the autonomous behavior robot 1 by the user. This is data for visual recognition. Spatial data may include misrecognized spatial elements. By visualizing the spatial data, it becomes easy for a person to confirm the recognition state (presence / absence of misrecognition) of the spatial element in the autonomous behavior robot 1.

Visualized data is data that can be displayed on the display device. The visualization data is a so-called floor plan, and a spatial element recognized as a table, chair, sofa, or the like is included in an area surrounded by a spatial element recognized as a wall. The visualization data generation unit 14 generates the shape of furniture or the like formed in the graphic extracted by the Hough transform as visualization data expressed by RGB data, for example. The spatial data generation unit 13 generates visualization data in which the plane drawing method is changed based on the direction of the plane in three dimensions of the spatial element. The direction of the three-dimensional plane of the spatial element is, for example, the normal direction of the plane formed by the figure generated in the point cloud data by Hough transforming the point cloud data generated in the point cloud data generation unit 12 It is. The visualization data generation unit 14 generates visualization data in which the plane drawing method is changed according to the normal direction. The drawing method is, for example, a hue attributed to a plane, a color attribute such as brightness or saturation, a pattern imparted to a plane, a texture, or the like. For example, when the normal line of the plane is the vertical direction (the plane is the horizontal direction), the visualization data generation unit 14 renders the plane with high brightness and draws in a bright color. On the other hand, when the normal of the plane is the horizontal direction (the plane is the vertical direction), the visualization data generation unit 14 renders the plane with low brightness and draws in a dark color. By changing the drawing method of the plane, the shape of the furniture or the like can be represented in three dimensions, and the user can easily confirm the shape of the furniture or the like. The visualization data may include coordinate information (referred to as “visualization coordinate information”) in the visualization data associated with the coordinate information of each spatial element included in the spatial data. Since the visualization coordinate information is associated with the coordinate information, the point in the visualization coordinate information corresponds to the point in the actual space, and the surface in the visualization coordinate information corresponds to the surface in the actual space. Therefore, when the user specifies the position of a certain point in the visualization data, the position of the point in the actual room corresponding to the point can be specified. In addition, a conversion function for converting the coordinate system may be prepared so that the coordinate system in the visualization data and the coordinate system in the spatial data can be mutually converted. Of course, the coordinate system in the visualization data and the coordinate system in the actual space may be mutually convertible.

The visualization data generation unit 14 generates visualization data as stereoscopic (3D (Dimensions)) data. The visualization data generation unit 14 may generate the visualization data as planar (2D) data. By generating the visualization data in 3D, the user can easily check the shape of the furniture or the like. The visualization data generation unit 14 may generate the visualization data in 3D when the spatial data generation unit 13 generates sufficient data to generate the visualization data in 3D. The visualization data generation unit 14 may generate the visualization data in 3D based on the 3D viewpoint position (viewpoint height, viewpoint elevation angle, etc.) designated by the user. By making it possible to specify the viewpoint position, the user can easily check the shape of furniture or the like. Further, the visualization data generation unit 14 may generate visualization data in which the wall or ceiling of the room is colored only for the back wall and the front wall or ceiling is transparent (not colored). By making the near wall transparent, the user can easily confirm the shape of the furniture or the like arranged at the end (inside the room) of the near wall.

The visualization data generation unit 14 generates visualization data to which a color attribute corresponding to the captured image acquired by the captured image acquisition unit 111 is added. For example, when the captured image includes woodgrain furniture and the color of the wood (for example, brown) is detected, the visualization data generation unit 14 gives a color approximate to the detected color to the extracted furniture figure. Generate visualization data. By assigning a color attribute according to the photographed image, the user can easily check the type of furniture or the like.

The visualization data generation unit 14 generates visualization data in which the drawing method between the fixed object that is fixed and the moving object that moves is changed. The fixed object is, for example, a wall of a room, a step, furniture that is fixed, and the like. The moving object is, for example, a chair, a trash can, furniture with casters, or the like. Moreover, the moving object may include a temporary object temporarily placed on the floor, such as a luggage or a bag. The drawing method is, for example, a hue attributed to a plane, a color attribute such as brightness or saturation, a pattern imparted to a plane, a texture, or the like.

 The classification of fixed, moving or temporary items can be identified by the time period existing at the location. For example, the spatial data generation unit 13 identifies the classification of the fixed object, the moving object, or the temporary object based on the time-dependent change of the point cloud data generated by the point cloud data generation unit 12, and obtains the spatial data. Generate. For example, the spatial data generation unit 13 determines that the spatial element is a fixed object when the spatial element has not changed from the difference between the spatial data generated at the first time and the spatial data generated at the second time. to decide. Further, the spatial data generation unit 13 may determine that the spatial element is a moving object when the position of the spatial element changes from the difference of the spatial data. In addition, the spatial data generation unit 13 may determine that the spatial element is a temporary object when the spatial element disappears or appears from the difference of the spatial data. The visualization data generation unit 14 changes the drawing method based on the classification identified by the spatial data generation unit 13. The drawing method change includes, for example, color coding, addition of hatching, addition of a predetermined mark, and the like. For example, the spatial data generation unit 13 may display a fixed object in black, a moving object in blue, or a temporary object in yellow. The spatial data generation unit 13 generates spatial data by identifying a classification of a fixed object, a moving object, or a temporary object. The visualization data generation unit 14 may generate visualization data in which the drawing method is changed based on the classification identified by the spatial data generation unit 13. Further, the spatial data generation unit 13 may generate visualization data obtained by changing the drawing method of the spatial element recognized by the image recognition.

The visualization data generation unit 14 can generate visualization data in a plurality of divided areas. For example, the visualization data generation unit 14 generates visualization data for each of the spaces partitioned by walls such as a living room, a bedroom, a dining room, and a hallway as one room. By generating visualization data for each room, for example, generation of spatial data or visualization data can be performed separately for each room, and generation of spatial data or the like is facilitated. In addition, it is possible to create spatial data or the like only for an area where the robot 2 may move. The visualization data providing unit 161 provides visualization data that allows the user to select an area. For example, the visualization data providing unit 161 may enlarge the visualization data of the area selected by the user or provide detailed visualization data of the area selected by the user.

The imaging target recognition unit 15 recognizes the spatial element based on the captured image acquired by the captured image acquisition unit 111. Spatial element recognition can be performed by using an image recognition engine that determines what a spatial element is based on, for example, image recognition results accumulated in machine learning. The spatial element image recognition can be recognized, for example, in the shape, color, pattern, or the like of the spatial element. For example, the imaging target recognition unit 15 may be able to recognize a spatial element by using an image recognition service provided in a cloud server (not shown). The visualization data generation unit 14 generates visualization data in which the drawing method is changed in accordance with the spatial element recognized by the imaging target recognition unit 15. For example, when the image-recognized space element is a sofa, the visualization data generation unit 14 generates visualization data in which a texture having a cloth texture is added to the space element. When the image-recognized space element is a wall, the visualization data generation unit 14 may generate visualization data with a wallpaper color attribute (for example, white). By performing such visualization processing, the user can intuitively grasp the recognition state of the space in the robot 2.

The second communication control unit 16 controls communication with the user terminal 3 owned by the user. The user terminal 3 is, for example, a smartphone, a tablet PC, a notebook PC, a desktop PC, or the like. The communication method with the user terminal 3 is arbitrary, and for example, wireless LAN, Bluetooth (registered trademark), short-range wireless communication such as infrared communication, or wired communication can be used. Each function of the visualization data providing unit 161 and the designation acquiring unit 162 included in the second communication control unit 16 communicates with the user terminal 3 using a communication function controlled by the second communication control unit 16.

The visualization data providing unit 161 provides the visualization data generated by the visualization data generation unit 14 to the user terminal 3. The visualization data providing unit 161 is, for example, a Web server, and provides visualization data as a Web page to the browser of the user terminal 3. The visualization data providing unit 161 may provide visualization data to a plurality of user terminals 3. By visually recognizing the visualization data displayed on the user terminal 3, the user can confirm the range in which the robot 2 can move as a 2D or 3D display. In the visualization data, the shape of furniture or the like is drawn by a predetermined drawing method. By operating the user terminal 3, the user can switch between 2D display and 3D display, zoom in or out of the visualization data, or move the viewpoint in 3D display, for example.

The user can visually check the visualization data displayed on the user terminal 3 and can confirm the generation state of the spatial data and the attribute information of the area. The user can instruct the creation of the spatial data by designating the area in which the spatial data is not generated from the visualization data via the user terminal 3. In addition, if the user visually recognizes the visualization data displayed on the user terminal 3 and there is an area where the spatial data seems to be inaccurate, such as the shape of a spatial element such as furniture is unnatural, An area can be designated via the user terminal 3 to instruct the regeneration of the spatial data. As described above, since the visualization coordinate information in the visualization data is associated with the coordinate information in the spatial data, the area in the visualization data that is specified to be regenerated by the user can be uniquely identified as the area in the spatial data. . Based on the regenerated spatial data, the visualization data generating unit 14 regenerates the visualization data and provides it from the visualization data providing unit 161. For example, the visualization data can be regenerated based on the spatial data by converting the spatial data into corresponding visualization data according to a specified rule. Note that the generation state of the spatial data may not change, for example, the spatial element may be misrecognized in the regenerated visualization data. In that case, the user may instruct the generation of spatial data by giving an instruction to change the operation parameter of the robot 2 via the user terminal 3. The operation parameters are, for example, shooting conditions (exposure amount, shutter speed, etc.) in the shooting unit 21 in the robot 2, sensitivity of a sensor (not shown), clearance conditions when allowing the robot 2 to move, and the like. For example, the operation parameter may be included in the spatial data as area attribute information.

The visualization data generation unit 14 generates visualization data including a display of a button for instructing creation of spatial data (including “re-creation”), for example. The user terminal 3 can transmit an instruction to create spatial data to the autonomous behavior robot 1 by the user operating the displayed button. The designation acquisition unit 162 acquires the spatial data creation instruction transmitted from the user terminal 3.

The designation obtaining unit 162 obtains an instruction to create spatial data of an area designated by the user based on the visualization data provided by the visualization data providing unit 161. The designation acquisition unit 162 may acquire an instruction to set (including change) the attribute information of the area. In addition, the designation acquisition unit 162 acquires the position of the area and the direction when the robot approaches the area, that is, the direction to be photographed. Acquisition of a creation instruction can be executed, for example, in the operation of a Web page provided by the visualization data providing unit 161. Thereby, the user can grasp how the robot 2 recognizes the space, and can instruct the robot 2 to redo the recognition process according to the recognition state.

The instruction unit 113 instructs the robot 2 to shoot in the area where the creation of spatial data is instructed. The shooting in the area instructed to create the spatial data may include, for example, shooting conditions such as the coordinate position of the robot 2 (shooting unit 21), the shooting direction of the shooting unit 21, and the resolution. The spatial data generation unit 13 adds the newly created spatial data to the existing spatial data when the spatial data for which creation is instructed relates to an ungenerated region, and the spatial data generation unit 13 If the instructed spatial data relates to re-creation, spatial data obtained by updating the existing spatial data is generated.

As described above, FIG. 1 illustrates the case where the autonomous behavior robot 1 includes the data providing device 10 and the robot 2, but the function of the data providing device 10 is included in the function of the robot 2. It may be a thing. For example, the robot 2 may include all the functions of the data providing apparatus 10. For example, the data providing apparatus 10 may temporarily substitute a function when the robot 2 has insufficient processing capability.

Further, in the present embodiment, “acquisition” may be acquired by the acquiring entity actively, or may be acquired passively by the acquiring entity. For example, the designation acquisition unit 162 may acquire the designation by receiving a spatial data creation instruction transmitted from the user terminal 3 by the user, and the user stores the data in a storage area (not shown) that is not shown. An instruction to create the spatial data may be obtained by reading from the storage area.

In addition, the autonomous behavior robot 1 includes a first communication control unit 11, a point cloud data generation unit 12, a spatial data generation unit 13, a visualization data generation unit 14, an imaging target recognition unit 15, a second communication control unit 16, and imaging. The functional units of the image acquisition unit 111, the spatial data provision unit 112, the instruction unit 113, the visualization data provision unit 161, and the designation acquisition unit 162 are examples of functions of the autonomous behavior robot 1 in the present embodiment. The functions of the autonomous behavior robot 1 are not limited. For example, the autonomous behavior type robot 1 does not have to have all the functional units described above, and may have a part of functional units. Moreover, the autonomous behavior type robot 1 may have a function part other than the above.

In addition, the above-described functional units included in the autonomous behavior robot 1 have been described as being realized by software as described above. However, at least one of the functions of the autonomous behavior robot 1 may be realized by hardware.

Also, any one of the above functions that the autonomous behavior robot 1 has may be implemented by dividing one function into a plurality of functions. Further, any two or more functions of the autonomous behavior robot 1 may be integrated into one function. That is, FIG. 1 represents the functions of the autonomous behavior robot 1 by function blocks, and does not indicate that each function is configured by a separate program file, for example.

Further, the autonomous behavior robot 1 may be a device realized by a single housing or a system realized by a plurality of devices connected via a network or the like. For example, the autonomous behavior robot 1 may be realized by a virtual device such as a cloud service provided by a cloud computing system, part or all of its functions. That is, the autonomous behavior robot 1 may realize at least one or more of the above functions in another device. The autonomous behavior robot 1 may be a general-purpose computer such as a tablet PC, or may be a dedicated device with limited functions such as a car navigation device.

Further, the autonomous behavior type robot 1 may realize part or all of the functions in the robot 2 or the user terminal 3.

Next, the hardware configuration of the autonomous behavior robot 1 will be described with reference to FIG. FIG. 2 is a block diagram illustrating an example of a hardware configuration of the autonomous behavior robot 1 according to the embodiment.

The autonomous behavior type robot 1 includes a CPU (Central Processing Unit) 101, a RAM (Random Access Memory) 102, a ROM (Read Only Memory) 103, a touch panel 104, and a communication I / F (Interface) 105. The autonomous behavior type robot 1 is a device that executes the autonomous behavior type robot control program described in FIG.

The CPU 101 controls the autonomous behavior robot 1 by executing the autonomous behavior robot control program stored in the RAM 102 or the ROM 103. The autonomous behavior robot control program is acquired from, for example, a recording medium that records the autonomous behavior robot control program or a program distribution server via a network, installed in the ROM 103, read from the CPU 101, and executed.

The touch panel 104 has an operation input function and a display function (operation display function). The touch panel 104 enables operation input using a fingertip or a touch pen to the user of the autonomous behavior robot 1. The autonomous behavior robot 1 in this embodiment will be described using a touch panel 104 having an operation display function. However, the autonomous behavior robot 1 has a display device having a display function and an operation input device having an operation input function separately. You may have. In that case, the display screen of the touch panel 104 can be implemented as a display screen of the display device, and the operation of the touch panel 104 can be implemented as an operation of the operation input device. Note that the touch panel 104 may be realized in various forms such as a head mount type, a glasses type, and a wristwatch type display.

The communication I / F 105 is a communication I / F. The communication I / F 105 executes short-range wireless communication such as a wireless LAN, a wired LAN, and infrared rays. Although only the communication I / F 105 is illustrated in FIG. 2 as the communication I / F, the autonomous behavior robot 1 may have each communication I / F in a plurality of communication methods.

Next, the operation related to the visualization data provision of the robot control program will be described with reference to FIG. FIG. 3 is a flowchart illustrating an example of the operation of the robot control program in the embodiment. In the following description of the flowcharts, it is assumed that the execution subject of the operation is the autonomous behavior type robot 1, but each operation is executed in each function of the autonomous behavior type robot 1 described above.

In FIG. 3, the autonomous behavior robot 1 determines whether a captured image has been acquired (step S11). Whether or not a captured image has been acquired can be determined by whether or not the captured image acquisition unit 111 has acquired a captured image from the robot 2. The determination as to whether or not a captured image has been acquired is made on a per-process basis for captured images. For example, since the moving image is continuously transmitted from the robot 2 when the captured image is a moving image, the determination as to whether or not the captured image has been acquired is based on whether the number of frames or the data amount of the acquired moving image is a predetermined value It can be done by whether or not. The acquired captured image may be acquired by the mobile robot as a main component for transmitting the captured image or may be acquired by the captured image acquisition unit 111 as the main component for taking the captured image from the mobile robot. When it is determined that a captured image has not been acquired (step S11: NO), the autonomous behavior robot 1 repeats the process of step S11 and waits for a captured image to be acquired.

On the other hand, when it is determined that a captured image has been acquired (step S12: NO), the autonomous behavior robot 1 generates point cloud data (step S12). For the generation of the point cloud data, the point cloud data generation unit 12 detects, for example, a point where the luminance change in the photographed image is larger than a predetermined luminance change threshold as a feature point, and the detected feature point is three-dimensionally generated. Can be performed by giving the coordinates of For example, the feature point may be detected by performing a differentiation process on the captured image and detecting a portion where the gradation change is larger than a predetermined gradation change threshold. Moreover, you may perform the provision of the coordinate with respect to a feature point by detecting the same feature point image | photographed from different imaging | photography angles. Whether or not a captured image is acquired in step S11 can be determined based on whether or not captured images captured from a plurality of directions have been acquired.

After executing the process of step S12, the autonomous behavior type robot 1 generates spatial data (step S13). The generation of the spatial data can be executed by the spatial data generation unit 13 by, for example, Hough transforming the point cloud data.

After executing the process of step S13, the autonomous behavior robot 1 provides the generated spatial data to the robot 2. The provision of the spatial data to the robot 2 may be sequentially provided as the spatial data is generated as shown in FIG. 3, or may be provided asynchronously with the processing shown in steps S11 to S18. Good. The robot 2 provided with the spatial data can grasp the movable range based on the spatial data.

After executing the process of step S14, the autonomous behavior robot 1 determines whether or not to recognize a spatial element (step S15). The determination of whether or not to recognize a spatial element can be executed by, for example, setting the imaging target recognition unit 15 whether or not to recognize a spatial element. Even if it is determined that the spatial element is recognized, if the recognition fails, it may be determined that the spatial element is not recognized.

If it is determined that the spatial element is recognized (step S15: YES), the autonomous behavior robot 1 generates first visualization data (step S16). The generation of the first visualization data can be executed in the visualization data generation unit 14. The first visualization data is visualization data generated after the imaging object recognition unit 15 recognizes a spatial element. For example, when the imaging target recognition unit 15 determines that the spatial element is a table, the visualization data generation unit 14 does not have point cloud data even if the top surface of the table is not captured. Visualization data can be generated as if the top surface is flat. Further, when it is determined that the spatial element is a wall, the visualization data generation unit 14 can generate visualization data by assuming that a portion that has not been shot is also a plane.

If it is determined that the spatial element is not recognized (step S15: NO), the autonomous behavior robot 1 generates second visualization data (step S17). The generation of the second visualization data can be executed in the visualization data generation unit 14. The second visualization data is visualization data that is generated based on the point cloud data and the spatial data generated from the captured image without the imaging target recognition unit 15 recognizing the spatial element. The autonomous behavior robot 1 can reduce the processing load by not performing the spatial element recognition process.

After executing the process of step S16 or the process of step S17, the autonomous behavior type robot 1 provides visualization data (step S18). The provision of the visualization data is executed when the visualization data provision unit 161 provides the visualization data generated by the visualization data generation unit 14 to the user terminal 3. The autonomous behavior type robot 1 may generate and provide visualization data in response to a request from the user terminal 3, for example. After executing the process of step S18, the autonomous behavior robot 1 ends the operation shown in the flowchart.

Next, the operation related to the space data creation instruction of the robot control program will be described with reference to FIG. FIG. 4 is a flowchart illustrating another example of the operation of the robot control program in the embodiment.

In FIG. 4, the autonomous behavior robot 1 determines whether or not a creation instruction has been acquired (step S21). The determination as to whether or not the creation instruction has been acquired can be made based on whether or not the designation acquisition unit 162 has acquired a creation instruction that instructs the user terminal 3 to create spatial data. The creation instruction is acquired, for example, when the user executes an operation for reacquisition of the spatial data from the user terminal 3. In addition, the creation instruction may be acquired when the user performs an operation of acquiring spatial data for an ungenerated area from the user terminal 3. When it is determined that the creation instruction has not been acquired (step S21: NO), the autonomous behavior robot 1 repeats the process of step S21 and waits for the creation instruction to be acquired.

On the other hand, if it is determined that a creation instruction has been acquired (step S21: YES), the autonomous behavior robot 1 executes a shooting instruction in an area instructed to create spatial data (step S22). The shooting instruction can be executed when the instruction unit 113 instructs the robot 2 to make a shooting instruction. The shooting instruction can include an area to be shot by the shooting unit 21 of the robot 2 and a shooting position. When the robot 2 that has received the imaging instruction provides the captured image to the autonomous behavior robot 1, the operation illustrated in FIG. 3 is executed, and spatial data and visualization data in the instructed area are created. After executing the process of step S22, the autonomous behavior robot 1 ends the operation shown in the flowchart.

FIG. 4 shows the operation when a creation instruction, which is an explicit instruction from the user, is acquired from the user terminal 3, but the autonomous behavior robot 1 has, for example, ungenerated spatial data. Alternatively, if the generation of high-accuracy spatial data fails, a shooting instruction may be issued without an explicit instruction from the user.

Further, the processing order in each step in the operation of the robot control program (robot control method) described in this embodiment does not limit the execution order.

Next, the display of the user terminal 3 based on the visualization data provided by the autonomous behavior robot 1 will be described with reference to FIGS. 5 to 10 are diagrams illustrating examples of display of the user terminal 3 according to the embodiment. 5 to 10 are display examples in which a Web page provided as visualization data from the visualization data providing unit 161 is displayed on the touch panel of a smartphone exemplified as the user terminal 3.

In FIG. 5, the user terminal 3 displays the 3D visualization data generated by the visualization data generation unit 14 based on the captured image of the living room captured by the robot 2. The visualization data generation unit 14 rasterizes and draws a figure such as furniture generated in the point cloud data. In FIG. 5, feature points of point cloud data are subjected to Hough transform to extract line segments (straight lines or curves), and spatial data in which the shape of furniture or the like is represented by 3D line segments is generated. In the figure, the case where the shape of furniture or the like is represented by straight lines in the x direction, the y direction, and the z direction is shown. In addition, the plane formed in the combination of straight lines in the x, y, and z directions has a normal direction of the xy plane in the vertical direction, a plane normal direction in the horizontal direction of the xz plane, y -For the z plane, the brightness of the texture applied to the plane is changed. For example, the brightness on the xy plane, which is the upper surface of the work table of the table or system kitchen, is low (the color is dark). In addition, the brightness is moderate for the xz plane and the brightness is high for the yz plane. By changing the brightness, which is a method for expressing a plane such as furniture, the user can easily recognize the shape of the furniture.

In addition, point cloud data is deleted from a wall or ceiling having a predetermined height from the floor in the figure to generate spatial data. Since the robot 2 moves on the floor surface, space data such as a ceiling is not necessary. By not generating spatial data for a range farther than a predetermined distance from the robot 2, the amount of spatial data can be reduced and the visualization data can be easily viewed.

On the user terminal 3, a “re-acquire” button 31 is displayed. When the user presses (operates) the reacquisition button 31, a creation instruction is transmitted to the autonomous behavior robot 1. When the creation instruction is transmitted, the autonomous behavior robot 1 provides the imaging instruction to the robot 2, recreates the spatial data and the visualization data based on the re-acquired captured image, and uses the visualization data as the user terminal 3. Re-offer to. That is, when the reacquisition button 31 is pressed, the visualization data of the living room displayed on the user terminal 3 is updated. After receiving the creation instruction, the autonomous behavior type robot 1 may cause the robot 2 to immediately take an action for photographing and recreate the spatial data and the visualization data at an early stage. The autonomous behavior type robot 1 does not control the movement of the robot 2 but moves the robot 2 according to parameters (however, parameters different from the attribute information) that affect the behavior of the robot 2 such as emotion parameters. An imaging instruction may be provided according to the position of 2, and spatial data and visualization data may be recreated based on the re-acquired captured image.
As described above, the robot 2 may act autonomously based on an internal parameter (for example, an emotion parameter). For example, the robot 2 may increase the emotion parameter indicating “boring” when it is determined that the robot 2 has been staying at the same place for a certain period of time. The robot 2 may start moving to an uncreated portion of the map on the condition that it is determined that “boring” has become a predetermined value or more. Therefore, the autonomous behavior robot 1 indirectly promotes the activity of the robot 2 by improving the “boring” parameter of the robot 1 to a predetermined value or more when receiving an instruction to create an uncreated part. Also good.
Further, the autonomous behavior type robot 1 may provide a shooting instruction to the robot 2 so as not to affect the movement plan of the robot 2 when the creation instruction is received. In other words, the autonomous behavior robot 1 may provide the robot 2 with an instruction to move to start shooting when the movement of the robot 2 at the time of receiving the creation instruction is partially or completely completed. Good. For example, it is assumed that the robot 2 is moving to the movement target point S, and the robot 2 receives a creation instruction during this movement. At this time, the robot 2 may move to a movement target point S or a predetermined point (for example, a hallway) from the current location to the movement target point S and then start an action based on the creation instruction.

In addition, a “2D display” button 32 is displayed on the user terminal 3. By pressing the 2D display button 32, the user terminal 3 transmits an instruction to change the 3D visualization data to 2D visualization data to the autonomous behavior robot 1. The autonomous behavior robot 1 generates visualization data that represents the generated spatial data in 2D and provides it to the user terminal 3. The display when the 2D display button 32 is pressed will be described with reference to FIG.

Note that the user can enlarge or reduce the visualization data by pinching in or out the touch panel on the user terminal 3. Further, the user can move the viewpoint of 3D display by sliding the touch panel.

FIG. 6 is a display example when the 2D display button 32 in FIG. 5 is pressed.

6A, the user terminal 3 displays visualization data obtained by converting the 2D visualization data displayed in FIG. 5 into 2D. A “3D display” button 33 is displayed on the user terminal 3. By pressing the 3D display button 33, the user terminal 3 transmits an instruction to change the 2D visualization data to 3D visualization data to the autonomous behavior robot 1. The autonomous behavior robot 1 generates visualization data that represents the generated spatial data in 3D and provides it to the user terminal 3. The display when the 3D display button 33 is pressed is the display described in FIG.

In FIG. 6 (A), a kitchen that was not displayed in 3D display is displayed in the living room. Since the 3D display is a display when viewed from the viewpoint position, a portion hidden by a wall or furniture cannot be displayed. However, by using 2D display, it is possible to display visualization data obtained by looking down on the room from above, making it easy to grasp the shape of the entire room. The boundary line 34 indicates the generation range of the spatial data. That is, the area above the boundary line 34 indicates that the captured image has not been acquired by the robot 2 and the spatial data has not been generated. When the user designates the boundary line 34 displayed on the touch panel or an area above the boundary line 34 and presses the re-acquisition button 31, the robot 2 moves to the designated area and photographs the room. The autonomous behavior type robot 1 generates spatial data of an ungenerated area based on the captured image and updates the spatial data. The update of the spatial data can be executed by, for example, matching the original spatial data and the newly generated spatial data to generate continuous spatial data. The autonomous behavior type robot 1 generates visualization data based on the updated spatial data and provides it to the user terminal 3. The house-shaped mark h shown in the figure indicates a home position where the robot 2 returns for charging.

FIG. 6B shows that the visualization data updated in the autonomous behavior robot 1 is displayed on the user terminal 3. FIG. 6B shows that a long and narrow corridor exists on the upper side of the living room, and that a Western-style room exists on the right side of the corridor. When the Western-style door is closed, the Western-style space data is not generated. For example, by displaying the position of the door, it may be displayed with a boundary line that an ungenerated area still exists. . Note that there are doors that could not be detected in the hallway, and there is actually an area above the hallway.

7 to 8 show switching between 2D display and simple 3D display. FIG. 7 shows that 2D visualization data generated in all rooms at home is displayed on the user terminal 3. When the user presses the 3D display button 33 in FIG. 7, the display is switched to the simple 3D display shown in FIG. Here, the simplified 3D display refers to a display form such as a bird's-eye view obtained by transforming the 2D display and looking down from above. The simple 3D display may include, for example, a three-dimensional display adapted to a predetermined shape in the height direction for furniture or the like. By displaying the visualization data in simple 3D, it becomes easy to grasp the shape of the entire room without increasing the data amount of the visualization data. When the 2D display button 32 shown in FIG. 8 is pressed, the display is switched to the 2D display shown in FIG.

FIG. 9 is a display example of visualization data generated in a plurality of divided areas. The user terminal 3 displays visualization data 35 by area. The area-specific visualization data 35 displays scrollable areas divided into living rooms, hallways, bedrooms, children's rooms, and the like. For example, when the user selects and presses one area, the visualization data of the pressed area is displayed. For example, FIG. 5 is visualization data when a living room is selected in the visualization data 35 by area. When the user presses “view other rooms” displayed in the lower part of FIG. 5 or the like, the area-specific visualization data 35 shown in FIG. 9 may be displayed so that other rooms can be selected.

FIG. 10 is a display example for explaining misrecognition of a spatial element depending on the photographing position of the photographing unit.

10A, the user terminal 3 displays a visualized image 37A of a shelf generated based on a photographed image photographed by the robot 2 in the direction of the arrow shown from the photographing position 36A. Since the back side of the shelf cannot be photographed from the photographing position 36A, the spatial data generation unit 13 does not correctly recognize the shape of the shelf. The shape of the erroneously recognized shelf exists up to the wall. Accordingly, the autonomous behavior robot 1 determines that it cannot move on the back side of the shelf. On the other hand, the user can easily confirm that the autonomous behavior robot 1 erroneously recognizes the shape of the shelf by visually recognizing the shape of the visualized image 37A.

FIG. 10B shows that the visualization data updated in the autonomous behavior robot 1 is displayed. In FIG. 6B, a visualized image 37B of a shelf generated based on a photographed image taken by the robot 2 in the direction indicated by the arrow from the photographing position 36B is displayed on the user terminal 3. Since the back side of the shelf can be photographed from the photographing position 36B, the spatial data generation unit 13 can correctly recognize the shape of the shelf together with the photographed image photographed from the photographing position 36A. A correctly recognized shelf shape has a gap with the wall. Therefore, it can be determined that the autonomous behavior robot 1 can move on the back side of the shelf. The user can easily confirm that the autonomous behavior robot 1 correctly recognizes the shape of the shelf by visually recognizing the shape of the visualized image 37B. Note that the user can instruct the robot 2 to take a picture by designating the position of the photographing position 36B and the photographing direction from the user terminal 3. For example, this designation method may be an operation of sliding on the screen of the user terminal 3 linearly from the position where the fingertip is touched. The starting point of the trajectory of the fingertip is set as the shooting position, and the direction in which the fingertip is slid is shot. Specify as direction. In conjunction with this operation, an arrow icon is displayed over the visualization data on the screen. Since the user knows the shape of the shelf, the user can instruct a shooting position and a shooting direction for correctly recognizing the shape of the shelf that the autonomous behavior robot 1 has erroneously recognized.

Note that a program for realizing the functions constituting the apparatus described in this embodiment is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed. Thus, the various processes described above in the present embodiment may be performed. Here, the “computer system” may include an OS and hardware such as peripheral devices. Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

Further, the “computer-readable recording medium” refers to a volatile memory (for example, DRAM (Dynamic) in a computer system serving as a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Random Access Memory)), etc. that hold a program for a certain period of time. The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what implement | achieves the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

The embodiment of the present invention has been described above with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes various modifications within the scope of the present invention. It is.

As such a modification, for example, the visualization data providing unit 161 of FIG. 1 may provide the user terminal 3 with the image (captured image) that is the source of the visualization data. In this case, the user terminal 3 may display the captured image on condition that the user's operation is detected while displaying the visualization data. For example, an image obtained by capturing a part of the floor plan displayed on the user terminal 3 may be displayed by the user. That is, an image used for specifying each spatial element is accumulated, and when the user designates a spatial element, an image associated with the spatial element is provided. Accordingly, when the user cannot determine the recognition state from the visualization data, the user can determine the recognition state using the image.

As yet another modification, the range to be recreated may be designated as the method for designating the recreated portion described with reference to FIG. This designation method may be an operation of sliding the fingertip on the screen of the user terminal 3 in a manner of drawing a circle with the fingertip, and is designated by moving the fingertip so as to surround the range to be recreated. In conjunction with this operation, the fingertip trajectory is drawn on the screen, and the fingertip trajectory is visualized so as to be superimposed on the visualization data.

When the internal parameter set of the robot 2 is a parameter set that varies according to external information of the robot 2 detected by a sensor or the like, the name of the space or the display mode of the space according to the internal parameter set in the space of the robot 2 May be set.
As a first example, when a parameter included in the internal parameter set satisfies a predetermined condition, a name corresponding to the type of the parameter may be set in the space. For example, if the internal parameter of the robot 2 is a parameter set that indicates the emotion of the robot 2 such as fun or lonely, when the condition that the parameter indicating fun is a predetermined value or more in a certain space is satisfied, A name such as “fun room” may be given to the space.

The data providing apparatus 10 may include an emotion parameter control unit (not shown). The emotion parameter control unit (not shown) may change the emotion parameter of the robot 2 according to the output value of the sensor. For example, when the robot 2 detects a touch or hug from the user by a built-in touch sensor, the emotion parameter control unit (not shown) may increase the emotion parameter “fun” of the robot 2. On the other hand, the touch sensor detects a contact that does not detect the user for a predetermined period or longer in the image captured by the camera of the robot 2 or has a high strength and a short duration (for example, a contact when struck). The emotion parameter control unit may reduce the emotion parameter of “fun”. When it is detected that the robot 2 is in the room R1, when the emotion parameter value “fun” exceeds a predetermined threshold, the spatial data generation unit sets the name “fun room” for the room R1. May be.

In this case, the names corresponding to some types of parameters are set, and the names corresponding to the remaining types of parameters may not be set for the remaining types of parameters. In particular, it is preferable that a name indicating a positive meaning (for example, a fun room) is given and a name indicating a negative meaning (for example, a boring room) is avoided.

As a second example, when a parameter included in the internal parameter set satisfies a predetermined condition, a figure or color corresponding to the type of the parameter (circle or pink when the parameter indicating fun is a predetermined value or more) Etc.) may be set in the space.

Also, according to the person or thing detected in the space in the past, the display of the space or the name of the space may be set. The data providing apparatus 10 may include a person detection unit (not shown) that acquires an image captured by the camera of the robot 2 and detects a person appearing in the captured image. Further, the data providing apparatus 10 may include an object detection unit (not shown) that acquires an image captured by the camera of the robot 2 and detects an object reflected in the captured image. For example, in a space in which a person named “Daddy” is detected at a predetermined frequency or more by the person detection unit, a picture of the Daddy or a figure showing the Daddy in a pseudo manner or a person such as “Daddy's Room” An attribute such as a name may be set in the space. In addition, in a space where the person named “Dad” and the person named “Mama” are detected at a predetermined frequency or higher by the person detection unit, detection is performed at the predetermined frequency or higher, such as “Dad and Mom's Room”. Attributes such as the name of the person to be performed may be set in the space. Here, the “predetermined frequency” may be the number of times the target person is detected per unit time, or may be the length of time during which the target person is detected in a predetermined time (for example, 1 hour). Further, in a space where a predetermined number of people or more are detected at a predetermined frequency or more, an attribute such as a name that does not depend on a specific person, such as “everyone's room”, may be set in the space.

Depending on the internal parameters of the robot 2 and people or objects, the display of the space or the name of the space (for example, “everybody's fun room”) may be set.

Further, the name given to the space and the display of the space may be different depending on the parameter set indicating the behavior characteristics of each robot 2 (for example, the parameter set indicating the personality or gender of each individual). The personality of each individual may be set by the user via the user terminal 3, and learning based on the detection result of the sensor provided in the robot 2 is performed in the personality learning unit (not shown) of the data providing apparatus 10. May be obtained. For example, an individual who has a parameter indicating that he / she is actively involved with another person when he / she learns the experience of detecting by a touch sensor that he / she has been held by a person who has been determined to be unknown by person detection processing (an individual who is not shy) May be set. For individuals who are not shy, attributes such as a figure indicating a person and the name of the person may be preferentially reflected in the display or name of the room (for example, “Everybody's Room”). On the other hand, when learning the experience of detecting a hit by an unknown person with a touch sensor, an individual with parameters indicating that he / she wants to avoid involvement with an unknown person (an individual who is shy) is set. May be. For an individual who is shy, for example, internal parameters may be preferentially reflected in the display or name of the room (for example, “fun room”).
As the individual characteristics of the robot, individual characters may be set in advance at the time of manufacture or at the time of shipment.

Further, a template such as a map or a spatial element may be changed according to a parameter set indicating the behavior characteristics of each robot 2.

Also, a display of characters or figures that prompt the user's predetermined actions such as accompanying the user according to the map may be output. Moreover, whether this display is displayed according to the character of the robot 2, or the display content may be changed. For example, when it is detected that the user uses an unidentified one at a predetermined frequency or more, the robot 2 may be set as an individual having parameters of behavioral characteristics that are curious. For example, based on the captured image, the emotion parameter control unit is required to touch a thing (spatial element) that is not recognized by the user and detect the user's smile. It may be set so that the parameter indicating is easy to rise and difficult to fall. Alternatively, simply set a predetermined observation period, and set a “curiosity” when the user touches an unidentified object for a predetermined number of times or for a predetermined time or more. May be. In the case of the robot 2 set in such a curious character, a message or a graphic indicating that “it seems that there is an undeveloped area over here, but want me to take it” may be output.

Further, when the frequency of detecting that the user uses an unidentified one is equal to or less than a predetermined value, the robot 2 may be set as an individual having a parameter indicating a cowardly behavior characteristic. In the case of the robot 2 exhibiting timid behavior characteristics, when a button for instructing creation of an ungenerated map is pressed, the user's behavior such as “accompanied by the user” is not allowed. A message or a figure for prompting may be output.

Robot 2 may acquire a map (spatial data) from another robot 2. The spatial data generation unit 13 may connect a map acquired from another robot 2 to a map acquired by itself. In this case, a map acquired from another robot may be displayed in a different expression from the map acquired by itself. According to such a control method, since the plurality of robots 2 can share knowledge with each other when generating the spatial data, the entire spatial data can be obtained earlier than when the spatial data is generated by only one robot 2. Can be generated.

As yet another modification, the spatial data generation unit 13 in FIG. 1 may set attribute information based on a person's flow line. The trajectory of movement of people in the building is called a flow line, and the flow line is determined by the layout and arrangement of furniture. For example, in a living room in which a sofa and a dining table are arranged, there are two flow lines: a flow line from a door serving as an entrance to the sofa and a flow line from the door to the dining table. These flow lines are areas where people frequently pass, and obstacles are removed so as not to obstruct people's movement. In other words, the flow line is an area where safety of walking should be implicitly ensured, and if it is on the flow line, a person can safely move unconsciously.

If there is a robot whose height is lower than the human eye in the area where the safety of walking is to be ensured, the human may not be aware of the robot and may come into contact with it. People's lives vary depending on the time of day and day of the week.For example, on weekday mornings, people move around the flow line without difficulty, and during the daytime on holidays, people take a leisurely flow line while worrying about their surroundings. Moving. That is, when moving along the flow line, the interest that a person turns to the surroundings also changes, and the possibility of contact with the robot also changes. For example, if the robot is positioned on the flow line during commuting or school, the possibility of contact with a person is higher than during the daytime on holidays. Also, during the daytime on weekdays, the number of people at home is reduced, so the possibility of contact is reduced.

The robot 2 acting autonomously estimates the flow line of the person and moves away from the estimated flow line, thereby avoiding contact with the person. When the robot grasps the human flow line, the robot can move while avoiding the human flow line. For example, the robot 2 estimates a human flow line based on information acquired from a space such as a floor plan, the size and type of furniture, and the position of the furniture, and is within a predetermined range from the estimated human flow line. If the robot behaves according to the condition that it does not enter, it will be easier to avoid contact with people.

The spatial data generation unit 13 may set attribute information indicating a flow line according to the number or movement trajectory of people in each time zone or the number or movement trajectory of objects.
For example, when it is indicated from the past data that the space overlaps with the movement locus of the person in a certain time zone (hereinafter referred to as “flow line region”), the instruction unit 113 Attribute information that prohibits the robot 2 from entering the flow line area (attribute information such as a clearance of 30 cm or more from the flow line area) may be set. The robot 2 moves by controlling the moving mechanism 29 in addition to the flow line area in addition to the restriction at the time of action. The restriction in the flow line area may be a restriction that prohibits the robot 2 from entering the flow line area, or may enter the flow line area, but stop in the flow line area for a predetermined time or more. It may be a restriction that it should not. Further, in the flow line area, assuming that the movement of a person is given priority, the movement direction may be controlled so as to leave the flow line area when the robot 2 recognizes the presence of the person. That is, in the flow line region, behavior restrictions are not always imposed, and behavior restrictions may be imposed according to a relative positional relationship with a person.

For example, in the past data, when the movement of a person is recorded in the entire range of the kitchen from 6:00 am to 7:00 am (when the movement trajectory of the kitchen and the person overlaps a predetermined number or more than a predetermined probability) The attribute information for prohibiting entry into the kitchen may be set from 6:00 am to 7:00 am. If the user notices that the robot 2 does not enter when the kitchen is used for cooking or the like, and the robot 2 enters when the kitchen is not busy, the robot 2 does not interfere with the user. In addition, it is possible to feel the wisdom that the robot 2 is acting in accordance with the user's situation.

Instead of or in addition to these, attribute information prohibiting staying for a predetermined time or longer in a specified space is set according to the number or movement trajectory of people in each time zone or the number or movement trajectory of objects. May be. For example, in the past data, when movement of a person is recorded in the entire range of the kitchen from 6:00 am to 7:00 am, the predetermined time (eg, 6:00 am to 7:00 am) (3 minutes) Attribute information for prohibiting the stay may be set. Alternatively, when the robot 2 detects a person within a predetermined distance from the kitchen or the kitchen, the robot 2 may avoid entering the kitchen, or avoid staying for a predetermined time or longer. Also good.

Also, attribute information may be set according to the visibility in space and the number or movement trajectory of people or the number or movement trajectory of objects. Instead of or in addition to these, attribute information that prohibits staying for a predetermined time or longer may be set according to the visibility in space and the number or movement trajectory of people, or the number or movement trajectory of objects.

For example, in an environment where the brightness of the space showing the visibility of the space is turned off at night, such as when the lighting is turned off at night, it is indicated from the past data that it is a flow line region that overlaps with the movement locus of the person. If there is, attribute information for prohibiting the robot 2 from entering the flow line area (attribute information such as a clearance of 30 cm or more from the flow line area) may be set. According to such a control method, it is possible to reduce the risk of contact between the human and the robot 2 in a space with low visibility such as darkness.
The number of people present, the movement trajectory, the number of objects present, and the movement trajectory can be recognized based on past output data of the sensor such as a captured image or output data of the temperature sensor.

As yet another modification, the flow line area may be set by being designated by the user via the user terminal 3 of FIG. For example, the user can specify the time zone and the flow line in the visualized image shown in FIG. More specifically, the flow line is designated by sliding the fingertip on the screen of the user terminal 3 on which the visualized image is displayed. A flow line superimposed on the floor plan is drawn in conjunction with the movement of the fingertip. The flow line designated by the user is obtained by the designation obtaining unit 162 in FIG.

As yet another modification, a spot is displayed on a screen representing visualization data (see FIGS. 7 and 8) based on data received from the user terminal 3, for example, so as to move the robot 2 when a predetermined condition is satisfied. May be set. For example, the user can designate a spot by tapping near the entrance on the screen displayed on the user terminal 3.
The predetermined condition is, for example, after detecting that the user has gone out from information indicating the user's position directly or indirectly, such as the position information of the user terminal 3, and then toward the space where the robot 2 exists. It may be a condition that it is detected that it is moving. Further, the predetermined condition is a condition that it is detected that the user is moving toward a predetermined spot from information indicating the user's position directly or indirectly, such as the position information of the user terminal 3. Also good. Further, the visualization data providing unit 161 that has detected such designation may display a mark corresponding to the spot.

Based on the tapped position, coordinate information indicating the spot is specified in the spatial data. This coordinate information is stored as spot attribute information. Furthermore, the direction of the robot 2 before or after the arrival of the spot may be set together. In this case, the orientation of the robot 2 may be automatically set according to the spatial element of the spatial data (for example, may be set to face the center direction of the door), or via the user terminal 3. May be specified by the user. In this case, if the mark corresponding to the spot output on the screen is a design representing the direction of the robot, the user can recognize the direction of the robot by the angle of the mark. A direction mark such as an arrow or a triangle may be displayed together with the mark to express the direction of the robot 2 before and after the spot arrival. For example, when the user touches a button for instructing right rotation or left rotation, the mark may be rotated or the direction of the direction mark may be changed. The orientation of the robot visualized in this way is also stored as spot attribute information.

The autonomous behavior robot 1 may generate a spot movement event when receiving a wireless signal emitted from, for example, a user terminal 3 of a user (for example, a smartphone carried by the user). When a spot movement event occurs, the autonomous behavior robot 1 specifies a route from the current position of the robot 2 to the spot. The robot 2 controls the moving mechanism 29 to follow the specified route, moves to the spot, and adjusts the posture in the set direction. Note that the spot setting may not be accepted at a stage where the map is not completed or at a stage where the degree of completion of the map is low.

For example, the autonomous behavior robot 1 acquires the position of the user terminal 3 of the user (for example, a smartphone) from GPS (Global Positioning System), and the position is outside a predetermined range (for example, outside the user's house). When moving within a predetermined range (for example, the user's house), the robot 2 may be instructed to move to a spot. Or when a user's user terminal 3 (for example, smart phone) returns a response signal with respect to the beacon installed in a house entrance, you may generate | occur | produce a spot movement event.

As yet another modification, the movement of the robot 2 may be controlled by an instruction via a screen representing the visualization data (see FIGS. 7 and 8). The user designates a moving destination by tapping a place to be moved by the robot 2 on the screen. When the movement destination is set, the movement destination mark is displayed on the screen of the visualization data. Immediately, the autonomous behavior type robot 1 specifies a route from the current position of the robot 2 to the moving destination. The robot 2 follows the route by controlling the moving mechanism 29 and moves to the moving destination. Similar to the example described above, the orientation of the robot 2 may be set. It should be noted that an instruction for a movement destination may not be accepted when the map is not completed or when the map is not complete. When the map is generated only in some rooms, the movement destination instruction may be received only in the room.

As yet another modification, the entry prohibition range of the robot 2 on the condition that the operation surrounding the area where the user does not want to enter the robot 2 is detected on the screen representing the visualization data (see FIGS. 7 and 8). May be set. The robot 2 searches for a route through a route avoiding the entry prohibition range, and moves according to the searched route.

For example, assume that the user wishes to prohibit the robot 2 from entering the bedroom. In this case, on the visualization data screen illustrated in FIG. 7, the user performs an operation of surrounding an area corresponding to the bedroom with a finger. The spatial data generation unit 13 sets a space corresponding to the closed area drawn on the screen detected by the user terminal 3 as an “entry prohibition range”, and generates spatial data. According to such a control method, the user can set the robot entry prohibition range with an intuitive operation on the visualization data screen of the smartphone.

In the process of point cloud data generation (S12) and spatial data generation (S13) shown in FIG. 3, a SLAM (Simultaneous Localization and Mapping) technique that simultaneously estimates self-location and creates an environmental map may be applied. As for the SLAM technology, application examples such as a self-propelled vacuum cleaner and a drone are known. The autonomous behavior robot 1 is different from the SLAM technology of a self-propelled cleaner that generates a plane map in that it generates spatial data representing a three-dimensional space. Further, since the autonomous behavior robot 1 generates spatial data representing a closed space surrounded by a wall surface and a ceiling, the autonomous behavior robot 1 is different from the SLAM technology of a drone that is used outdoors and generates spatial data that is not closed. In a closed space, the characteristics of the wall and ceiling can be used, so it is easier to estimate the self-position than in the case of outdoors without walls and ceilings. In addition, since the number of feature points included in the closed space is limited, the number of verifications by matching with feature points included in the currently photographed image is smaller than in the case of an infinitely wide outdoor space. Therefore, there are advantages that the estimation accuracy is high and the time until the estimation result is calculated is short.

Claims

A moving mechanism;
A shooting section for shooting the surrounding space;
A spatial data generation unit that generates spatial data recognizing the space based on a captured image captured by the imaging unit;
A visualization data generation unit that generates visualization data that visualizes the spatial elements included in the space, based on the generated spatial data;
An autonomous behavior type robot comprising: a visualization data providing unit that provides the generated visualization data.
An instruction obtaining unit for obtaining designation of an area included in the provided visualization data;
The autonomous behavior robot according to claim 1, wherein the spatial data generation unit regenerates spatial data recognizing a space in the acquired area corresponding to the designation.
The said visualization data generation part produces | generates the said visualization data reflecting the characteristic of the recognized said spatial element based on the spatial element recognized from the picked-up image image | photographed in the said imaging | photography part. Autonomous robot.
The said visualization data production | generation part produces | generates the said visualization data which provided the attribute of the color to the said space element based on the color information of the picked-up image image | photographed in the said imaging | photography part. The autonomous behavior type robot described.
The autonomous behavior type robot according to any one of claims 1 to 4, wherein the visualization data generation unit generates the visualization data in which a fixed spatial element and a movable spatial element are distinguished.
The autonomous data generation unit according to claim 5, wherein the visualization data generation unit generates the visualization data in which the fixed spatial element and the movable spatial element are distinguished based on a temporal change of the spatial data. Behavioral robot.
The autonomous behavior type according to any one of claims 1 to 6, wherein the visualization data generation unit generates the visualization data obtained by visualizing the spatial data included in a predetermined range from a position moved by the moving mechanism. robot.
A point cloud data generation unit that generates three-dimensional point cloud data based on the captured image captured by the imaging unit;
The autonomous behavior robot according to any one of claims 1 to 7, wherein the spatial data generation unit generates the spatial data based on the generated point cloud data.
The spatial data generation unit generates the spatial data by specifying an outline of the spatial element in the point cloud data,
The autonomous behavior robot according to claim 8, wherein the visualization data generation unit generates the visualization data obtained by visualizing the space element based on the identified outline.
A visualization data generation unit that generates visualization data that visualizes the spatial elements included in the space, based on the spatial data obtained by the robot recognizing the space;
A data providing apparatus comprising: a visualization data providing unit that provides the generated visualization data.
A designation obtaining unit for obtaining designation of an area included in the provided visualization data;
The data providing apparatus according to claim 10, further comprising: an instruction unit that instructs the robot to recognize the space in the acquired area related to the designation.
On the computer,
A visualization data generation function for generating visualization data that visualizes spatial elements included in the space, based on spatial data obtained by the robot recognizing the space;
A data providing program for realizing a visualization data providing function for providing the generated visualization data.