WO2014102995A1

WO2014102995A1 - Monitoring system, method, and information-recording medium containing program

Info

Publication number: WO2014102995A1
Application number: PCT/JP2012/084018
Authority: WO
Inventors: 洋登永吉; 義崇平松
Original assignee: 株式会社日立製作所
Priority date: 2012-12-28
Filing date: 2012-12-28
Publication date: 2014-07-03
Also published as: JPWO2014102995A1

Abstract

The purpose of the present invention is to expand a field of vision of an environment camera when remote monitoring is performed using a robot provided with a mobile camera. Accordingly, a spatial information processing unit provided with a map estimates the position of a fixation point of the robot camera with reference to the map and the position and direction of the robot. Photographing a predetermined relative position in relation to position of the fixation point using the environment camera, and displaying images photographed by the robot in a line up on a display screen allows what is occurring outside an image photographed by the robot to be presented to an operator.

Description

Information recording medium storing monitoring system, method and program

The present invention relates to a remote monitoring system using a robot.

For example, when a robot equipped with a camera is used, various places can be photographed and viewed from a remote place according to the request of the robot operator. At this time, a wide viewing angle is required to improve operability and browsing.

Generally, a wide-angle camera, a fish-eye camera, or a wide viewing angle camera that can shoot the entire circumference in a cylindrical shape is often used. However, as a matter of course, a wide viewing angle camera is not suitable for photographing a small object or a distant object, and is preferably used in combination with a camera that has a narrow field of view but can shoot at a telephoto position.

Patent Document 1 discloses a technique related to the expansion of the field of view of a vehicle, and Patent Document 2 discloses a method for predicting a route of a robot in robot operation.

JP 2008-230558 A JP 2011-53726 A

Patent Document 1 discloses that a display is provided on both sides of a rearview mirror and an image from a camera provided on a side mirror is displayed on the display in order to extend the driver's field of view. However, if the two types of cameras cannot be installed in the chassis due to the load on the drive unit, the size of the storage unit, and the design, the field of view of the robot cannot be expanded with this technology.

In order to solve the above problems, for example, the configuration described in the claims is adopted. The present application includes a plurality of means for solving the above-mentioned problems. For example, a monitoring method using a robot, in which a first image is provided by a first camera included in a remotely operable robot. A step of photographing, a step of calculating a gazing point that is the center of the first image from the first image, a step of calculating an extended gazing point around the gazing point, and a second that shoots the extended gazing point A step of selecting a camera, and a step of displaying the first image and the second image taken by the second camera side by side on a display device for an operator of the robot. The center of the second image, and the second camera is provided in a space in which the robot moves.

Alternatively, the monitoring system may be a first camera included in a remote-controllable robot and a first image captured by the first camera, and a gaze point that is the center of the first image and a gaze point A calculation unit that calculates a surrounding extended gazing point, a camera selection unit that selects a second camera that shoots the extended gazing point, a first image, and a second image captured by the second camera And a display device used by an operator who operates the robot, the extended gazing point is the center of the second image, and the second camera is provided in a space in which the robot moves. It is characterized by.

視野 By using the system of the present invention, the field of view of the robot can be expanded and the operability of the robot is improved.

It is a figure which shows the surrounding environment and apparatus of the robot in Example 1. FIG. FIG. 2 is a block diagram in the first embodiment. It is a figure which shows the example of a three-dimensional map. It is a figure which shows the example of environmental camera information. It is a figure which shows the flow of the monitoring method in Example 1. FIG. It is a figure which shows the gaze point and extended gaze point of a robot. It is a figure which shows the calculation method of a gaze point position. It is a figure shown about rotation of an imaging direction vector. It is a figure which shows the example of path | route history information. It is a figure which shows a gaze point and an extended gaze point. It is a figure which shows the example of the display screen for robot operators. It is a figure which shows the example of the display screen for robot operators. It is a figure which shows the information passed between apparatuses. It is a figure which shows the information passed between apparatuses. It is a figure which shows the example of a two-dimensional map. It is a figure which shows the flow of the information between apparatuses. It is a figure which shows a gaze point and an extended gaze point.

FIG. 1 is a diagram showing an example of an environment for implementing the present invention. The robot 100, the environmental camera 101, the installation object 140, and the installation object 141 are installed in the room 130, and the space information processing unit 120, the remote operation processing unit 121, the display device 121, and the input device 122 are installed and connected. Yes. It is desirable that there are a plurality of environmental cameras 101.

Fig. 2 shows the block configuration. A robot 100, an environmental camera 101, a spatial information processing unit 120, and a remote operation processing unit 121 are connected by a network 230. The display device 121 and the input device 122 are connected to the remote operation processing unit 121. The display device 121 provides information to the operator, and the input device 122 receives input from the operator. The remote operation processing unit 121 generates an image to be displayed on the display device 121 in accordance with an instruction from the spatial information processing unit 120, and transmits information from the input device 122 to the spatial information processing unit 120.

The robot 100 includes a control unit 210, a camera 211, a position / direction estimation unit 212, a measurement unit 213, and a drive unit 214. Here, the measuring unit 213 is a sensor such as a laser range finder, for example, and can measure the distance to surrounding objects. The position / direction estimation unit 212 can estimate the position and direction of the robot by using, for example, a technique called SLAM (Simultaneous Localization and Mapping) using the measured distance information.

Further, the measuring unit 213 may be a sensor that measures the driving state of the driving unit 214. In this case, the position / direction estimating unit 212 can estimate the position and direction of the robot from the driving state of the robot.

Further, the driving unit 214 may include a mechanism for driving the camera 211 of the robot 100. For example, in the case of a humanoid robot, the camera 211 is often mounted on the head of the robot. In that case, the driving unit 214 can change the direction of the camera 211 by driving the head of the robot. In this case, the position / direction estimation unit 212 measures the driving state of the driving unit 214 by the measuring unit 213 and also estimates the direction of the camera 211.

Furthermore, the position / direction estimation unit 212 may have information on the height of the camera 211. For example, if the height of the robot 100 can be changed by the driving unit 214 and the height of the camera 211 is also changed, the measuring unit 213 may further include a sensor for measuring the height of the camera 211.

The spatial information processing unit 120 includes a spatial information DB (Data Base) 221. The spatial information DB 221 holds a map 300 as shown in FIG. 3 and environmental camera information 400 as shown in FIG. The map 300 is a three-dimensional map and records surfaces that block light, such as walls and desk surfaces. The map information 330 corresponds to the room 130, the map information 340 corresponds to the installation object 140, the map information 341 corresponds to the installation object 141, and the surface information is recorded as polygons, for example.

A design drawing may be used for the map 300, it may be created manually, or a result actually measured by a device such as a laser range finder or a distance image camera may be used.

The environmental camera information 400 includes a camera ID 410 assigned to each camera, a camera position 411 represented by X, Y, and Z on the map 300, a camera direction 412 represented by φ and θ on the map 300, and the current camera. An angle of view 413, a camera direction adjustment range 414 that is an adjustable camera direction range, and a camera angle of view range 415 that is an adjustable camera angle of view range are included. In the case of a fixed camera, the width of the φ and θ adjustment ranges is zero in the camera direction adjustment range 414.

A method for acquiring images for remote monitoring using a robot will be described with reference to FIGS. This processing is performed by the spatial information processing unit 120 unless otherwise noted. When the processing is started (501), the spatial information processing unit 120 acquires information on the position and direction of the robot 100 and the height and direction of the camera 211 from the position / direction estimation unit 212 of the robot 100 (502). When the height information of the camera 211 cannot be obtained, a predetermined value is used.

Here, the robot gaze point 601 and the extended gaze point 602 will be described with reference to FIG. The gaze point 601 indicates the center of the image captured by the robot camera, and the extended gaze point 602 is in the vicinity of the gaze point 601 and indicates the center of the image captured by the environment camera 101. Thereby, the surrounding image of the gazing point 601 can be acquired. Note that the shooting direction of the camera 211 of the robot 100 shooting the gazing point is shown as a vector 610, and the shooting direction of the environmental camera 101 shooting the extended gazing point is shown as a vector 611.

Next, a calculation method of the gaze point position 721 that is the position of the gaze point 601 will be described with reference to FIG. Since the spatial information processing unit 120 acquires the position and direction of the robot 100 and the height and direction of the camera 211, the camera position 700 that is the position of the camera 211 and the shooting direction of the camera 211 are displayed on the map 300. A shooting direction vector 701 is known. The shooting direction vector 701 is extended from the camera position 700, and is extended until a certain aspect, that is, the map information 340 corresponding to the installation 140 in FIG. The point where the collision occurs becomes a gaze point position 721. If it does not collide, it is assumed that there is a surface at a predetermined distance. In this way, the spatial information processing unit 120 calculates the position of the robot gazing point 601 (503).

Subsequently, the extended gaze position 722 that is the position of the extended gaze point 602 is calculated as follows (504). First, a plane 710 perpendicular to the shooting direction vector 701 is placed on the position 721 of the gazing point 601. Subsequently, an extended gazing point vector 702 that is a vector obtained by rotating the shooting direction vector 701 is calculated.
The photographing direction vector 701 is rotated as shown in FIG. First, a vertical plane extending from the base 800 and the base 801 is assumed at the start point of the shooting direction vector 701. However, the base 800 is on the XY plane, and the base 801 is parallel to the Z axis.

The rotated vector 702 is rotated according to the first rotation 810 and the second rotation 811. The first rotation 810 is a rotation in a plane stretched between the base 800 and the base 801 with respect to the base 800. This is a counterclockwise rotation when the direction of the shooting direction vector 701 is viewed from the camera position 700, and is a parameter that determines in which direction the extended gazing point is positioned with respect to the gazing point. If the extended gazing point is to be positioned horizontally with respect to the gazing point, the magnitude of the first rotation 810 may be set to 0 ° or 180 °.

The second rotation 811 is a rotation in a plane parallel to the imaging direction vector 701 and parallel to the vector obtained by rotating the base 800 with the first rotation 810. The second rotation 811 is a parameter that determines how far the extended gazing point is placed from the gazing point. The magnitude of the second rotation 811 may be determined according to the angle of view of the camera 211 and the angle of view of the environmental camera that captures the extended gazing point 602. For example, a range that cannot be captured, that is, a value at which no blind spot is generated, may be set as the value of the second rotation 811 between the image captured by the camera 211 of the robot 100 and the image captured by the environment camera 101. . Further, if the image captured by the environment camera 101 is captured in a range as different as possible from the image captured by the camera 211 of the robot 100, a wider range of images can be provided to the robot operator. Therefore, it is desirable to use the minimum value among the above values.

The extended gaze vector 702 obtained in this way is extended until it hits the plane 710. The point of collision is the extended gazing point position 722. If it does not collide, it is assumed that there is a surface at a predetermined distance.

Next, the size of the deviation between the predicted route and the predicted gazing point is determined (505). The predicted path is a predicted position of the robot 100 after a predetermined time has elapsed, and here, a value predicted in the past is compared with the current robot position. If the prediction is incorrect, the predicted route is calculated again (506). The predicted gazing point is a prediction of the position of the gazing point 601, and here, a past predicted value is compared with the current actual gazing point. If the prediction is incorrect, the predicted gaze point is calculated again (507). Although no prediction is performed in the initial state, in this case, the determination 505 proceeds to the next process on the assumption that the condition is true.

Next, calculation of a predicted route (506) will be described. Two cases are considered here. The first is a case where a destination has already been given and a route to the destination has already been calculated. This is referred to as planned movement. The second case is a case where the robot operator uses the input device 122 to control the robot in real time. This is called real-time operation movement. In the case of planned movement, the route has already been calculated, so that route may be used. Note that the method disclosed in Patent Document 2 may be used to calculate the route when the destination is given. The route may be calculated by the control unit 210 of the robot 100 or may be calculated by the spatial information processing unit 120. However, since the route itself needs to be held by the control unit 210, in either case, the spatial information processing unit 120 only needs to acquire the route from the control unit 210. If the height of the camera 211 is acquired from the position / direction estimation unit 211, the predicted route on the map 300 can be expressed as a predicted route 1000 as shown in FIG.

In the case of real-time operation movement, the spatial information DB includes a route history 900 which is a history of route information as shown in FIG. In FIG. 9, a route history 911 is a route information history in the first case, and a route history 912 is a route information history in the second case, and is stored as a two-dimensional coordinate information group. At this time, the two-dimensional coordinate information is not the coordinates themselves, but may represent discrete block-like positions such as 50 cm × 50 cm or 1 m × 1 m, for example. As a technique for predicting a future destination based on the route history 912, a pedestrian flow line analysis method based on a mixed Markov model is known. Based on this movement destination prediction and the technique of Patent Document 26, the route of the robot 100 is obtained again. The route thus obtained becomes the predicted route 1000.

Next, seek a forecast gaze point. With respect to the planned movement, a predicted gazing point is obtained on the assumption that the relative direction of the camera 211 with respect to the traveling direction of the robot 100 is maintained. That is, as shown in FIG. 10, prediction execution points are set on the prediction path 1000 at predetermined intervals. A predicted camera direction vector 1010 obtained by rotating the tangent vector of the predicted path 1000 at those points by the relative direction of the camera 211 is obtained. The predicted camera direction vector 1010 is extended, and each position where the predicted camera direction vector 1010 hits a certain plane becomes a predicted gaze position 1020. If it does not collide, it is assumed that there is a surface at a predetermined distance.

Although not shown, in the case of real-time operation movement, it is necessary to look at the front of the robot 100. Therefore, prediction execution points on the prediction path 1000 are taken at predetermined intervals, and the tangent vector of the prediction path 1000 at that point becomes the prediction camera direction vector 1010 as it is. As in the case of the planned movement, they are extended and the respective positions where they collide with a certain surface are set as predicted gaze point positions 1020. If it does not collide, it is assumed that there is a surface at a predetermined distance.

The subsequent processing is calculation of a predicted extended gaze point (508). As shown in FIG. 17, the predicted extended gaze position 1600 is performed in the same manner as the calculation of the extended gaze position 722 in FIG.

The next process is environmental camera selection (509). Condition 1: Selection of environment camera Condition 1: The corresponding environment camera 101 can shoot the current extended gaze point position 722 by adjusting pan / tilt Condition 2: The environment camera 101 is predicted extended by adjusting pan / tilt Note The viewpoint 1600 can be photographed. Condition 3: The angle formed by the predicted camera direction vector 1010 corresponding to the predicted extended gaze position 1600 and the vector connecting the predicted extended gaze position 1600 and the environment camera 101 is equal to or less than a predetermined value. 4: Four conditions are used that satisfy the condition 3 for the longest time.

First, since condition 1 cannot be taken unless this condition is satisfied in the first place, there is no point in selecting a camera that does not satisfy this condition. Conditions 2 to 4 have the purpose of preventing the user from being confused if the camera is frequently switched.

Also, the time for condition 3 can be obtained as follows. The predicted extended gazing point positions 1600 are arranged in chronological order. A predicted extended gazing point position 1600 can be photographed by pan / tilt adjustment for an environmental camera 101, and a predicted camera direction vector 1010 corresponding to the predicted extended gazing point position 1600 and a predicted extended gazing point position 1600 Whether or not the condition that the angle formed by the vector connecting the camera 101 is equal to or less than a predetermined value is satisfied is sequentially determined for each predicted extended gaze position 1600. The number of extended gazing point positions 1600 that satisfy the condition may be the environmental camera 101 that satisfies the condition 4 that is continuously the largest since the present time. Here, since it is a problem of which environmental camera 101 is selected, it is not always necessary to obtain the absolute time.

In determining whether or not these conditions are satisfied, the spatial information processing unit 120 uses the holding information 400 that is information about the environmental camera 101. If there are a plurality of extended gazing point positions 722, conditions 1 to 4 are determined in order, and one of the environmental cameras 101 is assigned to each of them. Note that a camera assigned to a certain extended gaze position 722 is excluded in advance when searching for a camera to be assigned to another extended gaze position 722.

Further, the predetermined value regarding the difference in angle in condition 3 is set to a wide value, for example, 60 ° if the object to be imaged is planar, and to a narrow value, for example, if the object to be imaged is a complicated solid shape. Set 30 °. If there are no more cameras assigned, camera selection ends at that point.

Subsequently, environmental camera control is performed (510). Control is performed so that the environment camera 101 selected in the environment camera selection (509) faces the extended gazing point 722.

Hereafter, a method of displaying the image of the camera 211 photographed by the above method and the image photographed by the camera toward the extended gazing point position 722 will be described. The image is created by the remote operation processing unit 121 and displayed on the display device 122. For example, as shown in FIG. 11, an image 1100 of a gazing point 601 and

images

1101 and 1102 of an extended gazing point 602 are displayed side by side on the display device 121. From the viewpoint of a person, an image around the gazing point 601 can be seen together, and a pseudo field of view can be enlarged.

FIG. 12 shows another display method. In FIG. 11, the

images

1100, 1101, and 1102 are displayed in a format that is directly pasted in the two-dimensional space. However, since the positions of the robot 100 and the environmental camera 101 are different from each other, the images are images taken from different viewpoints. Therefore, even if the characters written on the wall are copied as shown in FIG. 6, each image has a different degree of distortion. On the other hand, in FIG. 12,

images

1200, 1201, and 1202 are displayed as three-dimensional computer graphics (CG). Assuming that each image is on a plane perpendicular to the shooting direction of the camera, an image obtained by perspective-transforming the image as viewed from the robot 100 is displayed. By doing so, there is an advantage that the operator can sensuously estimate in which direction the environment camera 101 that has captured the

image

1201 or 1202 is originally located. In addition, as shown in FIG. 6, if a character written on a plane is copied, there is an advantage that each character has the same degree of distortion.

FIG. 13 and FIG. 14 show information passed between the blocks when the remote monitoring image generation is executed. FIG. 13 shows information passed from the robot 100 to the spatial information processing unit 120. Information on the position and direction of the robot 100 and the direction and height of the camera 211 is passed from the position / direction estimation unit 212 of the robot 100. As shown in FIG. 14, the spatial information processing unit 120 passes to the remote operation processing unit 121 an image captured by the environmental camera 101 and information about the position and direction of the camera. The remote information processing unit 121 generates a screen as shown in FIG. 11 or 12 based on the information and passes the display screen to the display device 122.

FIG. 16 shows the flow of processing and data in the method described above. FIG. 16 is a diagram mainly showing a program operating in the spatial information processing unit 120 and a data flow. In FIG. 16, a gaze point calculation unit 1603 is a gaze point calculation step 503, an extended gaze point calculation unit 1604 is an extended gaze point calculation step 504, a prediction deviation determination unit 1605 is a prediction deviation determination step 505, and a prediction path calculation unit 1606. Is a predicted route calculation step 506, a predicted gaze point calculation unit 1607 is a predicted gaze point calculation step 507, a predicted extended gaze point calculation unit 1608 is a predicted extended gaze point calculation step 508, and an environmental camera selection unit 1609 is an environment The camera selection step 509 and the environmental camera control unit 1610 are programs corresponding to the environmental camera control step 510, respectively.

Based on the above, the monitoring system for robot control according to the present invention is the center of the first image from the first camera included in the remotely operable robot and the first image captured by the first camera. Photographed by a calculation unit that calculates a viewpoint and an extended gazing point around the gazing point, a camera selection unit that selects a second camera that shoots the extended gazing point, a first image, and a second camera And a display device used by an operator who operates the robot to display the second image displayed side by side, the extended gazing point is the center of the second image, and the second camera moves by the robot It is provided in the space.

According to the present invention, it is possible to provide the robot operator with an image with a field of view that is greater than the angle of view of the camera of the robot, and the operability of the robot is improved.

In the first embodiment, the case where a three-dimensional map is held has been described, but the same processing is possible even with a two-dimensional map 1500 as shown in FIG. In FIG. 15,

map information

1530, 1540, and 1541 are shown in the room 130 and the installation objects 140 and 141, respectively. At this time, although the height information is not known, it is possible to treat each installation as a three-dimensional map by assuming that each installation has an infinite height.

In the present embodiment, in addition to the effects of the first embodiment, it is not necessary to hold a three-dimensional map, so the load on the information processing unit can be reduced.

In the first embodiment, the number of robots 100 is one, but a plurality of robots 100 may be used. In this case, by controlling the direction of the robot 100 equipped with a camera that is not an operation target, it can be used as an environment camera 101 capable of pan / tilt adjustment.

In this embodiment, since the environment camera can be moved by using a camera provided by another robot as the environment camera, the shooting time specified in the condition 4 can be made longer. Therefore, the viewpoint change of the operator can be reduced, and the operability can be improved as compared with the first embodiment.

DESCRIPTION OF SYMBOLS 100 Robot 101 Environmental camera 120 Spatial information processing part 121 Remote operation processing part 122 Input device 130 Room 140 Installed object 141 Installed object 230 Network 201 Robot control part 211 Robot camera
212 Position / Direction Estimation Unit 213 Robot Measurement Unit
214 Robot Drive Unit 221 Spatial Information DB
300 Map 400 Environmental Camera Information 330 Map Information 340 Map Information 601 Gaze Point 602 Extended Gaze Point 700 Camera Position 701 Shooting Direction Vector 721 Gaze Point Position 722 Extended Gaze Point Position 800 Base 801 Base 810 First Rotation 820 Second Rotation 900 Route history 1000 Predicted route 1010 Predicted camera direction vector 1020 Predicted gazing point position 1100 Image of gazing point 1101 Image of extended gazing point 1102 Image of extended gazing point 1200 Image of gazing point displayed three-dimensionally 1201 Extended gazing point displayed three-dimensionally Image 1202 An image of an extended gazing point displayed three-dimensionally.

Claims

Capturing a first image with a first camera of a remotely operable robot;
Calculating a gaze point that is the center of the first image from the first image;
Calculating an extended gaze point around the gaze point;
Selecting a second camera that captures the extended gaze point;
The display device for the operator of the robot has the step of displaying the first image and the second image taken by the second camera side by side,
The extended gaze point is the center of the second image;
The monitoring method, wherein the second camera is provided in a space in which the robot moves.
In the monitoring method according to claim 1,
And predicting the movement path of the robot;
Predicting the movement path of the point of interest based on the predicted movement path of the robot;
Predicting the movement path of the extended gaze point based on the predicted movement path of the gaze point.
In the monitoring method according to claim 2,
Further, the step of selecting the second camera includes
A monitoring method comprising: selecting a camera capable of photographing the extended gazing point that moves along the calculated movement route for the longest time.
The monitoring method according to claim 3,
The monitoring method, wherein the second camera is a camera provided by a robot different from the robot provided with the first camera.
A first camera provided in a remotely operable robot;
From the first image captured by the first camera, a calculation unit that calculates a gaze point that is the center of the first image, and an extended gaze point around the gaze point,
A camera selection unit for selecting a second camera for photographing the extended gazing point;
A display device used by an operator who operates the robot, displaying the first image and the second image taken by the second camera side by side;
The extended gaze point is the center of the second image;
The monitoring system, wherein the second camera is provided in a space in which the robot moves.
In the monitoring system according to claim 5,
Furthermore, a first route prediction unit that predicts the movement route of the robot,
Based on the predicted movement path of the robot, a second path prediction unit that predicts the movement path of the gazing point;
And a third route prediction unit for predicting the movement route of the extended gazing point based on the predicted movement route of the gazing point.
The monitoring system according to claim 6,
Furthermore, the camera selection unit
A monitoring system, wherein a camera capable of photographing the extended gazing point moving along the calculated movement route for the longest time is selected.
The monitoring system according to claim 7,
The monitoring system, wherein the second camera is a camera provided by a robot different from the robot provided with the first camera.
Capturing a first image with a first camera of a remotely operable robot;
Calculating a gaze point that is the center of the first image from the first image;
Calculating an extended gaze point around the gaze point;
Selecting a second camera different from the first camera that captures the extended gazing point;
A program for causing the computer to execute the step of displaying the first image and the second image captured by the second camera side by side on the display device for the operator of the robot is recorded. ,
The computer-readable information recording medium, wherein the extended attention point is a center of the second image.
In the information recording medium according to claim 9,
And predicting the movement path of the robot;
Predicting the movement path of the point of interest based on the predicted movement path of the robot;
A program for causing a computer to execute a step of predicting the movement path of the extended gazing point based on the predicted movement path of the gazing point is recorded.
The information recording medium according to claim 10,
Further, the step of selecting the second camera includes
An information recording medium, wherein a camera capable of photographing the extended gaze point moving along the calculated movement route for the longest time is selected.
The information recording medium according to claim 11,
The information recording medium according to claim 1, wherein the second camera is a camera included in a robot different from the robot including the first camera.