JP7138856B2 - Bird's eye view presentation system - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act Date of publication December 5, 2017 Journal of the Japan Society for Precision Engineering Vol.83, No.12, 2017 Pages 1216-1223

The present invention relates to a bird's-eye view video presentation system.

In recent years, along with the use of remote-operable mobile devices (hereinafter referred to as "remote mobile devices") in recovery work at disaster sites, video presentation of remote mobile devices for the purpose of improving the work efficiency of recovery work Technology is being actively developed. For example, a bird's-eye view video presentation system, which is exemplified as one of the video presentation technologies for disaster sites, is equipped with multiple cameras equipped with wide-angle lenses in a remote mobile device, and processes the video acquired from the multiple cameras. Thus, it is possible to present a simulated image of the remote mobile device as if it were viewed from a third person's point of view in the sky.

Japanese Patent Publication No. 2017-538208 JP 2017-173298 A JP 2018-50119 A

In a conventional bird's-eye view video presentation system, objects around the remote mobile device are not accurately drawn on the bird's-eye video due to the method of projecting the camera video, etc. As a result, the environment surrounding the remote mobile device is may not be accurately communicated to the operator.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a bird's-eye view video presentation system capable of presenting as accurately as possible the circumstances surrounding a mobile device that can be moved by remote control.

In order to solve the above-described problems and achieve the object, a bird's-eye view video presentation system according to the present invention includes, as one aspect, a remote controllable mobile device and an information processing device capable of data communication with the mobile device. wherein the mobile device includes a communication unit, a plurality of cameras for capturing wide-angle images of the surroundings of the mobile device, and a detection device that acquires three-dimensional information indicating the position of an object around the mobile device by irradiating it with a laser beam. The information processing device includes a communication unit and a first bird's-eye view video for generating a first bird's-eye video by viewpoint-converting the video acquired by the camera into a video viewed from a virtual bird's-eye viewpoint that overlooks the mobile device. a generator, a second bird's-eye image generator that generates a second bird's-eye image by projecting the three-dimensional information onto a coordinate system defining the bird's-eye viewpoint, and the moving device and the object in the second bird's-eye image. a third image generation unit that generates a third image in which a blind spot area is set from the mobile device based on the relative positional relationship of the first bird's-eye image, the second bird's-eye image, and the third An integrated video generating unit that generates an integrated bird's-eye view video by integrating video, and a display unit that displays the integrated bird's-eye video.

INDUSTRIAL APPLICABILITY The bird's-eye view image presentation system according to the present invention has the effect of being able to present as accurately as possible the circumstances surrounding a mobile device that can be moved by remote control.

FIG. 1 is a diagram illustrating a configuration example of a bird's-eye view video presentation system according to an embodiment. FIG. 2 is a block diagram schematically showing the functional configuration of the bird's-eye view video presentation system according to the embodiment. FIG. 3 is a diagram showing the concept of each coordinate system according to the embodiment. FIG. 4 is a conceptual diagram when obtaining an extrinsic parameter matrix according to the embodiment. FIG. 5 is a conceptual diagram of adding texture information to 3D point group information and projecting the 3D point group onto the floor according to the embodiment. FIG. 6 is a conceptual diagram showing a method of generating a shielded area according to the embodiment. FIG. 7 is a conceptual diagram showing the concept of video integration processing according to the embodiment. FIG. 8 is a flowchart illustrating an example of the flow of bird's-eye view image generation processing executed by the information processing apparatus according to the embodiment. FIG. 9 is a diagram showing an experimental environment for verification experiments. FIG. 10 is a diagram showing an example of a bird's-eye view image according to a comparative example. FIG. 11 is a diagram showing an example of a bird's-eye view image when the mobile device is moved from the position shown in FIG. FIG. 12 is a diagram showing an example of a video of the mobile device shown in FIG. 11 photographed from a third-person viewpoint. FIG. 13 is a diagram illustrating an example of a bird's-eye view image according to the embodiment; FIG. 14 is a diagram showing an example of a bird's-eye view image when the mobile device is moved from the position shown in FIG. FIG. 15 is a diagram showing an example of a video of the mobile device shown in FIG. 14 photographed from a third-person viewpoint.

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of a bird's-eye view video presentation system according to the present invention will be described in detail below based on the drawings. In addition, this invention is not limited by this embodiment. In addition, components in the embodiments described below include components that can be easily replaced by those skilled in the art, or components that are substantially the same. Furthermore, the constituent elements in the embodiments described below can be omitted, replaced, or changed in various ways without departing from the spirit of the present invention. In the following embodiments, necessary constituent elements will be described in order to exemplify one embodiment of the bird's-eye view video presentation system according to the present invention, and other constituent elements will be omitted.

[System configuration]
FIG. 1 is a diagram showing a configuration example of a bird's-eye view video presentation system 1 according to an embodiment. FIG. 2 is a block diagram schematically showing the functional configuration of the bird's-eye view video presentation system 1 according to the embodiment.

As shown in FIG. 1 , a bird's-eye view video presentation system 1 according to the embodiment includes a mobile device 10 capable of remote control and an information processing device 100 capable of data communication with the mobile device 10 . The mobile device 10 and the information processing device 100 are connected to the communication network 5 so as to be able to communicate data with each other. The communication network 5 may be constructed including public communication lines, dedicated communication lines, and the like.

The mobile device 10 travels in a disaster site or the like where the object 3 or the like exists. The mobile device 10 includes a plurality of cameras 11 a to 11 d with fisheye lenses, a detection device 12 , a communication section 13 , an actuator 14 , a storage section 15 and a control section 16 . The mobile device 10 may run according to an operation command received from the information processing device 100 or may run according to an operation command received from a remote device other than the information processing device 100 . As a mobile device 10, Reference X (New Energy and Industrial Technology Development Organization: Unmanned disaster response system research and development project Development of measurement and work element technology Development of amphibious monitoring device, 2011-2012 achievement report (2013)) is exemplified by the equipment developed in the disaster response project.

The cameras 11a to 11d photograph the surroundings (surrounding environment) of the mobile device 10 at a wider angle than a predetermined photographing range. The cameras 11a to 11d are equipped with, for example, fisheye lenses with a wide angle of view of around 180 degrees. In the following description, images based on image data acquired by the cameras 11a to 11d may be referred to as "camera images". When the cameras 11a to 11d are installed in the mobile device 10, for example, refer to Reference 2 (Ren Komatsu, Hiromitsu Fujii, Jun Yamashita, Hajime Asama: Overhead view video presentation system for robot remote control in preparation for failure due to camera layout design Development, Journal of Precision Engineering, 81, 12 (2015) 1206).

The detection device 12 radially irradiates a plurality of laser beams toward the periphery of the mobile device 10 to indicate the positions of objects around the mobile device 10 based on the TOF (Time of Flight) principle. It is a three-dimensional range sensor that acquires dimensional information. The detection device 12 radially irradiates a plurality of laser beams in all directions parallel to the traveling surface 2 on which the mobile device 10 travels at a disaster site or the like, and the emitted laser beams exist around the mobile device 10. By measuring the reflection time from the surface of the object until it returns, it is possible to acquire information on a 3D point group, which is a set of 3D points indicating the position of the surface of the object, as 3D information. Each three-dimensional point forming the three-dimensional point group acquired by the detection device 12 is represented by a three-dimensional coordinate value indicating the position on the surface of the object. A three-dimensional coordinate value is composed of coordinate values corresponding to each of the X-axis direction, Y-axis direction, and Z-axis direction at an arbitrary origin.

As illustrated in FIG. 1, the detection device 12 has an omnidirectional range of 360 degrees in the horizontal direction centered on the moving device 10 and 41.3 degrees in the vertical direction (+10.67 degrees to - It has a cylindrical detection range 12DR of 30.67 degree range). As the laser beam, for example, infrared rays with a wavelength of 905 nm (nanometers) can be used. As the detection device 12, for example, a LiDAR (Light Detection and Ranging) capable of realizing 360-degree omnidirectional three-dimensional measurement at least in the horizontal direction is exemplified. If the detection device 12 can realize omnidirectional three-dimensional measurement of 360 degrees at least in the horizontal direction, it is an MMS (Mobile Mapping System) equipped with an LRF (Laser Range Finder) and a GPS (Global Positioning System). Three-dimensional information may be obtained.

The communication unit 13 performs data communication for exchanging various data with the information processing apparatus 100 via the communication network 5 . The communication unit 13 can receive operation commands related to running. The communication unit 13 can send operation commands to the control unit 16 . The communication unit 13 can transmit video data acquired by the cameras 11 a to 112 d and three-dimensional information detected by the detection device 12 to the information processing device 100 . The communication unit 13 can support various communication methods that enable communication with the information processing device 100 . Either wireless communication or wired communication may be applied as the communication method of the communication unit 13 .

The actuator 14 converts a control signal from the control unit 16 into power for causing the mobile device 10 to travel on the travel surface 2 such as a disaster site. Actuator 14 includes, for example, an electric motor.

The storage unit 15 stores programs and data for realizing various processes executed by the control unit 16 . The functions provided by the program stored in the storage unit 15 include the function of controlling the running of the mobile device 10, and the image data acquired by the cameras 11a to 112d and the three-dimensional information detected by the detection device 12 to the information processing device. 100. The storage unit 15 is a non-volatile or volatile memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (registered trademark) (Electrically Erasable Programmable Read Only Memory). may be implemented by optical semiconductor memory, a magnetic disk, a floppy disk, an optical disk, a compact disk, a minidisk, or a DVD.

The control unit 16 executes various processes related to the mobile device 10 based on the programs and data stored in the storage unit 15 . In particular, in the present embodiment, the control unit 16 transmits video data acquired by the cameras 11 a to 112 d and three-dimensional information detected by the detection device 12 to the information processing device 100 via the communication unit 13 . The control unit 16 may be implemented by a processor such as a CPU (Central Processing Unit) microprocessor, microcomputer, DSP, or system LSI (Large Scale Integration).

The mobile device 10 may include a crawler that can follow a rough surface of a disaster site with steps, a carriage provided with the crawler at the bottom, and a body part arranged on the carriage. The cameras 11a-11d and the detection device 12 described above may be installed on the body.

The information processing device 100 uses the video data and three-dimensional information received from the mobile device 10 to generate a bird's-eye view image presenting the positions of objects around the mobile device 10 . Information processing apparatus 100 includes communication unit 111 , display unit 113 , storage unit 115 , and control unit 117 .

The communication unit 111 executes data communication for exchanging various data with the mobile device 10 via the communication network 5 . The communication unit 111 can receive video data and 3D information transmitted from the mobile device 10 . The communication unit 111 can support various communication schemes that enable communication with the mobile device 10 . Either wireless communication or wired communication may be applied as the communication method of the communication unit 111 .

The display unit 113 displays various information. The display unit 113 can display a bird's-eye view image generated by the control unit 117, which will be described later. The display unit 113 may include a display device such as a liquid crystal display (LCD), an organic electro-luminescence display (OELD), or an inorganic electro-luminescence display (IELD). Display unit 113 may include an input device such as a touch screen.

Storage unit 115 stores programs and data for realizing various processes executed by control unit 117 . The functions provided by the programs stored in the storage unit 115 include a function of generating a bird's-eye view image presenting the positions of objects around the mobile device 10 . The data stored by the storage unit 115 includes video data and 3D information received from the mobile device 10 . The storage unit 15 is a non-volatile or volatile memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (registered trademark) (Electrically Erasable Programmable Read Only Memory). may be implemented by optical semiconductor memory, a magnetic disk, a floppy disk, an optical disk, a compact disk, a minidisk, or a DVD.

The control unit 117 executes various processes related to the mobile device 10 based on the programs and data stored in the storage unit 115 . In particular, in the present embodiment, the control unit 117 executes processing for generating a bird's-eye view image presenting the positions of objects around the mobile device 10 . The control unit 117 may be implemented including a processor such as a CPU (Central Processing Unit) microprocessor, a microcomputer, a DSP (Digital Signal Processor), or a system LSI (Large Scale Integration). The control unit 117 reads out a program stored in the storage unit 115, develops it in a working memory such as a RAM, and causes a processor such as a CPU to execute instructions included in the program developed in the working memory. As a result, the control unit 117 operates based on each function provided by the first bird's-eye view image generation unit 117a, the second bird's-eye image generation unit 117b, the third image generation unit 117c, and the integrated bird's-eye image generation unit 117d, which will be described below. each process can be executed.

The control unit 117 includes a first bird's-eye image generation unit 117a, a second bird's-eye image generation unit 117b, a third image generation unit 117c, and an integrated bird's-eye image generation unit 117d.

The first bird's-eye view image generating unit 117a converts the video data received from the mobile device 10, that is, the video (camera video) acquired by the cameras 11a to 11d into a video viewed from a virtual bird's-eye viewpoint that looks down on the mobile device 10. A first bird's-eye view image is generated by conversion. Under the assumption that all the objects captured by the cameras 11a to 11d of the mobile device 10 exist on a plane in the robot coordinate system, the first bird's-eye view image generator 117a perspectively projects the camera images onto the plane. After that, viewpoint conversion processing is performed to generate the first bird's-eye view image. The first bird's-eye view image generation unit 117a is based on Reference Document 1 (Takaki Sato, Hiromitsu Fujii, Alessandro Moro, Jun Yamashita, Hajime Asama: Construction of a superimposed omnidirectional bird's-eye view video presentation method using multiple fisheye cameras and LRF, The first bird's-eye view image can be generated by using the conventional method proposed in the 13th Society of Instrument and Control Engineers System Integration Division Lecture Proceedings (2012).

[Definition of coordinate system]
FIG. 3 is a diagram showing the concept of each coordinate system according to the embodiment. A coordinate system 21 shown in FIG. 3 is the coordinate system of the mobile device 10 . In FIG. 3, the _{xwyw plane} of the robot coordinate system is the same as the floor surface (running surface 2 of the moving device 20), and the positive direction of the _zw axis is vertically upward with respect to the floor surface. Hereinafter, the coordinate system 21 shown in FIG. 3 will be appropriately referred to as "robot coordinate system 21". A coordinate system 23 shown in FIG. 3 is, for example, a coordinate system that defines the position and orientation of the camera 11a. Hereinafter, the coordinate system 23 shown in FIG. 3 will be appropriately referred to as "camera coordinate system 23". Although not shown, like the camera coordinate system 23, a coordinate system that defines the positions and orientations of the cameras 11b to 11d is set. A coordinate system 25 shown in FIG. 3 is a coordinate system that defines the detection points acquired by the detection device 12 . Hereinafter, the coordinate system 25 shown in FIG. 3 will be appropriately referred to as "sensor coordinate system 25". A coordinate system 27 shown in FIG. 3 is, for example, a coordinate system that defines points on the camera image 11G captured by the camera 11a. Hereinafter, the coordinate system 27 shown in FIG. 3 will be appropriately referred to as "camera image coordinate system 27". Although not shown, like the camera image coordinate system 27, a coordinate system is set to define points on the camera images captured by the cameras 11b to 11d. A coordinate system 29 shown in FIG. 3 is a coordinate system that defines the positions of points on a virtual camera image 31 captured by a virtual camera 31 that is assumed to be at a position where the mobile device 20 is viewed from above. Hereinafter, the coordinate system 29 shown in FIG. 3 will be appropriately referred to as "virtual camera image coordinate system 29". Let P _f =[x _f , y _f , z _f ] ^T represent the point P _w =[x _w , y _w , z _w ] ^T in the robot coordinate system 21 in the camera coordinate system 23 . Let P _d =[x _d , y _d , z _d ] ^T represent the point P _w =[x _w , y _w , z _w ] ^T in the robot coordinate system 21 in the sensor coordinate system 25 . Let m _f =[u _f , v _f ] ^T represent the point P _w =[x _w , y _w , z _w ] ^T in the robot coordinate system 21 in the camera image coordinate system 27 . The first bird's-eye view image generator 117a converts each point in each coordinate system into a point _mv =[ _uv , _vv ] ^T in the virtual camera image coordinate system 29. FIG. In the following description, the coordinate representation of point N in an arbitrary coordinate system by a homogeneous coordinate system is assumed ^to be N .

[Creation of first bird's-eye view video]
The first bird's-eye view image generation unit 117a generates a point P _f =[x _f , y _f , z _f ] ^T in the camera coordinate system 23 shown in Equation (1) below, and a point m _f in the camera image coordinate system 27. =[u _f , v _f ] Based on the relationship with ^T , a perspective projection image is generated from which the distortion peculiar to the image captured by the fisheye lens is removed. Formula (1) shown below is based on reference 3 (Davide Schramuzza, Agostion Martinelli, and Roland Siegwart: A flexible technique for accurate omnidirectional camera calibration and structure from motion, Proceeding of IEEE International Conference of Computer Vision Systems (2006) 45) and Reference 4 (Davide Schramuzza, Agostion Martinelli, and Roland Siegwart: A toolbox for easily calibrating omnidirectional cameras, Proceedings of IEEE/RSJ International Conference of Intelligent Robots and Systems (2006) 5695).

“f(u _f , u _v )” in equation (1) is a function related to the distance √{u _f ² +u _v ² } from the origin in camera image coordinate system 27, and “a” in equation (1) is , are constants set to express the relationship between each point in the camera coordinate system and each direction vector in the camera image coordinate system by an equation.

Subsequently, the first bird's-eye view image generation unit 117a perspectively projects the perspective projection image generated based on the above equation (1) onto a specific plane. The robot coordinate system 21 and the camera image coordinate system 27 can be related by a perspective projection matrix _Hw→f , which is a 3×4 matrix, and the following equation exists between the robot coordinate system 21 and the camera image coordinate system 27: The relationship (2) is established.

Assume that all objects photographed by the cameras 11 a to 11 d of the mobile device 10 exist on a plane in the robot coordinate system 21 . For example, when perspectively projecting a camera image onto a plane (floor) with z _w =0, the third column component of the perspective projection matrix H _w→f can be omitted as shown in the following equation (3).

Here, H'w _→f and P ^~ ' _w in equation (3) are matrices in which the third column of H'w _→f and the third row of P ^~ ' _w are omitted. Equation (3) determines the relationship between the robot coordinate system 21 and the camera image coordinate system 27 .

Subsequently, the first bird's-eye view image generating unit 117a performs viewpoint conversion processing of the perspective projection image perspectively projected onto a specific plane based on the relationship shown in the above-described formula (3), and generates the first bird's-eye view image. Generate. First, the first bird's-eye view image generation unit 117 a obtains the relationship between the robot coordinate system 21 and the virtual camera image coordinate system 29 . Between the robot coordinate system 21 and the virtual camera image coordinate system 29, the following formula (4) holds.

H′w _→v in Equation (4) is a matrix obtained by omitting the third column component of the perspective projection matrix Hw _→v that relates the robot coordinate system 21 and the virtual camera image coordinate system 29 . Using the above equations (3) and (4), the relationship shown in equation (5) is established between the camera image coordinate system 27 and the virtual camera image coordinate system 29 .

However, in Equation (5), H′ _f→w =(H′ _w→f ) ⁻¹ . The first bird's-eye view image generation unit 117a can generate a first bird's-eye view image of the mobile device 20 from above, from the camera image (video data) received from the mobile device 10, according to Equation (5).

Now, as can be seen from equation (5), we need to find H'w _→v and H'f _→w . In this embodiment, reference 5 (Kosuke Sato, Hiromitsu Fujii, Alessandro Moro, Kazuya Sugimoto, Akira Nozue, Yoichi Mimura, Katsumi Obata, Jun Yamashita, Hajime Asama: Development of overhead view video system for unmanned construction, Nihon Kikai Conference Proceedings, 81, 823 (2015) 1), and reference 6 (Keisuke Asari, Yohei Ishii, Hitoshi Hongo, Hiroshi Kano: Simplification of calibration environment for bird's-eye image generation, 13th Image Sensing Symposium Proceedings ( 2017) IN1-13), H'w _→v and H'f _→w are estimated by a nonlinear optimization method. Based on the estimated H'w _→v and H'f _→w matrices, a one-to-one ratio of each pixel of the camera images acquired by the cameras 11a to 11d and the virtual camera image acquired by the virtual camera 31 is obtained. Correspondence can be acquired in advance. Note that the one-to-one correspondence between the pixels of the camera image and the virtual camera image may be stored in advance as a lookup table so that it can be used when generating the first bird's-eye view image.

[Creation of second bird's-eye view video]
The second bird's-eye view image generation unit 117b generates a second bird's-eye view image of objects around the mobile device 10 based on the three-dimensional information detected by the detection device 12 . The three-dimensional information is information of a three-dimensional point group, which is an aggregate of three-dimensional points indicating the positions of surfaces of objects around the mobile device 10, and is represented by three-dimensional coordinate values. The second bird's-eye view image generating unit 117b first uses the three-dimensional object 33 whose shape is known to obtain a 4×4 extrinsic parameter matrix _Md representing the position and orientation of the detection device 12 with respect to the robot coordinate system 21 .

FIG. 4 is a conceptual diagram when obtaining an extrinsic parameter matrix according to the embodiment. R ₁ , R ₂ , and R ₃ are rotation matrices of 3×3 matrices with respect to the _xd -axis, _yd -axis, and _zd -axis when converting the sensor coordinate system 25 to the robot coordinate system 21 , and the three-dimensional translation is Assuming that the movement vector is td _→w , _Md can be expressed by the following equation (6). However, "0" in equation (6) represents a three-dimensional zero column vector.

If R ₁ , R ₂ , R ₃ , and t _d→w are obtained from Equation (6), the extrinsic parameter matrix Md is uniquely determined. By setting the detection device 12 on a plane parallel to the floor surface, the condition that the positive direction of the _zd axis is vertically upward with respect to the floor surface is established, so R ₁ =R ₂ =I can be obtained. However, I is assumed to be a unit matrix of a 3×3 matrix. Subsequently, as illustrated in FIG. 4, three-dimensional objects 33 with known shapes, specifically, two rectangular parallelepiped objects are prepared, and one surface of one object is aligned with the y _w z _w plane. , and one side of the other object is arranged to coincide with the _{zwxw plane} . Among the three- _dimensional information acquired by the _detection device 12 in such an environment, in particular, three _{- dimensional point group} information (3 ) to estimate R ₃ and t _d→w . As described above, the extrinsic parameter matrix _Md is obtained. With the extrinsic parameter matrix M _d , the robot coordinate system 21 and the sensor coordinate system 25 can be related by the following equation (7), and the information of the three-dimensional point group (each three-dimensional point) in the sensor coordinate system 25 ( three-dimensional coordinate values) can be converted into a representation in the robot coordinate system 21 .

In this manner, the second bird's-eye view image generating unit 117b can obtain information (three-dimensional coordinate values) of the three-dimensional point group converted into representation in the robot coordinate system 21. FIG. The second bird's-eye view image generation unit 117b generates information of three-dimensional points satisfying the condition z _w >0 in the height direction among the information (three-dimensional coordinate values) of the three-dimensional point group converted into the representation in the robot coordinate system 21 . 3D point group existing above the floor surface, that is, 3D point group information (three-dimensional coordinate values) can be extracted.

After extracting three-dimensional point group information (three-dimensional coordinate values) indicating the positions of objects existing around the mobile device 10, the second bird's-eye view image generation unit 117b converts the three-dimensional point group information to Texture information is added, and the information of the three-dimensional point group to which the texture information is added is projected onto the floor surface.

FIG. 5 is a conceptual diagram of adding texture information to 3D point group information and projecting the 3D point group onto the floor according to the embodiment. The second bird's-eye view image generation unit 117b adds texture information obtained by the cameras 11a to 11d to the information (three-dimensional coordinate values) of the three-dimensional point group indicating the positions of objects existing around the mobile device 10. Therefore, based on the techniques proposed in References 3 and 4 mentioned above, a 4×4 extrinsic parameter matrix K representing the positions and orientations of the cameras 11a to 11d with respect to the robot coordinate system 21 is obtained. Then, the second bird's-eye view image generation unit 117b can use the extrinsic parameter matrix K to associate the robot coordinate system 21 and the camera coordinate system 23 with the following equation (8).

Subsequently, the second bird's-eye view image generation unit 117b uses the following equation (9) obtained from equations (7) and (8) to obtain the three-dimensional point group information ( three-dimensional coordinate values) are converted from the sensor coordinate system 25 to the camera coordinate system 23 .

A one-to-one correspondence relationship between each point in the camera image coordinate system 27 and each direction vector in the camera coordinate system 23 has already been obtained from the above equation (1). Therefore, the second bird's-eye view image generating unit 117b uses equations (1) and (9) to obtain direction vectors passing through each three-dimensional point included in the three-dimensional point group acquired by the detection device 12. be able to. Therefore, the second bird's-eye view image generation unit 117b generates each 3D point (for example, the 5) can be given texture information (eg, RGB values) of the object.

Subsequently, the second bird's-eye view image generation unit 117b projects the information (three-dimensional coordinate values) of the three-dimensional point group to which the texture information is added onto the floor surface. Specifically, as illustrated in FIG. 5, the second bird's-eye view image generation unit 117b converts a point 35A=[ _xw , _yw , _zw ] ^T in the robot coordinate system 21 to a point 35B=[ x _w , y _w , 0] ^T is orthographically transformed. The second bird's-eye view image generation unit 117b performs orthogonal projection transformation on each three-dimensional point included in the three-dimensional point group.

Subsequently, the second bird's-eye view image generation unit 117b projects each point (for example, point 35B, etc.) after the orthogonal projection transformation onto the virtual camera image coordinate system 29 using the above equation (4). 2 Complete the generation of the bird's-eye view video.

[Generation of third image]
Based on the relative positional relationship between the mobile device 10 and objects around the mobile device 10 in the second bird's-eye view video, the third video generation unit 117c sets a blind area for the mobile device 10. generate video. The shielded area is an area in which three-dimensional measurement cannot be performed by the detection device 12 and three-dimensional information cannot be obtained because the laser beam of the detection device 12 is blocked by an object around the moving device 10 . The shielded area can also be rephrased as an area into which the mobile device 10 cannot enter due to objects around the mobile device 10 . Since the detection device 12 cannot acquire three-dimensional information, it is not possible to present accurate information about the shielded area on the bird's-eye view image. Therefore, in the present embodiment, the information processing apparatus 100 determines that an area located behind an object that is a blind spot when viewed from the mobile apparatus 10 is a shielded area, and clearly indicates the shielded area on the bird's-eye view image. prompts the operator's awareness of uncertain areas in the surrounding environment.

FIG. 6 is a conceptual diagram showing a method of generating the shielded area 39 according to the embodiment. As exemplified in FIG. 6, the third image generating unit 117c generates the second bird's-eye image generated by the second bird's-eye image generating unit 117b. A shielding area 39 is set behind an object that is a blind spot, that is, behind a three-dimensional point group 37 on the virtual camera image 31 projected onto the virtual camera image coordinate system 29 of the virtual camera 31 .

[Generation of integrated bird's-eye view video]
The integrated bird's-eye view image generator 117d superimposes and integrates the first bird's-eye view image, the second bird's-eye image, and the third image, thereby presenting the positional relationship between the mobile device 10 and objects around the mobile device 10 to the operator. Generate the final integrated bird's-eye view video for FIG. 7 is a conceptual diagram showing the concept of video integration processing according to the embodiment.

As exemplified in FIG. 7, the integrated bird's-eye view image generation unit 117d first stores the information of the three-dimensional point group (three-dimensional coordinates value) is superimposed and integrated. Subsequently, the integrated bird's-eye view image generation unit 117d superimposes and integrates the image obtained by integrating the second bird's-eye image G2 with the third image G3 on the first bird's-eye image G1. Generate the final integrated bird's-eye view video to be presented.

FIG. 8 is a flowchart illustrating an example of the flow of bird's-eye view image generation processing executed by the information processing apparatus according to the embodiment. The processing shown in FIG. 8 is executed by the control unit 117 based on the programs and data stored in the storage unit 115 .

As shown in FIG. 8, the control unit 117 generates a first bird's-eye view image obtained by converting camera images obtained by the cameras 11a to 11d into a bird's-eye view (step S101). A bird's-eye view corresponds to a virtual bird's-eye view that looks down on the mobile device 10 from the sky.

Subsequently, the control unit 117 generates a second bird's-eye view image by projecting the three-dimensional information of the objects around the mobile device 10 acquired by the detection device 12 onto the virtual camera coordinate system 29 (step S102).

Subsequently, the control unit 117 generates a third image in which a blind area 39 is set as a blind spot from the moving device 10 based on the relative positional relationship between the moving device and the object in the second bird's-eye view image ( step S103).

Subsequently, the control unit 117 integrates the first bird's-eye video, the second bird's-eye video, and the third video to generate a combined bird's-eye video (step S104), and ends the processing shown in FIG.

[Verification experiment]
The results of the verification experiment of the bird's-eye view video provided by the bird's-eye view video presentation system 1 according to the embodiment will be described below. FIG. 9 is a diagram showing an experimental environment for verification experiments. FIG. 10 is a diagram showing an example of a bird's-eye view image according to a comparative example. FIG. 11 is a diagram showing an example of a bird's-eye view image when the mobile device is moved from the position shown in FIG. FIG. 12 is a diagram showing an example of a video of the mobile device shown in FIG. 11 photographed from a third-person viewpoint. FIG. 13 is a diagram illustrating an example of a bird's-eye view image according to the embodiment; FIG. 14 is a diagram showing an example of a bird's-eye view image when the mobile device is moved from the position shown in FIG. FIG. 15 is a diagram showing an example of a video of the mobile device shown in FIG. 14 photographed from a third-person viewpoint.

[Experiment environment]
FIG. 9 is a diagram showing an experimental environment for a bird's-eye view video verification experiment provided by the bird's-eye view video presentation system 1 according to the embodiment. As illustrated in FIG. 9 , the verification experiment was conducted indoors in an environment where a set of obstacles 50 were arranged in front of the mobile device 10 . Obstacles 51 and 52 constituting the obstacle 50 each have a shape that simulates a part of a pipe, and are installed in a suspended state with different heights and horizontal positions. One obstacle 51 was installed so as to have the same height as the cameras 11 a to 11 d provided in the mobile device 10 . In addition, in the experimental environment, among the three-dimensional point clouds acquired by the detection device 12, while confirming the three-dimensional point _{clouds on the xwyw plane ,} the _{ywzw plane} , and the _zwxw plane, The extrinsic parameter matrix Md was calculated by estimating “R ₃ ” and “t _d→w ” described above.

As illustrated in FIGS. 10 and 11, according to the bird's-eye view image 301 and the bird's-eye view image 302 generated by the conventional method proposed in Reference Z, either before the movement of the mobile device 10 or after the movement of the mobile device 10 3, among the set of obstacles 50 placed in front of the mobile device 10, the obstacle 51 placed relatively close to the mobile device 10 is not drawn in the bird's-eye view image. Further, in the bird's-eye view image 302 shown in FIG. 11, among the obstacles 50, the obstacle 52 arranged at a position relatively distant from the mobile device 10 is drawn in a vertically distorted state. . As can be seen from the image 303 illustrated in FIG. 12, in the situation illustrated in FIG. 11, the mobile device 10 and the obstacle 50 collide. is not accurately presented, it is difficult for the operator to avoid collision between the mobile device 10 and the obstacle 50 by referring to the bird's-eye view image 301 and the bird's-eye view image 302 .

On the other hand, as illustrated in FIGS. 13 and 14 , according to the bird's-eye view image 311 and the bird's-eye view image 312 generated by the processing of the information processing apparatus 100 according to the embodiment, before the movement of the mobile device 10 and when the mobile device 10 moves In each of the following, an obstacle 51 and an obstacle 52 are drawn that form a set of obstacles 50 placed in front of the mobile device 10 . In particular, as can be seen from the video shown in FIG. 15, in the bird's-eye view image 312 shown in FIG. In this way, the bird's-eye video presentation system 1 according to the embodiment can accurately determine the positions of objects existing around the mobile device 10, such as the bird's-eye image 311 illustrated in FIG. 13 and the bird's-eye image 312 illustrated in FIG. The operator can be presented with a bird's-eye view image to be presented to the operator. Therefore, at an actual disaster site or the like, the operator can recognize the surrounding environment of the mobile device 10 based on the bird's-eye view image generated by the information processing device 100 . The effect of avoiding collisions with existing objects can be expected.

Further, according to the bird's-eye view image 311 and the bird's-eye view image 312 according to the embodiment, the shielding area 39 behind the obstacle 50 is drawn in a transparent color, and the operator recognizes the texture information of the sign 41 placed on the floor. can do. For this reason, the bird's-eye view video presentation system 1 according to the embodiment presents to the operator a bird's-eye view video that enables the operator to recognize information important for the operation of the mobile device 10 at a disaster site or the like, such as information on signs placed on the floor. can be done.

Note that the distance d between the dotted lines in FIG. 13 is 235 pixels in pixel conversion, which corresponds to 1198 mm (millimeters) when converted to the scale of the real world. The measured value of the distance d between the dotted lines in FIG. 13 is 1200 mm. It proved to be presented as accurately as possible.

As described above, according to the bird's-eye view video presentation system 1 according to the embodiment, the detection device 12 performs omnidirectional three-dimensional measurement, and the three-dimensional information acquired by the three-dimensional measurement is used to determine the position of the mobile device 10. The operator can be provided with an image in which the positions of surrounding objects are depicted as accurately as possible. As a result of being able to provide an operator with an image in which the positions of objects around the mobile device 10 are drawn as accurately as possible, it is possible to remotely operate the mobile device 10 at an indoor disaster site that is difficult for people to enter. To easily avoid the operator's misunderstanding of the surrounding environment of the remote mobile device and the collision of the remote mobile device with obstacles when operating the remote mobile device that can be moved by the remote mobile device to perform recovery work and investigation at the disaster site. It is possible to improve the safety of restoration work and investigation of disaster sites.

The above embodiments are intended to facilitate understanding of the data management system, the data recording device 200 and the data search device 300 according to the present invention. The search device 300 is not limitedly interpreted. The data management system, data recording device 200, and data retrieval device 300 according to the present invention can be arbitrarily changed or improved without departing from the scope of the invention. The components of the data management system, data recording device 200, and data retrieval device 300 according to the present invention include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those that fall within the so-called equivalent range. .

10 mobile device 11a to 11d camera 12 detection device 13 communication unit 14 actuator 15 storage unit 16 control unit 100 information processing device 111 communication unit 113 display unit 115 storage unit 117 control unit 117a first overhead image generation unit 117b second overhead image generation Unit 117c Third video generation unit 117d Integrated bird's-eye view video generation unit

Claims (2)

A bird's-eye view video presentation system including a remotely operable mobile device and an information processing device capable of data communication with the mobile device,
The moving device
a communications department;
a plurality of cameras for wide-angle imaging of the surroundings of the mobile device;
a detection device that acquires three-dimensional information indicating the positions of objects around the moving device by irradiating radial laser light in all directions parallel to the traveling surface of the moving device;
The information processing device is
a communications department;
a first bird's-eye view image generation unit that generates a first bird's-eye view image by viewpoint-converting the image acquired by the camera into a video viewed from a virtual bird's-eye viewpoint that overlooks the mobile device;
a second bird's-eye view image generator that generates a second bird's-eye view image by projecting the three-dimensional information onto a coordinate system that defines the bird's-eye view point;
a third image generation unit configured to generate a third image in which a blind spot area from the moving device is set based on the relative positional relationship between the moving device and the object in the second bird's-eye view image;
an integrated video generation unit configured to generate an integrated bird's-eye video by integrating the first bird's-eye video, the second bird's-eye video, and the third video;
and a display unit that displays the integrated bird's-eye view image ,
The integrated image generation unit acquires an image obtained by superimposing the second overhead image on the third image, and superimposes the acquired image on the first overhead image. system. The second bird's-eye view image generation unit
2. The bird's-eye view image presentation system according to claim 1, wherein texture information is added to said three-dimensional information based on the image acquired by said camera.

Images