JP7162218B2 - Bird's eye view presentation system - Google Patents

Description

The present invention relates to a bird's-eye view video presentation system.

In recent years, along with the use of remote-operable mobile devices (hereinafter referred to as "remote mobile devices") in recovery work at disaster sites, video presentation of remote mobile devices for the purpose of improving the work efficiency of recovery work Technology is being actively developed. For example, a bird's-eye view video presentation system, which is exemplified as one of the video presentation technologies for disaster sites, is equipped with multiple cameras equipped with wide-angle lenses in a remote mobile device, and processes the video acquired from the multiple cameras. Thus, it is possible to present a simulated image of the remote mobile device as if it were viewed from a third person's point of view in the sky.

Japanese Patent Publication No. 2017-538208 JP 2017-173298 A JP 2018-50119 A

In the conventional bird's-eye view image presentation system, objects around the remote mobile device are not accurately drawn on the bird's-eye view image, and as a result, there are cases where the operator of the remote mobile device cannot be accurately informed of the surrounding conditions surrounding the remote mobile device. be.

The present disclosure has been made in view of the above problems, and aims to provide a bird's-eye view video presentation system that can more accurately present the surroundings surrounding a mobile device that can be moved by remote control. do.

In order to achieve the above object, a bird's-eye view video presentation system according to one aspect of the present disclosure includes a plurality of first cameras that can be remotely controlled and shoot the surroundings at a wide angle, and shoots in the traveling direction and in the depth direction. a moving device including a second camera that acquires the depth information of the moving device, a marker installed in an area photographed by the first camera and the second camera and having known dimensional information, the moving device and data an information processing device capable of communication, wherein the information processing device converts an image acquired by the first camera into an image viewed from a virtual bird's eye view of the mobile device. a first bird's-eye view generating unit that generates a first bird's-eye view image; and a second bird's-eye view image by converting the image including the depth information in the depth direction acquired by the second camera into an image viewed from the bird's-eye viewpoint. and correcting the second bird's-eye video based on the relative positional relationship between the markers included in the first bird's-eye video and the markers included in the second bird's-eye video, and a third bird's-eye view image generator that generates a third bird's-eye view image including three-dimensional information by projecting the second bird's-eye image corrected to the first bird's-eye image.

As a desirable aspect of the bird's-eye view video presentation system, the third bird's-eye view video generator generates a height of a predetermined threshold or more and a depth of a predetermined threshold or less from the ground in the third bird's-eye view video based on the three-dimensional information. Highlight obstacles.

Advantageous Effects of Invention According to the present disclosure, it is possible to present as accurately as possible the circumstances surrounding a mobile device that can be moved by remote control.

FIG. 1 is a diagram illustrating a configuration example of a bird's-eye view video presentation system according to an embodiment. FIG. 2 is a block diagram schematically showing the functional configuration of the bird's-eye view video presentation system according to the embodiment. FIG. 3 is a diagram showing the concept of each coordinate system according to the embodiment. FIG. 4 is a diagram showing part of an example of the first bird's-eye view image according to the embodiment. FIG. 5 is a diagram showing an example of a second bird's-eye view image according to the embodiment. FIG. 6 is a diagram illustrating an example of a third bird's-eye view image according to the embodiment; FIG. 7 is a flowchart illustrating an example of optimization processing for variables in the transform function according to the embodiment. FIG. 8 is a flowchart illustrating an example of bird's-eye view image generation processing executed by the information processing apparatus according to the embodiment.

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of a bird's-eye view video presentation system according to the present invention will be described in detail below with reference to the drawings. It should be noted that the present invention is not limited to the description of the following embodiments. Components in the following embodiments include those that can be easily replaced by those skilled in the art, or those that are substantially the same. Furthermore, the constituent elements in the embodiments described below can be omitted, replaced, or changed in various ways without departing from the spirit of the present invention. In the following embodiments, necessary constituent elements will be described in order to exemplify one embodiment of the bird's-eye view video presentation system according to the present invention, and other constituent elements will be omitted.

[System configuration]
FIG. 1 is a diagram illustrating a configuration example of a bird's-eye view video presentation system according to an embodiment. FIG. 2 is a block diagram schematically showing the functional configuration of the bird's-eye view video presentation system according to the embodiment.

As shown in FIG. 1 , a bird's-eye view video presentation system 1 according to the embodiment includes a mobile device 10 capable of remote control and an information processing device 100 capable of data communication with the mobile device 10 . The mobile device 10 and the information processing device 100 are connected to the communication network 5 so as to be able to communicate data with each other. The communication network 5 may be constructed including public communication lines, dedicated communication lines, and the like.

The mobile device 10 travels in a disaster site or the like where the object 3 or the like exists. The mobile device 10 includes a plurality of first cameras 11 a to 11 d each having a fisheye lens, a second camera 12 , a communication section 13 , an actuator 14 , a storage section 15 and a control section 16 . The mobile device 10 may run according to an operation command received from the information processing device 100 or may run according to an operation command received from a remote device other than the information processing device 100 . As a mobile device 10, reference document 1 (New Energy and Industrial Technology Development Organization: Unmanned disaster response system research and development project Development of measurement and work element technology Development of amphibious monitoring device, 2011-2012 achievement report (2013)) is exemplified by the equipment developed in the disaster response project.

The first cameras 11a to 11d photograph the surroundings (surrounding environment) of the mobile device 10 at a wider angle than a predetermined photographing range. The first cameras 11a to 11d each have a fisheye lens with a wide angle of view of around 180 degrees, for example. In this embodiment, the first cameras 11a to 11d are provided on the left and right oblique front and oblique rear sides with respect to the moving direction of the mobile device 10, but may be provided on the front, rear, left and right, respectively. In the following description, the image based on the image data acquired by the first cameras 11a to 11d may be referred to as "first camera image 11G". When the first cameras 11a to 11d are installed in the mobile device 10, for example, refer to Reference 2 (Ren Komatsu, Hiromitsu Fujii, Jun Yamashita, Hajime Asama: Overhead View Video Presentation for Robot Remote Control in Case of Failure Due to Camera Layout Design It can be installed based on the layout proposed in System Development, Journal of Precision Engineering, 81, 12 (2015) 1206).

The second camera 12 is provided in the traveling direction of the mobile device 10 . The second camera 12 captures the traveling direction of the mobile device 10 and acquires depth information in the depth direction. The second camera 12 is a stereo camera that also records information in the depth direction by simultaneously photographing the object from a plurality of different directions. The second camera 12 only needs to capture a depth image including at least information in the depth direction. The second camera 12 may be, for example, an RGB-D sensor. An RGB-D sensor is a sensor that acquires RGB images in addition to depth images. RGB-D sensors include a twin-lens infrared camera that captures depth images and a single-lens RGB camera that captures RGB images. The horizontal viewing angle of the second camera 12 is, for example, 88.2 degrees or more and 94.2 degrees or less. The viewing angle in the vertical direction is, for example, 62.5 degrees or more and 68.5 degrees or less. The second camera 12 can acquire, as three-dimensional information, information on a three-dimensional point group, which is an aggregate of three-dimensional points indicating the position of the surface of an object. Each three-dimensional point forming the three-dimensional point group acquired by the second camera 12 is represented by a three-dimensional coordinate value indicating the position on the surface of the object. A three-dimensional coordinate value is composed of coordinate values corresponding to each of the X-axis direction, Y-axis direction, and Z-axis direction at an arbitrary origin.

The communication unit 13 performs data communication for exchanging various data with the information processing apparatus 100 via the communication network 5 . The communication unit 13 can receive operation commands related to running. The communication unit 13 can send operation commands to the control unit 16 . The communication unit 13 can transmit the video data acquired by the first cameras 11 a to 11 d and the video data and depth information acquired by the second camera 12 to the information processing device 100 . The communication unit 13 can support various communication methods that enable communication with the information processing device 100 . Either wireless communication or wired communication may be applied as the communication method of the communication unit 13 .

The actuator 14 converts a control signal from the control unit 16 into power for causing the mobile device 10 to travel on the travel surface 2 such as a disaster site. Actuator 14 includes, for example, an electric motor.

The storage unit 15 stores programs and data for realizing various processes executed by the control unit 16 . The functions provided by the program stored in the storage unit 15 are the function of controlling the running of the mobile device 10, the image data acquired by the first cameras 11a to 11d, the image data acquired by the second camera 12, and the depth It includes a function of transmitting information to the information processing apparatus 100 . The storage unit 15 is a non-volatile or volatile memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (registered trademark) (Electrically Erasable Programmable Read Only Memory). may be implemented by optical semiconductor memory, a magnetic disk, a floppy disk, an optical disk, a compact disk, a minidisk, or a DVD.

The control unit 16 executes various processes related to the mobile device 10 based on the programs and data stored in the storage unit 15 . In particular, in the present embodiment, the control unit 16 transmits video data acquired by the first cameras 11a to 11d and three-dimensional information detected by the second camera 12 to the information processing apparatus 100 via the communication unit 13. Send. The control unit 16 may be implemented by a processor such as a CPU (Central Processing Unit) microprocessor, microcomputer, DSP, or system LSI (Large Scale Integration).

The mobile device 10 may include a crawler that can follow a rough surface of a disaster site with steps, a carriage provided with the crawler at the bottom, and a body part arranged on the carriage. The first cameras 11a to 11d and the second camera 12 described above may be installed on the body.

The information processing device 100 uses the video data and three-dimensional information received from the mobile device 10 to generate a bird's-eye view image presenting the positions of objects around the mobile device 10 . Information processing apparatus 100 includes communication unit 111 , display unit 113 , storage unit 115 , and control unit 117 .

The communication unit 111 executes data communication for exchanging various data with the mobile device 10 via the communication network 5 . The communication unit 111 can receive video data and 3D information transmitted from the mobile device 10 . The communication unit 111 can support various communication schemes that enable communication with the mobile device 10 . Either wireless communication or wired communication may be applied as the communication method of the communication unit 111 .

The display unit 113 displays various information. The display unit 113 can display a bird's-eye view image generated by the control unit 117, which will be described later. The display unit 113 may include a display device such as a liquid crystal display (LCD), an organic electro-luminescence display (OELD), or an inorganic electro-luminescence display (IELD). Display unit 113 may include an input device such as a touch screen.

Storage unit 115 stores programs and data for realizing various processes executed by control unit 117 . The functions provided by the programs stored in the storage unit 115 include a function of generating a bird's-eye view image presenting the positions of objects around the mobile device 10 . The data stored by the storage unit 115 includes video data and 3D information received from the mobile device 10 . The storage unit 15 is a non-volatile or volatile memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (registered trademark) (Electrically Erasable Programmable Read Only Memory). may be implemented by optical semiconductor memory, a magnetic disk, a floppy disk, an optical disk, a compact disk, a minidisk, or a DVD.

The control unit 117 executes various processes related to the mobile device 10 based on the programs and data stored in the storage unit 115 . In particular, in the present embodiment, the control unit 117 executes processing for generating a bird's-eye view image presenting the positions of objects around the mobile device 10 . The control unit 117 may be implemented including a processor such as a CPU (Central Processing Unit) microprocessor, a microcomputer, a DSP (Digital Signal Processor), or a system LSI (Large Scale Integration). The control unit 117 reads out a program stored in the storage unit 115, develops it in a working memory such as a RAM, and causes a processor such as a CPU to execute instructions included in the program developed in the working memory. Thereby, the control unit 117 can execute each process based on each function provided by the first bird's-eye image generation unit 117a, the second bird's-eye image generation unit 117b, and the third bird's-eye image generation unit 117c, which will be described below. .

The control unit 117 has a first bird's-eye image generation unit 117a, a second bird's-eye image generation unit 117b, and a third bird's-eye image generation unit 117c.

The first bird's-eye view image generation unit 117a generates image data received from the mobile device 10, that is, images (first camera images) acquired by the first cameras 11a to 11d from a virtual bird's-eye view point of the mobile device 10. A first bird's-eye view image is generated by converting the viewpoint into the viewed image. Under the assumption that all objects captured by the first cameras 11a to 11d of the mobile device 10 exist on a certain plane in the robot coordinate system 21, the first bird's-eye view image generator 117a generates the first camera image 11G. After performing perspective projection onto a plane, viewpoint conversion processing is performed to generate a first bird's-eye view image. The first bird's-eye view image generation unit 117a is based on Reference Document 3 (Takaki Sato, Hiromitsu Fujii, Alessandro Moro, Jun Yamashita, Hajime Asama: Construction of a superimposed omnidirectional bird's-eye video presentation method using multiple fisheye cameras and LRF, The first bird's-eye view image can be generated by using the conventional method proposed in the 13th Society of Instrument and Control Engineers System Integration Division Lecture Proceedings (2012).

[Definition of coordinate system]
FIG. 3 is a diagram showing the concept of each coordinate system according to the embodiment. A coordinate system 21 shown in FIG. 3 is the coordinate system of the mobile device 10 . In FIG. 3, the _{xwyw plane} of the robot coordinate system 21 is the same as the floor surface (running surface 2 of the mobile device 10), and the positive direction of the _zw axis is vertically upward with respect to the floor surface. Hereinafter, the coordinate system 21 shown in FIG. 3 will be appropriately referred to as "robot coordinate system 21". A coordinate system 23 shown in FIG. 3 is, for example, a coordinate system that defines the position and orientation of the first camera 11a. Hereinafter, the coordinate system 23 shown in FIG. 3 will be appropriately referred to as "first camera coordinate system 23". Although illustration is omitted, a coordinate system that defines the positions and orientations of the first cameras 11b to 11d is set like the first camera coordinate system . A coordinate system 25 shown in FIG. 3 is a coordinate system that defines detection points acquired by the second camera 12 . Hereinafter, the coordinate system 25 shown in FIG. 3 will be appropriately referred to as "second camera coordinate system 25". A coordinate system 27 shown in FIG. 3 is, for example, a coordinate system that defines points on the first camera image 11G captured by the first camera 11a. Hereinafter, the coordinate system 27 shown in FIG. 3 will be appropriately referred to as "first camera image coordinate system 27". Although not shown, like the first camera image coordinate system 27, a coordinate system that defines points on the first camera image 11G captured by the first cameras 11b to 11d is set. A coordinate system 29 shown in FIG. 3 is a coordinate system that defines points on the second camera image 12</b>G captured by the second camera 12 . Hereinafter, the coordinate system 29 shown in FIG. 3 will be appropriately referred to as "second camera image coordinate system 29". A coordinate system 31 shown in FIG. 3 is a coordinate system that defines the positions of points on a virtual camera image 33G captured by a virtual camera 33 that is assumed to be located at a position where the mobile device 10 is viewed from above. Hereinafter, the coordinate system 31 shown in FIG. 3 will be appropriately referred to as "virtual camera image coordinate system 31". Let point P _f =[x _f , y _f , z _f ] ^T represent the point P _w =[x _w , y _w , z _w ] ^T in the robot coordinate system 21 in the first camera coordinate system 23 . Let P _d =[x _d , y _d , z _d ] ^T represent the point P _w =[x _w , y _w , z _w ] ^T in the robot coordinate system 21 in the second camera coordinate system 25 . _Let the representation of the point _Pw =[ _xw , _yw , _zw ] ^T in the robot coordinate system 21 in the first camera image coordinate system 27 be the point mf=[ _uf , _vf ] ^T . Let the representation of the point _Pw =[ _xw , _yw , _zw ] ^T in the robot coordinate system 21 in the second camera image coordinate system 29 be the point _md =[ _ud , _vd ] ^T . The first bird's-eye view image generator 117a converts each point in each coordinate system into a point _mv =[ _uv , _vv ] ^T in the virtual camera image coordinate system 31. FIG. In the following description, the coordinate representation of point N in an arbitrary coordinate system by a homogeneous coordinate system is assumed ^to be N .

[Set marker]
A marker 35 is installed in a predetermined area photographed by the first cameras 11 a to 11 d and the second camera 12 of the mobile device 10 . Marker 35 has at least known dimensional information. In this embodiment, the marker 35 is a rectangular thin plate with known side lengths. The shape of the marker 35 is not particularly limited. A plurality of markers 35 may be installed in a predetermined area. The markers 35 may include identifiers for distinguishing each marker 35, for example. The dimension information and identifier of the marker 35 are pre-stored in the storage unit 115 .

[Creation of first bird's-eye view video]
The first bird's-eye view image generation unit 117a generates a point P _f =[x _f , y _f , z _f ] ^T in the first camera coordinate system 23 shown in Equation (1) below, and the first camera image coordinate system 27 Based on the relationship with the point m _f =[u _f , v _f ] ^T at , a perspective projection image is generated from which the distortion peculiar to the image captured by the fisheye lens is removed. Formula (1) shown below is derived from reference 4 (Davide Schramuzza, Agostion Martinelli, and Roland Siegwart: A flexible technique for accurate omnidirectional camera calibration and structure from motion, Proceeding of IEEE International Conference of Computer Vision Systems (2006) 45) and Reference 5 (Davide Schramuzza, Agostion Martinelli, and Roland Siegwart: A toolbox for easily calibrating omnidirectional cameras, Proceedings of IEEE/RSJ International Conference of Intelligent Robots and Systems (2006) 5695).

“f(u _f , u _v )” in Equation (1) is a function related to the distance √{u _f ² +u _v ² } from the origin in the first camera image coordinate system 27, and “a ' are constants set to express the relationship between each point in the first camera coordinate system 23 and each direction vector in the first camera image coordinate system 27 by an equation.

Subsequently, the first bird's-eye view image generation unit 117a perspectively projects the perspective projection image generated based on the above equation (1) onto a specific plane. The robot coordinate system 21 and the first camera image coordinate system 27 can be related by a 3×4 matrix of perspective projection matrix _Hw→f , and the relationship between the robot coordinate system 21 and the first camera image coordinate system 27 is , the relationship of the following equation (2) holds.

Assume that all the objects photographed by the first cameras 11a to 11d of the mobile device 10 exist on a plane in the robot coordinate system 21. FIG. For example, when perspectively projecting the first camera image 11G onto a plane (floor surface) of z _w =0, the third column component of the perspective projection matrix H _w→f can be omitted as shown in the following equation (3). .

Here, H'w _→f and P ^~ ' _w in equation (3) are matrices in which the third column of H'w _→f and the third row of P ^~ ' _w are omitted. Equation (3) determines the relationship between the robot coordinate system 21 and the first camera image coordinate system 27 .

Subsequently, the first bird's-eye view image generating unit 117a performs viewpoint conversion processing of the perspective projection image perspectively projected onto a specific plane based on the relationship shown in the above-described formula (3), and generates the first bird's-eye view image. Generate. Here, a method for generating the first bird's-eye view image viewed in the _zw -axis direction will be described. First, the first bird's-eye view image generation unit 117 a obtains the relationship between the robot coordinate system 21 and the virtual camera image coordinate system 31 . Between the robot coordinate system 21 and the virtual camera image coordinate system 31, the following formula (4) holds.

H′w _→v in Equation (4) is a matrix obtained by omitting the third column component of the perspective projection matrix Hw _→v that relates the robot coordinate system 21 and the virtual camera image coordinate system 31 . Using the above equations (3) and (4), the relationship shown in equation (5) is established between the first camera image coordinate system 27 and the virtual camera image coordinate system 31 .

However, in Equation (5), H′ _f→w =(H′ _w→f ) ⁻¹ . The first bird's-eye view image generation unit 117a can generate a first bird's-eye view image of the mobile device 10 from the sky, from the first camera image 11G (video data) received from the mobile device 10, according to Equation (5). The first bird's-eye view image is the virtual camera image 33G.

Now, as can be seen from equation (5), we need to find H'w _→v and H'f _→w . In this embodiment, reference 6 (Kosuke Sato, Hiromitsu Fujii, Alessandro Moro, Kazuya Sugimoto, Akira Nozue, Yoichi Mimura, Katsumi Obata, Jun Yamashita, Hajime Asama: Development of overhead view video system for unmanned construction, Nihon Kikai Academic Proceedings, 81, 823 (2015) 1), and reference 7 (Keisuke Asari, Yohei Ishii, Hitoshi Hongo, Hiroshi Kano: Simplification of calibration environment for bird's-eye image generation, Proceedings of the 13th Image Sensing Symposium ( 2017) IN1-13), H'w _→v and H'f _→w are estimated by a nonlinear optimization method. Based on the estimated H′w _→v and H′f _→w matrices, each of the first camera image 11G acquired by the first cameras 11a to 11d and the virtual camera image 33G acquired by the virtual camera 33 is calculated. A one-to-one correspondence of pixels can be obtained in advance. Note that the one-to-one correspondence between each pixel of the first camera image 11G and the virtual camera image 33G may be stored in advance as a lookup table so that it can be used when generating the first bird's-eye view image. In the example of the method for generating the first bird's-eye view image described above, the first bird's-eye view image is limited to the image viewed in the _zw -axis direction, but the direction of the bird's-eye view is not limited to this.

[Creation of second bird's-eye view video]
The second bird's-eye view image generation unit 117b generates a point P _d =[x _d , y _d , z _d ] ^T in the second camera coordinate system 25 and a point m _d =[u _d , v in the second camera image coordinate system 29 . _d ] Based on the relationship with ^T , viewpoint conversion processing is performed on the second camera image 12G to generate a second bird's-eye view image. At this time, the coordinate system of the second bird's-eye view image is aligned with the virtual camera image coordinate system 31 . Viewpoint conversion processing from the second camera image 12G to the second bird's-eye view image is performed only by translation and rotation. It is also possible to match the bird's-eye viewpoint of the first bird's-eye video with the viewpoint of the second camera 12 . In this case, the first bird's-eye view image generation unit 117a performs viewpoint conversion processing on the perspective projection image generated from the first camera image 11G to the second camera coordinate system 25 to generate the first bird's-eye image. The second bird's-eye view image generation unit 117b omits the viewpoint conversion processing from the second camera image 12G to the second bird's-eye view image.

[Creation of third bird's-eye view video]
FIG. 4 is a diagram showing part of an example of the first bird's-eye view image according to the embodiment. FIG. 5 is a diagram showing an example of a second bird's-eye view image according to the embodiment. FIG. 6 is a diagram illustrating an example of a third bird's-eye view image according to the embodiment; The first bird's-eye view image G1 shown in FIG. 4 is created in the same manner as the above-described first bird's-eye view image (virtual camera image 33G (see FIG. 3)). The second bird's-eye view image G2 shown in FIG. 5 is created in the same manner as the above-described second bird's-eye view image. A marker 35 is included in the first bird's-eye view image G1 and the second bird's-eye view image G2.

The _coordinates of points M _i (M ₁ , M ₂ , . The point has the same coordinates in the robot coordinate system 21 as the _coordinates of the point S _i (S ₁ , S ₂ , . The coordinates of the point M _i have a difference with respect to the coordinates of the corresponding point S _i in the virtual camera image coordinate system 31 . The third bird's-eye view image generator 117c sets the points M _i and _Si based on the information of the markers 35 stored in the storage unit 115 and the first bird's-eye video G1 and the second bird's-eye video G2. The point M _i is the corner of the marker 35 of the first bird's-eye view image G1 in this embodiment. The point _Si is the corner of the marker 35 of the second bird's-eye view image G2 in this embodiment. The points M _i may be set at the four corners of all the markers 35, although only the points M ₁ , M ₂ , M ₃ and M ₄ are shown in FIG. Points S _i may be set at the four corners of all the markers 35, although only points S ₁ , S ₂ , S ₃ and S ₄ are shown in FIG. The points Mi and _Si may be set while the operator _watches the first bird's-eye view image G1 and the second bird's-eye view image G2. The points M _i and S _i do not have to be set at the corners of the marker 35 as long as they can be identified by the first bird's-eye view image G1 and the second bird's-eye view image G2. For example, a figure with known dimensions may be drawn on the marker 35 and predetermined positions of the figure may be set as points _Mi and _Si . The third bird's-eye view image generation unit 117c transforms the point group M such that the difference from the point group S is minimized in the point set _{ M, S _} of the points Mi and Si. The transformation function T is defined by Equation (6) below.

"θ" in equation (6) is a variable that defines the position, orientation, and magnification of the six degrees of freedom. The distance function is calculated by Equation (7) below.

“m” in Equation (7) is mεT(M, θ). "s" in equation (7) is sεS. 'θ' is randomly modified by the optimization process. FIG. 7 is a flowchart illustrating an example of optimization processing for variables in the transform function according to the embodiment. “θbest” in FIG. 7 is the maximum value of “θ”. "err" in FIG. 7 is the threshold of the error tolerance. The values of "initial θ", "θbest" and "err", which are the initial values of "θ", are arbitrarily set in advance before the optimization processing shown in Fig. 7 is executed. When the mobile device 10 starts moving, the third bird's-eye view image generator 117c proceeds to step S51 and starts processing. After finishing the process of the flowchart shown in FIG. 7, the third bird's-eye view image generator 117c starts the process from step S51 until the mobile device 10 stops. "θ" is updated by adding random numbers [-0.5, 0.5] to the position and orientation, and [-0.05, 0.05] to the scale factor. A random number is represented by "θrand".

In step S51, the third bird's-eye view image generator 117c applies Equation (6). In step S52, the third bird's-eye view image generator 117c calculates Equation (7). In step S53, the third bird's-eye view image generator 117c determines whether the value converted in step S52 is smaller than the threshold err. When it is determined in step S53 that dist is smaller than the threshold err, the third bird's-eye view image generator 117c proceeds to step S54. When it is determined in step S53 that dist is greater than or equal to the threshold err, the third bird's-eye view image generator 117c proceeds to step S55. In step S54, the third bird's-eye view image generator 117c updates the value of "θbest" to "θ". After executing step S54, the third bird's-eye view image generator 117c ends the series of processes, and restarts the process from step S51. In step S55, the third bird's-eye view image generator 117c updates the value of "θ" to "θbest+θrand". After executing step S55, the third bird's-eye view image generator 117c ends the series of processes, and restarts the process from step S51.

The point group M is set at the corners of the marker 35 included in the first bird's-eye view image G1 in this embodiment. The point group S is set at the corners of the marker 35 included in the second bird's-eye view image G2 in this embodiment. That is, the third bird's-eye image generator 117c corrects the second bird's-eye image G2 based on the relative positional relationship between the markers 35 included in the first bird's-eye image G1 and the markers 35 included in the second bird's-eye image G2. do. For example, the third bird's-eye image generation unit 117c changes the coordinates of the four points corresponding to the corners of the marker 35 included in the second bird's-eye image G2 to the coordinates of the four points corresponding to the corners of the marker 35 included in the first bird's-eye image G1. The second bird's-eye view image G2 is corrected so as to match the coordinates. The third bird's-eye view image generator 117c projects the corrected second bird's-eye view image G2 onto the first bird's-eye view image G1. The third bird's-eye image generator 117c projects the corrected second bird's-eye image G2 onto the first bird's-eye image G1 to generate the third bird's-eye image G3 shown in FIG. Since the third bird's-eye view image G3 includes three-dimensional information, it is possible to distinguish between the ground and obstacles. If the three-dimensional coordinate information of the position of the marker 35 included in the first bird's-eye view image G1 is not included in the point group information in the virtual camera image coordinate system 31, linear interpolation is performed between the closest points.

The third bird's-eye view image generator 117c highlights obstacles in the third bird's-eye view image G3 by a predetermined method. The obstacle is an object 40 existing above the ground, a hole recessed below the ground, or the like. The third bird's-eye view image generation unit 117c highlights a point group having a height above the predetermined threshold from the ground as the obstacle 41 on the ground side. The predetermined threshold is, for example, 20 cm above the ground. The third bird's-eye view image generation unit 117c highlights point groups whose depth is equal to or less than a predetermined threshold from the ground as obstacles on the underground side. The predetermined threshold is, for example, 20 cm below ground. A predetermined threshold value is preset in the storage unit 115 . The highlighted display indicating the obstacle 41 on the ground side is, for example, a red solid display. The highlighting indicating the underground obstacles is, for example, a blue-filled display. The highlighting may be, for example, a line surrounding the obstacle. The method of highlighting is preset in storage unit 115 .

FIG. 8 is a flowchart illustrating an example of bird's-eye view image generation processing executed by the information processing apparatus according to the embodiment. The processing shown in FIG. 8 is executed by the control unit 117 based on the programs and data stored in the storage unit 115 .

As shown in FIG. 8, in step S101, the control unit 16 of the mobile device 10 selects a predetermined area using the first cameras 11a to 11d and the second camera 12 based on a predetermined program stored in the storage unit 15. take a picture. The control unit 16 transmits the first camera image 11G captured by the first cameras 11a to 11d and the second camera image 12G captured by the second camera 12 to the communication unit 13 via the communication network 5. The data is transmitted to the communication unit 111 of the processing device 100 . The control unit 117 of the information processing apparatus 100 causes the storage unit 115 to store the acquired first camera image 11G and second camera image 12G.

In step S102, the first bird's-eye view image generation unit 117a of the control unit 117 converts the viewpoint of the first camera image 11G stored in the storage unit 115 into a bird's-eye view point, and generates the first bird's-eye view image G1. A bird's-eye view corresponds to a virtual bird's-eye view that looks down on the mobile device 10 from an arbitrary direction. The control unit 117 causes the display unit 113 to present the first bird's-eye view image G1.

In step S103, the second bird's-eye view image generation unit 117b of the control unit 117 converts the viewpoint of the second camera image 12G stored in the storage unit 115 into a bird's-eye view point, and generates a second bird's-eye view image G2. The control unit 117 causes the display unit 113 to present the second bird's-eye view image G2.

In step S104, the third bird's-eye view image generation unit 117c of the control unit 117 projects the second bird's-eye view image G2 on the first bird's-eye image G1 based on the position information of the marker 35. FIG. That is, the third bird's-eye-view image generator 117c creates the first bird's-eye image G1 based on the relative positional relationship between the markers 35 included in the first bird's-eye image G1 and the markers 35 included in the second bird's-eye image G2. 2 Project the bird's-eye view image G2. The third bird's-eye image generator 117c projects the second bird's-eye image G2 onto the first bird's-eye image G1 to generate the third bird's-eye image G3. Since the third bird's-eye view image G3 includes three-dimensional information, it is possible to distinguish between the ground and obstacles.

In step S105, the third bird's-eye view image generation unit 117c of the control unit 117 generates a height above a predetermined threshold and a height below a predetermined threshold from the ground based on the three-dimensional coordinate information of the point group included in the third bird's-eye view image G3. Extract points at a depth of . The predetermined threshold value is a value set in the storage unit 115 in advance. The third bird's-eye view image generation unit 117c highlights the extracted points by a predetermined method stored in the storage unit 115. FIG. The control unit 117 causes the display unit 113 to present the third bird's-eye view image G3. When step S105 is executed, the bird's-eye view video presentation system 1 ends the series of processes shown in FIG.

As described above, the bird's-eye view video presentation system 1 includes the mobile device 10 , the marker 35 and the information processing device 100 . Mobile device 10 is remotely operable. Mobile device 10 includes a plurality of first cameras 11 a to 11 d and a second camera 12 . The first cameras 11a to 11d capture wide-angle images of the surroundings. The second camera 12 photographs the direction of travel. The second camera 12 acquires depth information in the depth direction. The markers 35 are installed in areas photographed by the first cameras 11a to 11d and the second camera 12. FIG. The dimensional information of the marker 35 is known. The information processing device 100 is capable of data communication with the mobile device 10 . The information processing apparatus 100 includes a first bird's-eye image generation unit 117a, a second bird's-eye image generation unit 117b, and a third bird's-eye image generation unit 117c. The first bird's-eye view image generator 117a converts the images (first camera images 11G) acquired by the first cameras 11a to 11d into images viewed from a virtual bird's-eye view of the mobile device 10. A first bird's-eye view image G1 is generated. The second bird's-eye view image generation unit 117b generates the second bird's-eye view image G2 by viewpoint-converting the image including the depth information in the depth direction acquired by the second camera 12 into the image viewed from the bird's-eye viewpoint. The third bird's-eye image generator 117c corrects the second bird's-eye image G2 based on the relative positional relationship between the markers 35 included in the first bird's-eye image G1 and the markers 35 included in the second bird's-eye image G2, and By projecting the corrected second bird's-eye view image G2 onto the one bird's-eye view image G1, a third bird's-eye view image G3 including three-dimensional information is generated.

According to the bird's-eye view video presentation system 1, since a bird's-eye view video including three-dimensional information can be generated, it is possible to distinguish between the ground and obstacles in the bird's-eye view video. It is possible to provide the operator with an image in which the positions of objects around the mobile device 10 are more preferably drawn. As a result, at an indoor disaster site where it is difficult for people to enter, when performing recovery work and investigation of the disaster site by operating a remote mobile device that can be moved by remote control, such as the mobile device 10, remote control can be performed. It is possible to easily avoid the operator's misunderstanding of the surrounding environment of the mobile device and the collision of the remote mobile device with obstacles, thereby enhancing the safety of restoration work and investigation at the disaster site.

In addition, in the bird's-eye view video presentation system 1, the third bird's-eye video generation unit 117c generates obstacles having a height of a predetermined threshold or more and a depth of a predetermined threshold or less from the ground in the third bird's-eye view video G3 based on the three-dimensional information. Highlight things. As a result, the positions of obstacles around the mobile device 10 can be more preferably presented to the operator.

Although the embodiments of the present invention have been described above, the embodiments are not limited by the contents of these embodiments. For example, two or more second cameras 12 may be provided. Also, in the present embodiment, the first cameras 11a to 11d are fish-eye cameras equipped with fish-eye lenses, but like the second camera 12, stereo cameras may be used. Even in this case, the second bird's-eye image generated based on the image captured by the second camera is projected onto the first bird's-eye image generated based on the image captured by the first camera by the method described above. can be done.

1 bird's-eye view image presentation system 2 running surface 3 object 5 communication network 10 mobile device 11a to 11d first camera 11G first camera image 12 second camera 12G second camera image 13 communication unit 14 actuator 15 storage unit 16 control unit 21 robot coordinates system 23 first camera coordinate system 25 second camera coordinate system 27 first camera image coordinate system 29 second camera image coordinate system 31 virtual camera image coordinate system 33 virtual camera 33G virtual camera image 35 marker 40 object 41 obstacle 100 information processing Device 111 communication unit 113 display unit 115 storage unit 117 control unit 117a first bird's-eye image generation unit 117b second bird's-eye image generation unit 117c third bird's-eye image generation unit G1 first bird's-eye image G2 second bird's-eye image G3 third bird's-eye image

Claims (2)

a mobile device including a plurality of first cameras that can be remotely controlled and capture wide-angle images of the surroundings, and a second camera that captures images in the direction of travel and acquires depth information in the depth direction;
a marker that is installed in an area photographed by the first camera and the second camera and whose dimensional information is known;
an information processing device capable of data communication with the mobile device;
with
The information processing device is
a first bird's-eye view image generation unit that generates a first bird's-eye view image by viewpoint-converting the image acquired by the first camera into a video viewed from a virtual bird's-eye viewpoint that overlooks the mobile device;
a second bird's-eye view image generation unit configured to generate a second bird's-eye view image by viewpoint-converting the image including the depth information in the depth direction acquired by the second camera into an image viewed from the bird's-eye viewpoint;
correcting the second bird's-eye video based on a relative positional relationship between markers included in the first bird's-eye video and markers included in the second bird's-eye video; a third bird's-eye view image generator that generates a third bird's-eye view image including three-dimensional information by projecting the bird's-eye image;
A bird's-eye view video presentation system. 2. The third bird's-eye view image generator highlights obstacles having a height equal to or greater than a predetermined threshold and a depth equal to or less than a predetermined threshold from the ground in the third bird's-eye view image based on the three-dimensional information. 2. The bird's-eye view video presentation system described in .

Images