JP7320660B1 - Omnidirectional image object detection device and omnidirectional image object detection method - Google Patents

Abstract

An object can be appropriately detected in a 360-degree omnidirectional image. An omnidirectional image object detection device 1 includes a cubemap image conversion unit 21 for converting an equirectangular image captured by a 360-degree omnidirectional camera 11 into a cubemap image; An object detection unit 23a that detects an object in a panoramic image of a plane, a detection frame determination unit 24a that determines the coordinates of a rectangle surrounding the object detected by the object detection unit 23a in the panoramic image, and an equirectangular image of 360 degrees. A three-dimensional information synthesizing unit 25 for displaying the rectangle on the display screen and synthesizing the rectangle at a position corresponding to the 360-degree display screen. [Selection drawing] Fig. 1

Description

The present invention relates to an omnidirectional image object detection apparatus and an omnidirectional image object detection method.

In recent years, many devices and services that provide virtual space have become widespread. In addition to being provided by computer graphics, such virtual space can also be provided by equirectangular images captured by a 360-degree omnidirectional camera.

In recent years, object detection is often performed on two-dimensional images by AI (Artificial Intelligence) or the like. However, since the equirectangular image is distorted when viewed as a two-dimensional plane image, it was not possible to perform object detection appropriately.

In Patent Document 1, means 41 for sequentially extracting omnidirectional frame images from an omnidirectional moving image obtained using an omnidirectional camera 20, means 42 for converting the sequentially extracted omnidirectional frame images into cube maps, and means 43 for object detection by deep learning for each element image of each direction of the cube map obtained; means 47 for storing in a record of the storage means 51 in association with the azimuth frame image; means 49 for restoring the cube map after object detection processing to the omnidirectional frame image and setting it at the original position in the omnidirectional moving image; An invention of an omnidirectional video processing device having

JP 2021-028813 A

According to the invention described in Patent Literature 1, it is possible to detect an object photographed in an omnidirectional image and set the position where the object is detected in an omnidirectional frame image. However, when an object is detected by converting a 360-degree omnidirectional image into a cube map image, the object located on the sides of the cube map cannot be detected appropriately.

Accordingly, an object of the present invention is to appropriately detect an object in a 360-degree omnidirectional image.

In order to solve the above-described problems, an omnidirectional image object detection apparatus according to the present invention includes an image conversion unit that converts an equirectangular image captured by an omnidirectional camera into a cubemap image; a first object detection unit that detects an object in a panoramic image of the plane of the; a first detection frame determination unit that determines coordinates of a rectangle surrounding the object detected by the first object detection unit in the panoramic image; a second object detection unit for detecting an object in a wrap around image obtained by shifting and wrapping a panoramic image out of the cube map image; a second detection frame determining unit for determining coordinates; and displaying the equirectangular image on a 360-degree display screen. and a synthesizing unit for synthesizing the rectangle with the corresponding position on the 360-degree display screen.

A method for detecting an object in an omnidirectional image according to the present invention includes the steps of converting an equirectangular image captured by an omnidirectional camera into a cubemap image, and detecting an object in panoramic images of a plurality of planes that constitute the cubemap image. determining the coordinates of a rectangle surrounding the object detected on the panoramic image; detecting the object in a wrapped image obtained by shifting and wrapping the panoramic image out of the cube map image; determining the coordinates of a rectangle surrounding the detected object on the plane of the image; displaying the equirectangular image on a 360-degree display screen and determining the rectangle in which the object is detected for the panoramic image and the object for the wraparound image; and synthesizing the detected rectangle with the corresponding position on the 360-degree display screen.

The object detection program of the present invention provides a computer with a procedure for converting an equirectangular image captured by an omnidirectional camera into a cube map image, a procedure for detecting an object in panoramic images of a plurality of planes that make up the cube map image, a procedure of determining the coordinates of a rectangle surrounding the object detected on the panoramic image; a procedure of detecting the object in a wrapped around image obtained by shifting and wrapping the panoramic image out of the cube map image; A procedure for determining the coordinates of a rectangle surrounding the detected object, displaying the equirectangular image on a 360 -degree display screen, determining the rectangle in which the object is detected in the panorama image and the rectangle in which the object is detected in the wrapping image. to the corresponding positions on the 360-degree display screen.
Other means are described in the detailed description.

According to the present invention, it is possible to appropriately detect an object in a 360-degree omnidirectional image.

1 is a configuration diagram of an object detection device according to an embodiment; FIG. It is a figure which shows an example of an equirectangular image. FIG. 4 is a diagram showing the coordinate system of an equirectangular image; FIG. 10 is a diagram showing an example of a method of displaying an equirectangular moving image; It is a figure which shows an example of a cube map image. FIG. 4 is a diagram showing a coordinate system of a cubemap image; FIG. 4 is a conceptual diagram showing a cube map image; FIG. 4 is a conceptual diagram of a panorama image of the horizontal periphery of a cube map image; FIG. 4 is a conceptual diagram of a panorama image obtained by horizontally shifting the horizontal periphery of a cube map image; FIG. 10 is an image for explaining preparation for three-dimensional coordinate transformation of an object detection frame; FIG. It is an image explaining the three-dimensional coordinate conversion by arrangement|positioning on the circumference of an object detection frame. It is an image explaining conversion of the direction of an object detection frame. It is a figure explaining a panorama image. It is a figure explaining a panorama image.

EMBODIMENT OF THE INVENTION Henceforth, the form for implementing this invention is demonstrated in detail with reference to each figure.
The present invention enables object detection not only in the current two-dimensional plane image, but also in three-dimensional space. According to the present invention, the position (three-dimensional coordinates) and orientation (three-dimensional angle) on the outer periphery of a 360-degree image can be acquired.

FIG. 1 is a configuration diagram of an object detection device 2 according to this embodiment. FIG. 1 shows a logical configuration from photographing by a 360-degree omnidirectional camera to display by an omnidirectional viewer.

The object detection device 2 includes a cube map image conversion unit 21, panorama acquisition units 22a and 22b, object detection units 23a and 23b, detection frame determination units 24a and 24b, a three-dimensional information synthesis unit 25, and a polygon generation unit. 26. The object detection device 2 is, for example, a computer including a CPU (Central Processing Unit), and implements each functional unit by executing an object detection program (not shown).

An equirectangular image 31 is input to the object detection device 2 as an omnidirectional image captured by the 360-degree omnidirectional camera 11 . The object detection device 2 detects an object captured in this equirectangular image 31 , generates a detection frame polygon, and synthesizes it on the 360-degree display screen 32 . The 360-degree display screen 32 is displayed on the display unit 13 after the viewpoint and the like are adjusted by an external controller.

The 360-degree omnidirectional camera 11 is a camera capable of photographing 360-degree omnidirectional images at once. The 360-degree omnidirectional camera 11 can shoot, for example, an 8K size equirectangular moving image.

FIG. 2 shows an equirectangular image 31 .
This equirectangular image 31 is obtained by mapping an image that should originally be projected onto a spherical surface onto a rectangle. The equirectangular projection, called the equirectangular projection, is used in panoramic photography to represent spherical panoramic images, as well as as a map projection. In the equirectangular image 31, latitudes when the spherical surface is likened to the earth are expressed as horizontal straight lines at predetermined intervals. The relationship between the pixel positions of the equirectangular image 31 and the corresponding pixel positions on the spherical surface is simple and easy to convert to other projections, but distortion occurs, especially at the poles.

The 360-degree omnidirectional camera 11 is a combination of two fisheye lenses. A circular image called a fisheye image is obtained from the 360-degree omnidirectional camera 11 . The 360-degree omnidirectional camera 11 converts the fisheye image into an equirectangular format and uses it for spatial rendering. This image format conversion is called stitching.

FIG. 3 is a diagram showing the coordinate system of the equirectangular image 31. As shown in FIG.
The equirectangular image 31 is originally mapped onto a spherical surface. A point P on the sphere may be represented by XYZ coordinates or by polar coordinates.

Returning to FIG. 1, the description continues. The cube map image conversion unit 21 of the object detection device 2 functions as an image conversion unit that converts the equirectangular image 31 captured by the 360-degree omnidirectional camera 11 into a cube map image. Cubemap images and their transformations are described in FIGS. 5 and 6 below.

Then, the panorama acquisition units 22a and 22b acquire a first panorama image and a second panorama image, respectively, using four consecutive cube map images. The panorama acquisition unit 22a functions as a first panorama acquisition unit that acquires first panorama images of a plurality of planes forming a cube map image. The first panorama image will be described later with reference to FIG.
The panorama acquisition unit 22b functions as a second panorama acquisition unit that acquires a wrapped image obtained by shifting and wrapping the panoramic image out of the cube map images. The second panorama image will be described later with reference to FIG.

The object detection units 23a and 23b detect objects from the first panorama image and the second panorama image, respectively. The object detection unit 23a functions as a first object detection unit that detects objects in the first panoramic image. The object detection unit 23b functions as a second object detection unit that detects objects in the second panoramic image.

The detection frame determination unit 24a functions as a first detection frame determination unit that determines the coordinates of the object detection frame on the first panorama image. The detection frame determination unit 24b functions as a second detection frame determination unit that determines the coordinates of the object detection frame on the second panoramic image. As a result, for example, it is possible to determine whether a person is detected at 145 degrees backward in the clockwise direction. Processing of these detection frame determination units 24a and 24b will be described later with reference to FIG.

The coordinates of the object detection frame are on a spherical surface, and the distance from the viewpoint cannot be detected. However, for a person, the distance from the viewpoint to the person is estimated from the size of the detected region and the average height of the person. Further, in the first embodiment, vertical object detection is not taken into consideration. In a second embodiment, which will be described later, it is possible to detect an object in the vertical direction.

The three-dimensional information synthesizing unit 25 converts the coordinates of the object detection frame on the first panoramic image into three-dimensional information, converts the coordinates of the object detection frame on the second panoramic image into three-dimensional information, and converts the coordinates of the object detection frame on the second panoramic image into three-dimensional information. This is a synthesizing unit that synthesizes coordinates. Processing of the three-dimensional information synthesizing unit 25 will be described later with reference to FIG.

The polygon generation unit 26 generates a polygon of an object detection frame at the coordinates synthesized by the three-dimensional information synthesis unit 25 and synthesizes it on the 360-degree display screen 32 . The processing of the polygon generator 26 will be described later with reference to FIG.

FIG. 4 is a diagram showing an example of a method of displaying the equirectangular image 31. As shown in FIG.
The display unit 13 is, for example, a head-mounted display. As shown in FIG. 4, the display unit 13 provides the user 5 with a 360-degree virtual reality (VR) space by pasting the above-described equirectangular image 31 on the omnidirectional sphere. The external controller 12 is, for example, various sensors provided in a head-mounted display. At this time, the image viewed by the user 5 is not distorted.

FIG. 5 is a diagram showing an example of the cube map image 33. As shown in FIG.
The cube map image 33 is a kind of omnidirectional video image format, and reproduces a space by attaching images of the top and bottom, the front, the left and right, and the back to a cube. The cubemap image 33 is obtained by transforming the equirectangular image 31 . The equirectangular image 31 can be projected onto the front, back, left and right, and top and bottom surfaces of the cube. By extracting the six planes on which the equirectangular image 31 is projected, it can be converted into a cube map image. This cube map image 33 is easy to use for object detection because each surface pixel is not curved.

Mathematically, when the radius of the sphere is r=1 and the polar coordinates θ and φ are 0<θ<π and -π/4<φ<7π/4, the following equation (1) is satisfied.

Consider projecting these onto a cube in the center.
First, divide into 4 regions at latitude -π/4<φ<π/4, π/4<φ<3π/4, 3π/4<φ<5π/4, 5π/4<φ<7π/4 . These are projected on one of the four sides, top or bottom.
Consider the side denoted by -π/4<φ<π/4.
The central projection of (sin θ cos φ, sin θ sin φ, cos θ) becomes (sin θ cos φ, sin θ sin φ, cos θ), which corresponds to the x=1 plane in the case of equation (2).

This can be transformed into equation (3).

The projection point is (1,tan φ,cotθ/cosφ).
If |cotθ/cosφ|<1, it is the front surface. Others are projected to the top or bottom, which requires another projection.
A better test at the top uses the fact that the minimum value of cosφ is cos(π/4)=1/√2. Therefore, cot θ/(1/√2)>1 or tan θ<1/√2. This corresponds to θ<35° or 0.615 radians.

FIG. 6 is a diagram showing the coordinate system of the cube map image 33. As shown in FIG.
At this time, conversion from the equirectangular form to the unit sphere is represented by the following equation (4).

Note that x _s , y _s , and z _s in Equation (4) indicate the coordinates of point P.
Transformation from the unit sphere to the equirectangular form is represented by the following equation (5).

<<1st Embodiment>>
In the first embodiment, an object is detected and a detection frame is provided only in the horizontal peripheral portion. In most cases, the desired detection object does not exist in the vertical direction, so there is no problem even under such a limitation.

FIG. 7 is an explanatory diagram showing each plane that constitutes the cube map image 33. As shown in FIG.
The cube map image 33 is composed of a front surface B, a left surface A, a right surface C, a rear surface D, an upper surface E, and a lower surface F. Since there is little distortion on each of these surfaces, the object can be detected.

FIG. 8 is a conceptual diagram of a panorama image 34 formed by each surface of the horizontal peripheral portion of the cube map image 33. As shown in FIG.
The panorama acquisition unit 22a collects the rear right side D2, the left side A, the front side B, the right side C, and the rear left side D1, which constitute the horizontal periphery of the cube map image 33, to obtain a panoramic image 34. FIG. The object detection unit 23 a performs object detection on this panoramic image 34 . For object detection, for example, an AI model SSD (Single Shot MultiBox Detector) or YOLO (You Only Look Once) or the like is used.

Since the original resolution is high, it is preferable to use a reduced image for the object detection unit 23a. The horizontal coordinates of the panorama image 34 correspond to angles of the entire circumference. Since the left and right edges of the panorama image 34 are cut off, the object detection unit 23a cannot detect an object hanging over the left and right edges.

FIG. 9 is a conceptual diagram of a panorama image 35 in which the horizontal peripheral portion of the cube map image 33 is shifted in the horizontal direction.
As for both ends of the panorama image 34, since the object is cut off, the object detection unit 23a may not be able to detect the object hanging over both ends. In order to solve this problem, in the present embodiment, a panorama image 35 whose entirety is horizontally shifted and wrapped around is prepared, and the object detection result of the panorama image 35 and the object detection result of the panorama image 34 are synthesized.

A panorama image 35 is an image in which the panorama image 34 is horizontally shifted and wrapped around. The panorama acquisition unit 22b collects the front right side B2, right side C, back side D, left side A, and front left side B1 to form a panorama image 35. FIG. The object detection unit 23b receives the panorama image 35 as an input and performs object detection on the two-dimensional image.

FIG. 10 is an image for explaining preparatory preparation for three-dimensional coordinate transformation of the object detection frame.
The object detection unit 23a detects each of the objects 51 to 53 photographed in the panoramic image 34 by two-dimensional object detection processing, and determines the coordinates of each of the detection frames 41 to 43. FIG. At this time, the object detection unit 23a can detect the two-dimensional coordinates of the panoramic image 34, and cannot detect the distance from the camera to the object.

The three-dimensional information synthesizing unit 25 converts the detection frame output by the two-dimensional object detection of the object detecting unit 23a into three-dimensional coordinates. The three-dimensional information synthesizing unit 25 captures the 360-degree display screen 32 as viewed from above, and grasps the relationship with the outer circumference.

11A and 11B are images for explaining three-dimensional coordinate transformation by arranging the object detection frame on the circumference.
The 360-degree display screen 32 is projected on the circumference. The position of the 360-degree display screen 32 corresponding to the left edge of the panoramic image 34 has an angle of -180 degrees with respect to the viewpoint. The position of the 360-degree display screen 32 corresponding to the right end of the panoramic image 34 has an angle of +180 degrees with respect to the viewpoint. The position of the 360-degree display screen 32 corresponding to the center of the panoramic image 34 has an angle of 0 degrees with respect to the viewpoint.

When the 360-degree display screen 32 is viewed from above, the detection frames 41 to 43 are arranged on the circumference of the 360-degree display screen 32 . Here, the three-dimensional information synthesizing unit 25 performs three-dimensional coordinate conversion. The X coordinate of the panorama image 34 corresponds to the outer peripheral angle of the 360-degree display screen 32 . Therefore, based on the X coordinates of the positions of the detection frames 41 to 43 of the panorama image 34, the outer peripheral angles of the detection frames 41 to 43 converted onto the 360-degree display screen 32 can be calculated.

FIG. 12 is an image for explaining the conversion of the directions of the detection frames 41 to 43 of the object.
The three-dimensional information synthesizing unit 25 calculates the orientations of the detection frames 41 to 43 based on the positions of the detection frames 41 to 43 in the circumferential projection. The directions of the detection frames 41 to 43 are the angles of the detection frames 41 to 43 in three-dimensional coordinates. Each of the detection frames 41 to 43 faces the origin (viewpoint) from the 360-degree display screen 32 . Therefore, the three-dimensional information synthesizing unit 25 calculates the orientation of each detection frame 41-43 from the position (peripheral angle) of each detection frame 41-43. The polygon generator 26 generates polygons of the detection frames 41 to 43 from the positions (peripheral angles) and orientations of the detection frames 41 to 43 and synthesizes them with the 360-degree display screen 32 .

<<Second embodiment>>
2nd Embodiment detects an object over a perimeter, and gives a detection frame. Description will be made below with reference to FIGS. 13 and 14. FIG.

FIG. 13 is a diagram showing panoramic images 35 and 36. As shown in FIG.
The panorama image 35 is a collection of the back upper side D3, the top side E, the front side B, the bottom side F, and the back side lower side D4 of the cube map image 33 . Objects are detected in this panorama image 35 . The Y coordinate of this panorama image 35 corresponds to the latitude on the predetermined longitude plane of the 360-degree display screen 32 . Thereby, the positions and orientations of the detection frames 41 to 43 on the 360-degree display screen 32 can be known.

However, with respect to the upper and lower ends of the panorama image 35, since the target object is cut off, there is a possibility that the object hanging over both ends cannot be detected. In order to deal with this problem, this embodiment prepares a panorama image 36 whose entirety is vertically shifted and wrapped around, and synthesizes the object detection result from the panorama image 36 and the object detection result from the panorama image 35 .

The panorama image 36 is an image in which the panorama image 35 is vertically shifted and wrapped. The panorama image 36 is configured by collecting the front lower side B3, the lower side F, the rear side D, the upper side E, and the front upper side B4. With this panorama image 36 as an input, object detection of a two-dimensional image is performed. Thereby, in this embodiment, the object can be detected without omission over the entire circumference.

FIG. 14 is a diagram showing panoramic images 37 and 38. As shown in FIG.
The panorama image 37 is a collection of the bottom surface F, right surface C, top surface E, and left surface A of the cube map image 33 . Objects are detected in this panorama image 37 . The Y coordinate of this panorama image 37 corresponds to the latitude on the predetermined longitude plane of the 360-degree display screen 32 . Thereby, the positions and orientations of the detection frames 41 to 43 on the 360-degree display screen 32 can be known.

However, with respect to the upper and lower ends of the panorama image 37, since the target object is cut off, there is a possibility that the object hanging over both ends cannot be detected. In order to deal with this problem, the present embodiment prepares a panorama image 38 whose entirety is vertically shifted and wrapped around, and synthesizes the object detection result from the panorama image 38 and the object detection result from the panorama image 37 .

The panorama image 38 is an image in which the panorama image 37 is vertically shifted and wrapped. The panorama image 38 is configured by collecting the upper surface E, the left surface A, the lower surface F, and the right surface C. As shown in FIG. With this panorama image 38 as an input, object detection of a two-dimensional image is performed. Thereby, in this embodiment, the object can be detected without omission over the entire circumference.

(Modification)
The present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. A part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Moreover, it is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

Some or all of the above configurations, functions, processing units, processing means, etc. may be realized by hardware such as integrated circuits. Each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function. Information such as programs, tables, and files that implement each function can be stored in recording devices such as memory, hard disks, SSDs (Solid State Drives), or recording media such as flash memory cards and DVDs (Digital Versatile Disks). can.

In each embodiment, control lines and information lines indicate those considered necessary for explanation, and not all control lines and information lines are necessarily indicated on the product. In fact, it may be considered that almost all configurations are interconnected.
Modifications of the present invention include, for example, the following (a) to (c).

(a) In addition to associating the position of the long axis of the panoramic image with the angle of the 360-degree display screen, the position of the short axis of the panoramic image may be associated with the angle of the 360-degree display screen.
(b) The equirectangular image may be converted into a two-dimensional panorama image by image conversion to an arbitrary polygon, not limited to cube map image conversion.
(c) The method of object detection processing is not limited to SSD and YOLO, and any method of object detection processing may be adopted.

11 360-degree omnidirectional camera 12 External controller 13 Display unit 2 Object detection device 21 Cube map image conversion unit 22a Panorama acquisition unit 22b Panorama acquisition unit 23a Object detection unit (first object detection unit)
23b object detection unit (second object detection unit)
24a detection frame determination unit (first detection frame determination unit)
24b detection frame determination unit (second detection frame determination unit)
25 Three-dimensional information synthesizing unit (synthesizing unit)
26 Polygon generator 31 Equirectangular image 32 360-degree display screen 33 Cube map images 34-38 Panoramic images 41-43 Detection frame 5 User 51-53 Object

Claims (6)

an image conversion unit that converts an equirectangular image captured by an omnidirectional camera into a cube map image;
a first object detection unit that detects an object in a panoramic image of a plurality of planes that constitute the cube map image;
a first detection frame determination unit that determines coordinates of a rectangle surrounding the object detected by the first object detection unit in the panoramic image;
a second object detection unit that detects an object in a wrapped image obtained by shifting and wrapping the panoramic image of the cube map image;
a second detection frame determination unit that determines coordinates of a rectangle surrounding the object detected by the second object detection unit on the surface of the wrapping image;
The equirectangular image is displayed on a 360-degree display screen, and the rectangle determined by the first detection frame determination unit and the rectangle determined by the second detection frame determination unit are displayed at corresponding positions on the 360 -degree display screen. a synthesizing unit that synthesizes into
An omnidirectional image object detection device comprising: wherein the synthesis unit corrects the orientation of the rectangle based on the position of the rectangle in the circumferential projection;
2. The omnidirectional image object detection apparatus according to claim 1, wherein: wherein the panoramic image is a horizontal perimeter of the cubemap image,
2. The omnidirectional image object detection apparatus according to claim 1, wherein: The panoramic image is
a horizontal periphery of the cube map image;
a combination of the front surface, the top surface, the back surface, and the bottom surface of the cube map image;
a combination of the left surface, the top surface, the right surface, and the bottom surface of the cube map image;
2. The omnidirectional image object detection apparatus according to claim 1, comprising: converting an equirectangular image captured by an omnidirectional camera into a cubemap image;
a step of detecting an object in a panoramic image of a plurality of planes that make up the cubemap image;
determining the coordinates of a rectangle enclosing the detected object on the panoramic image;
a step of detecting an object in a wraparound image obtained by shifting and wrapping a panoramic image out of the cubemap images;
determining the coordinates of a rectangle enclosing the detected object on the plane of the wrap image;
displaying the equirectangular image on a 360-degree display screen, and synthesizing the rectangle in which the object is detected in the panorama image and the rectangle in which the object is detected in the wraparound image at corresponding positions on the 360-degree display screen; and,
An object detection method for an omnidirectional image, comprising: to the computer,
The procedure for converting an equirectangular image taken with an omnidirectional camera into a cubemap image,
a procedure for detecting an object in a panoramic image of a plurality of planes that constitute the cubemap image;
determining the coordinates of a rectangle surrounding the object detected on the panoramic image;
a step of detecting an object in a wrapped image obtained by shifting and wrapping the panoramic image of the cube map image;
determining the coordinates of a rectangle enclosing the detected object on the plane of the wrap image;
A procedure of displaying the equirectangular image on a 360-degree display screen, and synthesizing the rectangle in which the object is detected in the panoramic image and the rectangle in which the object is detected in the wraparound image at corresponding positions on the 360-degree display screen. ,
Object detection program for running.

Images