KR102310958B1 - Wide viewing angle stereo camera apparatus and depth image processing method using the same - Google Patents

Abstract

Disclosed are a stereo camera device having a wide viewing angle and a depth image processing method using the same. According to one embodiment of the present invention, the stereo camera device includes: a reception unit receiving a first image and a second image of a subject photographed through first and second lenses arranged in a vertical direction; a conversion unit converting the received first and second images into a map projection method; and a processing unit extracting the depth of the subject by performing stereo matching on the first image and the second image converted by the map projection method in the vertical direction.

Description

Wide viewing angle stereo camera device and depth image processing method using the same {WIDE VIEWING ANGLE STEREO CAMERA APPARATUS AND DEPTH IMAGE PROCESSING METHOD USING THE SAME}

The present invention relates to a stereo camera device having a wide viewing angle and a depth image processing method using the same, and more particularly, by arranging two lenses, for example, a fisheye lens in a vertical direction, to improve a horizontal viewing angle and a vertical viewing angle, thereby providing a wide viewing angle. To a stereo camera device having a and a depth image processing method using the same.

In order to implement a human-computer interface, a lot of research has been done on an RGB-D camera that outputs a color image and a depth map that is 3D information together. The Kinect sensor, an RGB-D camera that Microsoft supplied for its games, provided a function to recognize multiple human gestures and was used for various human-computer interfaces as well as games, which was its original purpose. -D camera products have been released and are expanding the scope of use to 3D content production, autonomous driving, and robots.

The RGB-D camera can be divided into a stereo camera and a Time of Flight (TOF) camera according to a method of obtaining depth information. A typical stereo camera consists of two cameras installed as far apart as the baseline, and the depth information is obtained by using the mismatch of viewpoints between the cameras. In order to obtain depth information, it is necessary to go through a stereo matching process that compares images from both cameras, and pattern light or structured light is sometimes used to increase the speed and accuracy of the stereo matching process. A TOF camera is a combination of a general camera that outputs 2D images and a TOF sensor that provides depth information. The TOF sensor measures the distance to the object by measuring the return time after irradiating a laser, etc. There are also restrictions on the measurement distance and the usage environment.

Common cameras use lenses designed for rectilinear projection. In linear projection, a straight line in the real world appears as a straight line in the image, so an image similar to that perceived by humans can be obtained. there is a problem. Usually, a straight projection lens is used up to a horizontal viewing angle of about 120 degrees, and a fisheye lens is used when a viewing angle greater than that is required. The fisheye lens projects the real world as if there is Barrel Distortion on the image, and it can be captured in the image with a viewing angle of more than 180 degrees.

As the use environment of RGB-D cameras is expanded to moving objects such as robots, drones, and autonomous vehicles, the demand for a wide viewing angle is increasing. could not provide Although some stereo cameras use a fisheye lens to provide a wider viewing angle, there is a disadvantage in that efficient stereo matching through 1D search like in a linear projection image is difficult in a fisheye lens image. Also, even if a fisheye lens is used, distance information cannot be obtained from the same direction as the baseline of the stereo camera, so the horizontal angle of view is limited to within 180 degrees.

Embodiments of the present invention provide a stereo camera device having a wide viewing angle in which two lenses, for example, a fisheye lens, are disposed in a vertical direction to improve a horizontal viewing angle and a vertical viewing angle to 200 degrees horizontally and 130 degrees vertically or more, which are human viewing angles. and a depth image processing method using the same.

A stereo camera apparatus according to an embodiment of the present invention includes: a receiver configured to receive a first image and a second image of a subject photographed through a first lens and a second lens arranged in a vertical direction; a conversion unit converting the received first and second images into a map projection method; and a processing unit extracting the depth of the subject by performing stereo matching on the first image and the second image converted by the map projection method in the vertical direction.

The converter may convert the first image and the second image into an equirectangular projection.

The transform unit may perform rotational transformation to match an epipolar line with a spherical longitude line using a spherical coordinate system.

The processor may perform stereo matching by searching for the first image and the second image converted by the map projection method along a vertical line corresponding to the longitude line.

The first lens and the second lens may include a fisheye lens.

The first lens and the second lens may be fisheye lenses having a horizontal viewing angle of 180 degrees or more.

The processing unit may obtain a depth image of the subject having a wide viewing angle of a horizontal viewing angle of 180 degrees or more and a vertical viewing angle of 180 degrees or more by using the extracted depth of the object.

The receiver includes an image sensor for photographing the first image and an image sensor for photographing the second image, and each of the image sensors for photographing the first image and the second image has a horizontal length and a vertical length. In the case of a rectangular shape longer than the length, it may be arranged in a horizontal direction to obtain a horizontal viewing angle wider than a vertical viewing angle.

In the depth image processing method according to an embodiment of the present invention, in the image processing method of a stereo camera device, a first image and a second image of a subject photographed through a first lens and a second lens arranged in a vertical direction are received. to do; converting the received first and second images into a map projection method; and extracting the depth of the subject by performing stereo matching on the first image and the second image converted by the map projection method in the vertical direction.

The converting may include converting the first image and the second image into an equirectangular projection.

In the transforming step, rotational transformation in which an epipolar line coincides with a spherical longitude line may be performed using a spherical coordinate system.

In the extracting, stereo matching may be performed by searching for the first image and the second image converted by the map projection method along a vertical line corresponding to the longitude line.

The first lens and the second lens may include a fisheye lens.

The first lens and the second lens may be fisheye lenses having a horizontal viewing angle of 180 degrees or more.

Furthermore, the image processing method according to an embodiment of the present invention includes the steps of acquiring a depth image of the subject having a wide viewing angle of 180 degrees or more and a vertical viewing angle of 180 degrees or more by using the extracted depth of the subject. may include more.

The receiving includes receiving the first image and the second image having a horizontal viewing angle wider than a vertical viewing angle using an image sensor for photographing the first image and an image sensor for photographing the second image, Each of the image sensors for photographing each of the first image and the second image is

It may have a rectangular shape in which the horizontal length is longer than the vertical length.

According to embodiments of the present invention, by arranging two lenses, for example, a fisheye lens in a vertical direction, to improve a horizontal viewing angle and a vertical viewing angle, a wide viewing angle may be obtained.

According to embodiments of the present invention, an RGB-D (depth) camera wider than a human's viewing angle is implemented by improving the horizontal and vertical viewing angles to have a wide viewing angle, and through this, it is possible to easily detect a nearby object.

The present invention provides a wider viewing angle than a person, so that the RGB-D camera can be mounted on a moving object such as a robot, vehicle, or drone, and it is possible to detect a person who is close within 1 m, so it is attached to a Human Following Robot. In this case, the whole body can be detected even when the following target is close, and stable tracking is possible because the tracking target does not deviate from the field of view even when the moving direction is changed. Accordingly, the present invention can be utilized not only in the human-computer interface, but also in the human-robot interface and various moving objects.

1 shows the configuration of a stereo camera device according to an embodiment of the present invention.
2 is a view illustrating an example for explaining the horizontal arrangement and viewing angle of each image sensor to obtain a horizontal viewing angle wider than the placement of the fisheye lens and the vertical viewing angle in the stereo camera device of the present invention.
3 is a view showing an exemplary diagram for explaining the geometry of the isopolar line in the stereo camera device of the present invention.
4 is a diagram illustrating an example for explaining a method of extracting depth.
5 shows an exemplary diagram for explaining the conversion of a fisheye lens image to an equirectangular projection method.
6 is a diagram illustrating an example of a depth for a disparity of the same size.
7 is a view showing an exemplary view for explaining the viewing angle of the stereo camera device of the present invention.
8 is a flowchart illustrating an image processing method of a stereo camera apparatus according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments, and is not intended to limit the present invention. As used herein, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, "comprises" and/or "comprising" refers to the presence of one or more other components, steps, operations and/or elements mentioned. or addition is not excluded.

Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used with the meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. The same reference numerals are used for the same components in the drawings, and repeated descriptions of the same components are omitted.

SUMMARY Embodiments of the present invention provide a stereo camera device capable of having a wide viewing angle by improving a horizontal viewing angle and a vertical viewing angle by arranging two lenses, for example, a fisheye lens in a vertical direction.

In this case, the present invention converts fisheye images photographed through each of the two fisheye lenses into a map projection method, for example, an equirectangular projection, and then converts the fisheye images converted to an equirectangular projection in a vertical or vertical direction. By performing stereo matching with , it is possible to extract the depth of the subject photographed by the fisheye images, and obtain an image of a wide viewing angle using the extracted depth.

In the stereo camera device of the present invention, since two fisheye lenses or fisheye lens cameras are vertically arranged to face the same direction, if a spherical coordinate system is used, the epipolar line coincides with the spherical longitude line (meridian), and the fisheye When the lens image is converted into an equirectangular projection method, the longitude line is expressed as a vertical line in the image, so efficient stereo matching can be performed by searching along the vertical line.

The stereo camera apparatus of the present invention extracts or calculates the depth of the subject from the angular disparity between the images converted to the equirectangular projection with respect to the images taken through the upper fisheye lens and the lower fisheye lens. can

The present invention will be described with reference to FIGS. 1 to 8 as follows.

1 shows a configuration of a stereo camera device according to an embodiment of the present invention, and shows a conceptual configuration of a stereo camera device or system.

Referring to FIG. 1 , a stereo camera apparatus 100 according to an embodiment of the present invention includes a receiver 110 , a converter 120 , and a processor 130 .

The receiver 110 receives a first lens disposed in a vertical direction, for example, a first fisheye lens (or a first fisheye lens camera) and a second lens, for example, a second fisheye lens (or a second fisheye lens camera). A first image (or a first fisheye image) and a second image (or a second fisheye image) of the subject to be photographed are received.

For example, the receiver 110 receives the first fisheye image and the second fisheye image of the subject photographed through the first fisheye lens and the second fisheye lens arranged in the vertical direction as shown in FIG. 2 . Here, the first fisheye lens and the second fisheye lens are vertically disposed to face the same direction, and may have a horizontal viewing angle of 200 degrees or more and a vertical viewing angle of 180 degrees or more. Of course, each of the fisheye lenses is not limited or limited to a horizontal viewing angle of 200 degrees or more and a vertical viewing angle of 180 degrees or more, and the horizontal and vertical viewing angles may vary depending on the fisheye lens used.

Furthermore, the receiving unit 110 may include a first image sensor of a first camera for capturing a first image and a second image sensor of a second camera for capturing a second image in a rectangular shape in which a horizontal length is longer than a vertical length. Each image sensor may be arranged in a horizontal direction to obtain a horizontal viewing angle (or horizontal viewing angle) that is wider than the viewing angle (or vertical viewing angle).

The converter 120 converts the first fisheye image and the second fisheye image received through the receiver 110 into a map projection method.

In this case, the converter 120 may convert the first fisheye image and the second fisheye image into an equirectangular projection.

Here, the equirectangular projection method is one of the map projection methods, and the latitude and longitude lines may be expressed as horizontal and vertical lines, respectively. For example, as shown in FIG. 3 , the conversion unit may convert a 220 degree fisheye image into an equirectangular image by converting it into an equirectangular image. This equirectangular projection method can be used to capture a wide viewing angle that cannot be included in a conventional Rectilinear Projection image, such as a 360-degree camera. In addition, when a fisheye lens image having a viewing angle of 180 degrees or more is converted through an equirectangular projection method, distortion may occur in which the upper and lower portions of the image are greatly stretched to the left and right.

Furthermore, the converting unit 120 rotates the first fisheye image and the second fisheye image before converting the first fisheye image and the second fisheye image into an equirectangular projection to match the direction of the meridian (image rectification) can also be performed. When the images are aligned, the comparison between the two images is simplified to a one-dimensional search, and the speed of stereo matching can be increased.

The processing unit 130 performs stereo matching on the fisheye images converted to the equirectangular projection by the transformation unit 120 , that is, the first equirectangular image and the second equirectangular image in the vertical or vertical direction. , extracts the depth or depth map of the subject photographed through the first fisheye lens and the second fisheye lens.

At this time, as shown in FIG. 4 , since the processing unit 130 is vertically disposed so that two fisheye lenses or fisheye lens cameras face the same direction, when using a spherical coordinate system, an epipolar line is a spherical longitude line. (meridian), and converting the fisheye lens image into an equirectangular projection, the longitude line is expressed as a vertical line in the image, so efficient stereo matching can be performed by searching along the vertical line.

₁-

₀로부터 피사체에 대한 깊이(d)를 추출 또는 산출할 수 있으며, 깊이는 아래 <수학식 1>에 의해 추출 또는 산출될 수 있다.Then, as shown in FIG. 5 , the processing unit 130 determines the angular disparity between the images converted to the equirectangular projection with respect to the images photographed through the upper fisheye lens and the lower fisheye lens, that is,

₁ -

_The depth d for the subject may be extracted or calculated from 0, and the depth may be extracted or calculated by the following <Equation 1>.

[Equation 1]

₁에 따라 달라지는 것을 알 수 있다. 또한, Baseline B와 같은 방향인 위쪽과 아래쪽 예를 들어,

₁이 -90도 또는 90도에서는 깊이가 구해지지 않는 것을 확인할 수 있다.Here, even if each disparity is the same, the depth is

It can be seen that _{1 depends on} Also, upward and downward in the same direction as Baseline B. For example,

₁ It can be seen that the depth is not obtained at -90 degrees or 90 degrees.

At this time, when the direction in which the camera looks is the Z-axis and the direction of the baseline is the Y-axis, the depth of each disparity having the same size may be different as shown in FIG. 6 .

In the conventional stereo with a limited viewing angle, the depth (or distance) and the size of the disparity were calculated by approximating that they are in inverse proportion to each other. Therefore, this must be taken into account.

Furthermore, the processor 130 may acquire an image of the subject having a horizontal viewing angle of 180 degrees or more and a wide viewing angle of 180 degrees or more by using the extracted depth of the object.

As described above, the stereo camera apparatus according to an embodiment of the present invention may have a wide viewing angle by arranging two lenses, for example, a fisheye lens, in a vertical direction to improve a horizontal viewing angle and a vertical viewing angle. For example, in the present invention, a wide viewing angle RGB-D camera device with a vertical viewing angle of up to 180 degrees and a horizontal viewing angle of much greater than 180 degrees can be implemented. Here, when a 280 degree fisheye lens is used, a horizontal viewing angle of 280 degrees and a vertical viewing angle of 180 degrees are possible.

In addition, according to embodiments of the present invention, the stereo camera device according to an embodiment of the present invention has a wide viewing angle by improving the horizontal and vertical viewing angles, thereby implementing an RGB-D (depth) camera wider than a human viewing angle. and, through this, it is possible to easily detect a nearby object. For example, as shown in FIG. 7 , in the stereo camera apparatus according to an embodiment of the present invention, since it is easy to detect a nearby object, it is possible to detect a person who is close to the body within 1 m.

In addition, when the stereo camera device according to an embodiment of the present invention is attached to a tracking robot, it is possible to detect the whole body even when the following target is close, and stable tracking is possible because the following target does not deviate from the field of view even when the moving direction is changed. Accordingly, the present invention can be utilized not only in the human-computer interface, but also in the human-robot interface and various moving objects.

As described above, the stereo camera device of the present invention converts an image photographed by arranging / photographing two fisheye lenses or fisheye lens cameras in a vertical direction or a vertical direction into an Equirectangular Projection, so that the upper and lower images are efficiently Stereo matching is possible, which takes advantage of the vertical arrangement in consideration of the following three contents.

First, in fisheye stereo with a field of view greater than 180 degrees, occlusion occurs even between the two cameras. For example, in the video the side camera looks. Therefore, the cameras should be positioned so that occlusion can be minimized.

Second, when using the Equirectangular Projection, there is a problem in that the upper and lower regions of the image corresponding to the pole are expressed in an excessively enlarged manner than in reality.

Third, in the existing stereo with limited viewing angle, the depth was calculated by approximating that the size of the depth is inversely proportional to the disparity. becomes smaller, making it difficult to obtain an accurate depth.

In addition, the stereo camera device of the present invention minimizes the area where depth cannot be obtained by matching the direction in which the interference occurs between two cameras or two lenses and the direction in which the depth cannot be obtained, and increases the viewing angle of the horizontal depth map by 180 degrees or more. can Of course, the stereo camera apparatus of the present invention may obtain a depth map of 280 degrees with respect to the horizontal direction when using a 280 degree fisheye lens.

Furthermore, in the stereo camera apparatus according to an embodiment of the present invention, when stereo matching is performed by searching for an upper camera image based on an image of a lower camera among the upper camera and the lower camera, a hole due to occlusion is formed. It occurs in the downward direction of the object, and the present invention may be more advantageous in the case of object detection, etc. because it occurs downward of the object even if a hole is created.

8 is a flowchart illustrating an image processing method of a stereo camera apparatus according to an embodiment of the present invention, and is a flowchart illustrating an operation in the stereo camera apparatus of FIGS. 1 to 7 .

Referring to FIG. 8 , in the image processing method of the stereo camera apparatus according to an embodiment of the present invention, a first image of a subject, for example, a first fisheye image, is photographed through a first lens and a second lens arranged in a vertical direction. and a second image, for example, a second fisheye image is received (S810).

When the first fisheye image and the second fisheye image are received in step S810, the received first fisheye image and the second fisheye image are converted into a map projection method (S820).

In this case, step S820 may convert the first fisheye image and the second fisheye image into an equirectangular projection. Also, in step S820, an epipolar line may be converted to coincide with the longitude line of the spherical surface using the spherical coordinate system.

When the first fisheye image and the second fisheye image are converted into the map projection method by step S820, the depth of the subject is extracted by performing stereo matching on the first fisheye image and the second fisheye image converted by the map projection method in the vertical direction, and , a depth image of a subject having a horizontal viewing angle of 180 degrees or more and a wide viewing angle of 180 degrees or more vertical viewing angles is obtained using the extracted depth of the subject (S830 and S840).

In this case, in step S830, stereo matching may be performed by searching for the first image and the second image converted by the map projection method along a vertical line corresponding to the longitude line.

Although the description of the method of FIG. 8 is omitted, each step constituting FIG. 8 may include all the contents described with reference to FIGS. 1 to 7 , which is apparent to those skilled in the art.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. may be embodied in The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible for those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (16)

a receiver configured to receive a first image and a second image of a subject photographed through the first and second lenses arranged in a vertical direction;
a conversion unit converting the received first and second images into a map projection method; and
A processing unit that extracts the depth of the subject by performing stereo matching on the first image and the second image converted by the map projection method in the vertical direction
including,
the conversion unit
The first image and the second image are converted into an Equirectangular Projection, and rotational transformation is performed to match the Epipolar line with the longitude line of the spherical surface using a spherical coordinate system,
the processing unit
In the first image and the second image converted by the map projection method, the longitude line is expressed as a vertical line, and the first image and the second image converted by the map projection method are searched along the vertical line corresponding to the longitude line to perform stereo matching, ,
the processing unit
A stereo camera device for acquiring an image of the subject having a horizontal viewing angle of 180 degrees or more and a wide viewing angle of a maximum vertical viewing angle of 180 degrees using the extracted depth of the subject.
delete delete delete According to claim 1,
The first lens and the second lens
A stereo camera device comprising a fisheye lens.
6. The method of claim 5,
The first lens and the second lens
A stereo camera device, characterized in that it is a fisheye lens with a horizontal viewing angle of 180 degrees or more.
delete According to claim 1,
the receiving unit
An image sensor for photographing the first image and an image sensor for photographing the second image,
Each of the image sensors for photographing each of the first image and the second image is
A stereo camera device, characterized in that it is arranged in a horizontal direction to obtain a horizontal viewing angle wider than a vertical viewing angle when the horizontal length is longer than the vertical length in a rectangular shape.
In the image processing method of a stereo camera device,
receiving a first image and a second image of a subject photographed through a first lens and a second lens arranged in a vertical direction;
converting the received first and second images into a map projection method; and
extracting the depth of the subject by performing stereo matching on the first image and the second image converted by the map projection method in the vertical direction
including,
The converting step is
The first image and the second image are converted into an equirectangular projection, and rotational transformation is performed to match the epipolar line with the longitude line of the spherical surface using a spherical coordinate system,
The extraction step
In the first image and the second image converted by the map projection method, the longitude line is expressed as a vertical line, and the first image and the second image converted by the map projection method are searched along the vertical line corresponding to the longitude line to perform stereo matching, ,
obtaining a depth image of the subject having a horizontal viewing angle of 180 degrees or more and a wide viewing angle of a maximum vertical viewing angle of 180 degrees using the extracted depth of the subject
A depth image processing method further comprising a.
delete delete delete 10. The method of claim 9,
The first lens and the second lens
A depth image processing method comprising a fisheye lens.
14. The method of claim 13,
The first lens and the second lens
A depth image processing method, characterized in that the horizontal viewing angle is a fisheye lens of 180 degrees or more.
delete 10. The method of claim 9,
The receiving step
receiving the first image and the second image having a horizontal viewing angle wider than a vertical viewing angle using an image sensor for photographing the first image and an image sensor for photographing the second image;
Each of the image sensors for photographing each of the first image and the second image is
A depth image processing method, characterized in that it has a rectangular shape in which a horizontal length is longer than a vertical length.

Images