TWM594322U - Camera configuration system with omnidirectional stereo vision - Google Patents

Abstract

An omnidirectional stereo vision camera configuration system. The camera configuration system mainly includes: at least four main camera lenses, which are respectively arranged on four reference lines that can form a rectangle; at least four auxiliary camera lenses, each main camera lens and each sub camera The lenses can be arranged as rectangles at four intervals along the same plane; the key to this creation is that the optical axis of each secondary camera lens or each main camera lens is configured to rotate in two parts to make the arrangement The optical axes of the camera lenses at the same reference line are parallel to each other, and the optical axes of the camera lenses arranged at the same reference line are perpendicular to the reference line to complete the camera calibration; thereby, at least two corresponding to each reference line can be made After performing the image fusion calculation on the depth map, the occlusion area is eliminated, and then the omnidirectional depth map without occlusion can be obtained through image stitching.

Description

Omnidirectional stereo vision camera configuration system

This creation involves Stereo Vision (Stereo Vision) technology, especially one that can arrange a plurality of camera lenses at intervals and arrange the lens centers of at least three camera lenses to be corrected on the same reference line, so that Obtaining an unobstructed depth map, and then an omnidirectional depth map (omnidirectional depth map) camera configuration system that can obtain an omnidirectional depth map after image stitching is completed.

In binocular vision, the camera needs to be calibrated and corrected, and the purpose of camera rectification is to achieve an ideal binocular camera system, so that the optical axes of at least two cameras are completely parallel (that is, the mirror centers are only There is an X component), and it is perpendicular to the baseline (the line connecting the lens centers of the left and right cameras is the baseline) in order to continue the depth calculation and 3D reconstruction.

Generally speaking, the lens centers (optical centers) of the left and right cameras before correction are not parallel, and the optical axes of the left and right cameras of the ideal binocular system after correction are parallel to each other, the optical axis and the image plane are perpendicular, and the imaging point is on the left and right images The heights are the same. Therefore, when performing stereo matching in the future, you only need to search the matching points of the left and right images in the search range of the same column, which can greatly improve the camera calibration efficiency.

At present, although it has been proposed to use 3D sensing camera (Stereoscopic Camera) to obtain omnidirectional depth information, for example, the Republic of China Invention Patent No. TW201832547 "Image Device for Generating Panoramic Depth Image, Related Methods and Related Image Devices" (hereinafter referred to as Taiwan) Case), the Taiwan case mainly arranges four ultra-wide-angle fisheye lenses (>190 degrees) back-to-back and up and down, and projects the captured wide-angle image to the equirectangular coordinate system, which is located on the same side A fisheye lens can be used to calculate a 180x180 degree stereo depth image, and finally the two side depth images are stitched together to obtain a 360x180 degree omnidirectional depth image. However, this type of wide angle image is projected to The equidistant cylindrical projection method of the latitude and longitude coordinates (Equirectangular Projection), the image resolution near 180 degrees is quite poor (due to the use of fisheye lenses will cause lens distortion), at the same time, it will also be caused by the use of stereo vision technology Occlusion problem, which will directly affect the accuracy of depth estimation.

Although others have proposed that N cameras can be arranged into a regular N polygon, and depth images are generated by two or two cameras, and finally panoramic stitching is completed to obtain omnidirectional depth images, such as US Patent No. US10244226B2 "Camera rig and stereoscopic image capture It is revealed (hereinafter referred to as the US case) that, although this method can produce a high-resolution depth image, the location of each camera in the US case will not be able to deal with the aforementioned occlusion problem, resulting in the depth image it produces Defective.

，亦可稱平移矩陣)，達到讓X軸與基線b12的向量平行，但由於第二相機偏離X軸太遠，導致偏移矩陣T的y、z分量較大，而當y、z分量達到一定程度(例如若高於1mm)，將造成後續計算深度圖的誤差過大，進而影響到後續將相機座標轉換至世界座標的準確性，且即便有其它方式(例如調整第二相機的焦距)可讓第二相機的鏡心位置逼近於X軸(理想位置)，但效果仍為有限，且屆時在計算深度圖時也會因此增加可觀的運算量，換言之，由於美國案的第一至第三相機的鏡心並非排列於同一基線(尤其像第二相機偏離X軸太遠)，故在無法取得第二相機所拍攝場景物體的XYZ資訊(相機座標)前，將無法計算出無遮擋視差圖。 According to the above, in order to obtain a de-occlusion depth map in the field of binocular vision, multi-lens (at least three) camera correction is necessary, and please refer to "Picture 1". The baselines b12 and b13 shown are the connection of the lens centers O of the first and second cameras before correction in the US case and the first and third cameras before correction in the US case, and the baselines b12' and b13' are the first The connection between the first camera and the second camera, and the center O of the first camera and the third camera after calibration, the baseline b13 is used as the reference line during the calibration, so the first and third cameras only need to rotate to make X The axis is parallel to the vector of the baseline b13, although the second camera can also pass through the offset matrix T(

, Can also be called a translation matrix), so that the X axis is parallel to the vector of the baseline b12, but because the second camera deviates too far from the X axis, the y and z components of the offset matrix T are large, and when the y and z components reach To a certain extent (for example, if it is higher than 1mm), it will cause excessive errors in the subsequent calculation of the depth map, which will affect the accuracy of the subsequent conversion of the camera coordinates to the world coordinates, and even if there are other ways (such as adjusting the focal length of the second camera) The position of the lens center of the second camera is approximated to the X axis (ideal position), but the effect is still limited, and the amount of calculation will also be increased when calculating the depth map. In other words, due to the first to third in the US case The lens centers of the cameras are not arranged on the same baseline (especially if the second camera is too far away from the X axis), so the unobstructed parallax map cannot be calculated before the XYZ information (camera coordinates) of the scene objects shot by the second camera cannot be obtained .

Accordingly, how to propose an omnidirectional stereo vision camera configuration system that can obtain an unobstructed parallax map without using a fisheye lens is a problem to be solved.

To achieve the above purpose, this author proposes an omnidirectional stereo vision camera configuration system. The camera configuration system mainly includes: a main camera group, an auxiliary camera group, and an arithmetic unit; wherein, the main camera group may include at least four main cameras Lenses, each main camera lens can be arranged in four reference lines that can form a rectangle; the auxiliary camera group can include at least four auxiliary camera lenses, and each main camera lens and each auxiliary camera lens can be along four reference lines Arranged in the same plane at a rectangular interval; the arithmetic unit can be connected to each main camera lens and each sub camera lens in information; the optical axis of each sub camera lens or each main camera lens is configured to be rotatable, so that the arrangement is based on the reference The optical axes of the auxiliary camera lens and the main camera lens of the line are parallel to each other, and the optical axes of the auxiliary camera lens and the main camera lens arranged on the reference line are perpendicular to the corresponding reference line to complete the camera calibration; they are on the same reference line and For the camera lenses that have completed camera calibration, the multiple images captured by the computing unit can be used by the computing unit to calculate at least two depth maps, and after the computing unit performs image fusion calculation on each depth map, it can eliminate the depth maps between each other. The occlusion area generates a de-occlusion depth map; finally, the arithmetic unit may perform image stitching on each de-occlusion depth map calculated by each reference line to obtain an omnidirectional depth map.

In this way, after the implementation of this creation, compared with the existing method of obtaining omnidirectional depth maps, this creation can at least achieve the beneficial effect of obtaining unobstructed omnidirectional depth, and even if the number of cameras used is four The above can still maintain the rectangular arrangement of these cameras without arranging them as polygons, which increases the configuration cost of the camera system.

In order to enable your reviewing committee to clearly understand the purpose, technical features and effects of this creation, the following description is accompanied by illustrations, please refer to it.

Please refer to "Figure 2", which is the architecture diagram of the camera configuration system created by the author. This creation proposes an omnidirectional stereo vision camera configuration system 10, which includes a main camera group 101, an auxiliary camera group 102, and an arithmetic unit 103 , Where: (1) The main camera group 101 includes at least four main camera lenses (1011~1014), and each main camera lens (1011~1014) can be arranged on four base lines that can form a rectangle; (2) The auxiliary camera group 102 includes at least four auxiliary camera lenses (1021 to 1024), and each main camera lens (1011 to 1014) and each auxiliary camera lens (1021 to 1024) are configured to be along four benchmarks The lines are arranged on the same plane at intervals, and are arranged as rectangles; (3) The computing unit 103 is respectively connected to the main camera lenses (1011~1014) and the sub camera lenses (1021~1024), wherein the computing unit 103 can have at least one A processor (not shown in the figure, such as a CPU and an MCU), which is used to run the arithmetic unit 103, and has functions such as logical operation, temporary storage of operation results, storage of execution instruction positions, and execution of image processing; (4) each sub-camera The lens (1021~1024) or the optical axis of each main camera lens (1011~1014) can be configured to be rotatable, so that the auxiliary camera lens (1021~1024) arranged on the reference line and the main camera The optical axes of the lenses (1011~1014) are parallel to each other, and the optical axes of the secondary camera lenses (1021~1024) and the main camera lenses (1011~1014) arranged on the reference line are perpendicular to the corresponding reference lines, Camera calibration is completed for both the main camera group 101 and the auxiliary camera group 102; (5) The camera lenses (1011~1014, 1021~1024) which are located on the same reference line and complete the camera correction, the captured images are available for calculation The unit 103 calculates at least two depth maps (also called disparity maps), and after the arithmetic unit 103 performs image fusion calculation on each depth map, it can eliminate the occlusion area between the depth maps and generate 1. Deblocking depth map; (6) The computing unit 103 can perform an image splicing on each deblocking depth map calculated by each reference line to obtain an omnidirectional depth map; (7) Continued In a preferred embodiment, each auxiliary camera lens (1021~1024) of the auxiliary camera group 102 or the main camera lens (1011~1014) of the main camera group 101 can be configured to the left based on the lens optical axis Or the main camera group 101 and the auxiliary camera group 102, which are rotated to the right by a certain angle and are located at the same reference line, the lens optical axes of them can be respectively oriented in the same direction or different directions before the camera calibration is completed; (8) In a preferred embodiment of the present invention, it may further include an inertial sensor 104 (IMU) coupled to the computing unit 103 ), which is used to transmit motion information and posture information of multiple degrees of freedom (DOF) to more accurately track how the subject and the shooting scene move in the real world. Among them, the degrees of freedom can be divided into two types: translation And rotation, translation can include X axis (front/rear), Y axis (left/right), Z axis (up/down), rotation can include pitch (Pitch), roll (Roll) and yaw (Yaw) , But not limited to these degrees of freedom.

； (4)承上，由於本創作於執行步驟S20時，並未改變各攝像鏡頭(1011~1014、1022~1024)的位置，故主相機組101與輔相機組102之攝像頭所接收到的光線都是相同的，依此，可透過旋轉鏡心成功模擬鏡頭光軸所面對的方向； (5)產生深度資訊(步驟S30)：由設置於同一基準線、且完成相機校正的主相機組101與輔相機組102所擷取的影像（例如「第5圖」所示的各拍攝影像，而本示意圖僅為舉例，並不以此些影像數量為限），供一運算單元103演算出不同角度的至少兩深度圖，且各深度圖經運算單元103執行影像融合演算後，可消除各深度圖彼此之間的遮擋區域，而生成一去遮擋深度圖，請搭配參閱「第6圖」，圖中左側的深度圖為運算單元103基於位於基準線L1的主攝像鏡頭101與副攝像鏡頭1021所擷取之影像，而生成之深度圖DL，而中間的深度圖為運算單元103基於位於基準線L1的主攝像鏡頭101與副攝像鏡頭1022所擷取之影像，而生成之深度圖DR，從圖中的深度圖DL與深度圖DR皆可發現部分的遮擋區域，而右側的深度圖D即為深度圖DL與深度圖DR作影像融合演算而生成的去遮擋深度圖； (6)產生全向深度圖(步驟S40)：運算單元103對於各基準線所演算出的各去遮擋深度圖，進行一影像拼接而獲取一全向深度圖。 Please refer to "Picture 3", which is the flow chart of the camera configuration of this creation, and please refer to "Picture 2", "Picture 4" ~ "Picture 6", this creation proposes an omnidirectional stereo vision camera The configuration method S includes: (1) arranging the camera group on the baseline (step S10): as shown in "Figure 4", arranging at least four main camera lenses (1011~1014) of a main camera group 101 in Four reference lines (L1~L4) forming a rectangle, at least four auxiliary imaging lenses (1021~1024) of an auxiliary camera group 102 are respectively arranged on each reference line (L1~L4), so that each main imaging lens ( 1011~1014) and each sub-camera lens (1021~1024) are arranged in a rectangle along the four reference lines (L1~L4) at the same plane interval; (2) correct the camera (step S20): rotate the auxiliary camera group 102 At least one auxiliary camera lens (1021~1024), or rotate the optical axis of the main camera lens (1011~1014) of the main camera group 101, so that the main camera lens (1011~1014) arranged on the reference line (L1~L4) The lens optical axes of the sub-camera lenses (1021~1024) are parallel to each other, and the optical axes of the main camera lenses (1011~1014) and the sub-camera lenses (1021~1024) arranged on the reference line (L1~L4) are both It is perpendicular to the corresponding reference line (L1~L4) to complete the camera calibration. Among them, the camera lenses (1011~1014, 1021~1024) can be configured to use the lens optical axis as the reference, and the two parts are to the left Or rotate it to the right by a specific angle, as shown in "Figure 4", and please refer to "Figure 2". The main camera lens 1011 at the reference line L1 shown in the figure can use its optical axis as the reference direction Rotate θ degrees to the left or right, and the secondary imaging lenses (1021, 1022) located on the left and right sides of the main imaging lens 1011 are arranged on the same reference line L1 as the lens centers of the main imaging lens 1011. The optical axis of the sub-camera lens (1021, 1022) is rotated by θ degrees, so that the y-z component (Ty, Tz) of the imaging lens (1011, 1021, 1022) located at the same reference line L1 in the offset matrix T tends to Close to 0, let the center coordinates of the camera lens (1011, 1021, 1022) form a zero rotation with each other, but only retain the X-axis offset (Tx); (3) Up, similarly, located at the reference line L2 The main camera lens 1012 is rotated by Φ degrees to the left or right based on its optical axis, and the auxiliary camera lenses (1022, 1023) located on the left and right sides of the main camera lens 1012 are The lens centers are arranged on the same reference line L2, so the optical axis of the sub-camera lens (1022, 1023) can be rotated by Φ degrees, so that the camera lens (1012, 1022, 1023) on the same reference line L2 is shifted Y, z components in matrix T (Ty, Tz) Close to 0, let the lens center coordinates of the camera lens (1012, 1022, 1023) form zero rotation with each other, and only retain the X-axis offset (Tx), as for the camera correction method of the reference line L3 and the reference line L4 , You can rotate the optical axis of the camera lens by the rotation angle φ and Ω respectively, which is the same as the above method for the reference lines L1 and L2, and so on, and will not be repeated here. Among them,

(4) Continued, because the author did not change the position of each camera lens (1011 ~ 1014, 1022 ~ 1024) when performing step S20, so the cameras received by the cameras of the main camera group 101 and the auxiliary camera group 102 The light is the same, so the direction of the lens optical axis can be successfully simulated by rotating the lens center; (5) Depth information is generated (step S30): the main camera is set on the same reference line and completes the camera calibration The images captured by the group 101 and the auxiliary camera group 102 (such as the captured images shown in "Figure 5", and this schematic diagram is only an example, and is not limited to the number of these images), for the calculation unit 103 to calculate At least two depth maps at different angles are produced, and after each depth map is subjected to image fusion calculation by the arithmetic unit 103, the occlusion area between the depth maps can be eliminated, and an unoccluded depth map is generated. Please refer to "Figure 6" The depth map on the left in the figure is the depth map DL generated by the operation unit 103 based on the images captured by the main camera lens 101 and the sub camera lens 1021 at the reference line L1, and the middle depth map is based on the operation unit 103 The depth map DR generated from the images captured by the main camera lens 101 and the sub camera lens 1022 at the reference line L1 can be found from the depth map DL and the depth map DR in the figure. Figure D is the unoccluded depth map generated by the image fusion calculation of the depth map DL and the depth map DR; (6) Generate an omnidirectional depth map (step S40): the computing unit 103 calculates each deocclusion calculated for each reference line Depth map, an image stitching to obtain an omnidirectional depth map.

Please refer to "Figure 7", which is another embodiment of the creation (1), and please refer to "Figure 2", this embodiment is similar to the technology of "Figure 2" to "Figure 6" The main difference is that, without relying on the projection of an active light source, the quality of 3D information is related to the texture level of the objects in the image. For example, blankets and newspapers are texture-rich objects, while white paper and monochrome walls are not. Textured objects, in addition, if the light source is insufficient, such as at night or indoors but not turned on, will also affect the quality of 3D information. Therefore, the omnidirectional stereo vision camera configuration system 10 of this embodiment is more It includes a diffractive optical element (DOE) 105 coupled to the arithmetic unit 103, and the diffractive optical element 105 can be respectively mounted on each main imaging lens (1011~1024) of the main camera group 101, and diffracted The optical element 105 is mainly used to project the light spot on the surface of the object to assist in determining the three-dimensional depth of the subject and the shooting scene, which means that when step S30 is executed, if the ambient light source is insufficient or the texture feature of the subject is not obvious, The diffractive optical element 105 coupled to the computing unit 103 can project a light spot on the surface of the object to give the texture and light source of the object to produce a desired pattern at a specific position or space to assist in determining the object and shooting The three-dimensional depth of the scene.

Please refer to "Figure 8", which is another embodiment of the creation (2), and please refer to "Figure 2", this embodiment is similar to the technology of "Figure 2" ~ "Figure 7" The main difference is that the omnidirectional stereo vision camera configuration system 10 of this embodiment further includes a light module 106 (Lidar) coupled to the computing unit 103 for pulse signals sent and received through measurement (for example Pulse laser) to calculate the depth information of the subject and the shooting scene, and the format of the depth information can be, for example, Point Cloud, where the information can include horizontal angle, vertical angle, distance, intensity, line, id, timestamp (Laser Timestamp) means that when step S30 is executed, the computing unit 103 can determine the time of flight (ToF) by measuring the time interval of the pulse signal sent and received through the light module 106 coupled thereto , And then calculate the depth information of the subject and the shooting scene, and before the light module 106 returns the depth information to the computing unit 103, the computing unit 103 can first perform each of the depth maps generated during step S30 and the When the images captured by the camera lens (1011~1014, 1021~1024) are image segmentation (image segmentation) for the computing unit 103 to obtain an omnidirectional depth map, it can be generated for a distant object or shooting scene More accurate depth information to make up for the problem that if only the light module 106 is used to detect the depth of a distant object or shooting scene, the returned depth information will have insufficient x and y information; Above, the omni-directional stereo vision camera configuration system 10 of this embodiment may further include a radar module 107 (Radar) coupled to the computing unit 103 for receiving the wireless reflection from the objects in the receiving space when step S30 is executed The radio wave is used by the computing unit 103 to calculate the depth information of the subject and the shooting scene, and the radar module 107 can be, for example, a millimeter wave radar (mmWave Rader). In other words, in this embodiment, the light module 106 and the radar are used The module 107 can solve the problem that the depth information of a distant object or scene may not be accurate when the creation of omnidirectional stereo vision through these camera lenses (1011~1014, 1021~1024).

The above are only the preferred embodiments of this creation, and are not intended to limit the scope of the implementation of this creation; anyone who is familiar with this skill will make equal changes and modifications without departing from the spirit and scope of this creation. It should be covered by the patent scope of this creation.

In summary, this creative department has patent requirements such as "industrial utility", "novelty" and "progressiveness"; the applicant has filed an application for a new type of patent with the Bureau of Law in accordance with the provisions of the Patent Law.

O: mirror heart b12: baseline b12’: Baseline b13: Baseline b13’: Baseline 10: Camera configuration system for omnidirectional stereo vision 101: Main camera group 101: Main camera group 1011: Main camera lens 1012: Main camera lens 1013: Main photographic lens 1014: main photographic lens 102: auxiliary camera group 1021: Vice photography lens 1022: Vice photography lens 1023: Vice photography lens 1024: secondary camera lens 103: arithmetic unit 104: Inertial sensor 105: Diffractive optics 106: Light module 107: Radar module L1~L4: baseline D: depth map DL: Depth map DR: depth map S: Camera configuration method for omnidirectional stereo vision S10: Configure the camera group on the baseline S20: calibrate the camera S30: Generate in-depth information S40: Generate omnidirectional depth map

Figure 1 is a schematic diagram of conventional camera calibration. Figure 2 is the architecture diagram of the camera configuration system created for this. Figure 3 is a flow chart of the camera configuration for this creation. Figure 4 is a schematic diagram of the implementation of this creation (1). Figure 5 is a schematic diagram of the implementation of this creation (2). Figure 6 is a schematic diagram of the implementation of this creation (3). Figure 7 is another embodiment (1) of this creation. Figure 8 is another embodiment (2) of this creation.

10: Camera configuration system for omnidirectional stereo vision

101: Main camera group

101: Main camera group

1011: Main camera lens

1012: Main camera lens

1013: Main photographic lens

1014: main photographic lens

102: auxiliary camera group

1021: Vice photography lens

1022: Vice photography lens

1023: Vice photography lens

1024: secondary camera lens

103: arithmetic unit

104: Inertial sensor

Claims (7)

An omnidirectional stereo vision camera configuration system, including: A main camera group including at least four main camera lenses, each of which is arranged on four reference lines that can form a rectangle; An auxiliary camera group, including at least four auxiliary camera lenses, and each of the main camera lens and each of the auxiliary camera lenses is arranged along the four datum lines on the same plane at intervals to form the rectangle; An arithmetic unit, which is connected to each of the main camera lens and the auxiliary camera lens to present information link; The optical axis of each of the sub imaging lenses or each of the main imaging lenses is configured to be rotatable, so that the optical axes of the sub imaging lens and the main imaging lens arranged on the reference line are parallel to each other, and arranged on the reference line The optical axes of the auxiliary camera lens and the main camera lens are both perpendicular to the corresponding reference line to complete the camera calibration; For the camera lenses that are located on the same reference line and have completed camera calibration, the captured multiple images are used by the arithmetic unit to calculate at least two depth maps, and the at least two depth maps are used by the arithmetic unit to perform image fusion calculations to eliminate Each occlusion area between the depth maps to generate an unoccluded depth map; and The computing unit performs image stitching on each of the unoccluded depth maps calculated by the reference lines to obtain an omnidirectional depth map. For example, the omnidirectional stereo vision camera configuration system of the first patent application scope, which further includes a diffractive optical element coupled to the computing unit, is used to project the light spot on the surface of the object, to assist in determining the subject and The three-dimensional depth of the shooting scene. An omnidirectional stereo vision camera configuration system as claimed in item 1 of the patent scope, wherein each auxiliary camera lens of the auxiliary camera group, or the main camera lens of the main camera group, is configured to use the optical axis as The datum rotates to the left or right. For example, the omnidirectional stereo vision camera configuration system of the first or second patent application scope further includes a light module coupled to the computing unit for the time interval of the pulse signal sent and received by the measurement To calculate the depth information of the subject and shooting scene. For example, the omnidirectional stereo vision camera configuration system of the fourth patent application, in which the light module has not yet returned the depth information to the arithmetic unit, the arithmetic unit is also used for the depth map and the cameras The image captured by the lens performs image segmentation. For example, the omnidirectional stereo vision camera configuration system of the first item of the patent scope, which further includes an inertial sensor coupled to the computing unit, is used to return multiple degrees of freedom of motion information and posture information. The omnidirectional stereo vision camera configuration system as claimed in item 1 of the patent scope further includes a radar module coupled to the computing unit for receiving radio waves reflected by objects present in the space for the computing The unit calculates the depth information of the subject and shooting scene.

Images