TW202406327A - Omni-directional image processing method - Google Patents

Abstract

An algorithm stored in a non-transitory computer-readable medium is able to perform "image shift" correction of a wide-angle field of view camera element from a first image acquired from the wide-angle field of view camera element and a main vibration vector from the motion vector detection circuit. The algorithm includes the following steps: a distribution function operation, calculating a vector component of each position vector of the first image in the main vibration vector and determining the corresponding distribution value; and a displacement conversion process, calculating the corresponding correction displacement with the distribution value and the main vibration vector for converting the first image into a second image, wherein the first image and the second image both have the larger image taking magnification ratio at a central area thereof than that at an edge area.

Description

Panoramic image processing method

This creation is about a panoramic image processing method; specifically about the correction of image offset, especially its processing speed.

120°)的攝像元件，例如：魚眼鏡頭(fish-eye lens)或反射式環狀鏡頭(panoramic annular lens)。僅需單顆鏡頭就能完成全景(omni-directional)攝影，不用移動部件，就能在半球形視場(hemispherical field-of-view)內提供平移、傾斜方向、旋轉和放大的影像。基於魚眼鏡頭收集光線的投影效果，在CCD或CMOS陣列上以圓形形分布的二維影像，可以使用高速電子電路進行數學校正，回歸至以投影角度表示的方形分布的二維影像。如「第1圖」所示，光線前進方向可以球座標(spherical coordinate)的餘緯角(colatitude)θ以及方位角(azimuth)Φ表示，這些光線經鏡片組101集光後，成像在距離f(有效焦距)處的成像平面，藉由二維影像感測器102光電轉換(photoelectrically conversion)為球形影像。 Camera systems are widely used in surveillance, inspection, security and remote sensing. Nowadays, in order to save installation space and simplify image processing and control steps, these systems have begun to use large-angle field of view (FOV).

120°) imaging element, such as a fish-eye lens or a panoramic annular lens. Only a single lens is needed to complete omni-directional photography, and without moving parts, it can provide pan, tilt, rotation and magnified images within a hemispherical field-of-view. Based on the projection effect of light collected by a fisheye lens, a two-dimensional image distributed in a circular shape on a CCD or CMOS array can be mathematically corrected using high-speed electronic circuits and returned to a two-dimensional image distributed in a square shape expressed by a projection angle. As shown in "Figure 1", the forward direction of light can be expressed by the colatitude θ and azimuth Φ of the spherical coordinate. After these light rays are collected by the lens group 101, they are imaged at a distance f The imaging plane at (effective focal length) is photoelectrically converted into a spherical image by the two-dimensional image sensor 102.

Generally speaking, a fisheye lens only bends with respect to the incident angle of light, which is the angle between the incident light and the optical axis 100 of the fisheye lens, which is represented by the co-latitude angle θ below. The incident light is imaged on the two-dimensional image sensor 102 after being incident from the spherical coordinates (θ, Φ), and the i-th horizontal reading Hrd and the longitudinal reading are carried out through photoelectric conversion. Take the jth Vrd to get the luminosity of this pixel (pixel). The positions of these pixels can be represented by the matrix (i, j); and the position vector of the imaging plane is expressed as (rcosΦ, rsinΦ), where π is 180 degrees. degrees (radian), and the optical axis 100 of the fisheye lens passes through the origin O of the imaging plane, and its distance from (i, j) is hereinafter referred to as the radius r.

In response to different shooting needs, the lens combination of the fisheye lens will also be different. The relationship between the co-latitude angle θ and the radius r can be roughly divided into three models, among which type I is the equidistance projection. )r=f． θ; type II is stereographic projection r=f. 2. tan(θ/2); and type III is equisolid angle projection r=f. 2. sin(θ/2). When a square of graph paper is placed in front of the lens of the type I, II and III projection relationships, the two-dimensional image sensor 102 will respectively display images from left to right as shown in "Figure 2". Unlike ordinary lenses whose field of view (FOV) is between 40° and 60°, the field of view of a fisheye lens can reach 180°. The width of the graph paper in the picture above is approximately 140°. The traditional small-angle field of view lens is called perspective projection, and the image is a square image with a diagonal length of about 43mm; while the lens that conforms to Type I, II, and III projections is a spherical image with a diagonal length. Only about 25-28mm.

Most of the fisheye lenses attached to mobile phones are Type I (far left picture). It is easiest to convert back to a square perspective image, and the brightness between the center and the edge will not differ too much. However, when the field of view exceeds 84°, image distortion will occur. It is very obvious; the fisheye lens commonly used for video conferencing or panoramic imaging is type II (middle picture), which retains the original shape (conformal mapping) of each object in the field of view by combining high-difference lenses or ring reflectors. There are problems such as over-bright edges and complex transfer functions, but it can ensure the correct appearance of the image; Type III (far right picture) is a fisheye lens that can switch the optical axis at any angle, especially suitable for first-person stereoscopic photography , but due to severe image distortion, it is not suitable to convert back to a square perspective image.

k₁．θ+k₂．θ³+k₃．θ⁵+k₄．θ⁷+k₅．θ⁹。這五個轉換參數k₁,k₂,…,k₅，加上光軸偏移與球面偏差的兩個校正參數，一共有七個變數與實際結果作比對，才能找出(i,j)至所述餘緯角θ以及所述方位角Φ的轉換矩陣(matrix)。然而，七個變數構成的轉換矩陣，在實際應用時極難取得穩定數值解，最後多以三階近似法，也就是r(θ)

k₁．θ+k₂．θ³，在出廠前驗算完成並設定好轉換矩陣在IC中。 There are still some differences between the above-mentioned models and the actual lens imaging, and except for Type I, other models have problems with numerical solutions. In the polynomial conversion method disclosed by J.Kannala & SSBrandt, it is shown that it takes at least the ninth order to completely convert the radius r from a small angle to a large angle r(θ)

k ₁ ． θ+k ₂ ． θ ³ +k ₃ ． θ ⁵ +k ₄ ． θ ⁷ +k ₅ ． θ ⁹ . These five conversion parameters k ₁ , k ₂ ,..., k ₅ , plus the two correction parameters of optical axis offset and spherical deviation, there are a total of seven variables that can be compared with the actual results to find out (i, j ) to the co-latitude angle θ and the azimuth angle Φ. However, it is extremely difficult to obtain a stable numerical solution for the transformation matrix composed of seven variables in practical applications. In the end, the third-order approximation method is often used, that is, r(θ)

k ₁ ． θ+k ₂ ． θ ³ , the calculation is completed and the conversion matrix is set in the IC before leaving the factory.

It can be seen from this that when the large-angle field of view camera element encounters "image offset" caused by hand shake or other vibrations, the offset amount and direction of each pixel are different. For example: the second picture of US9,800,780 The fourth picture. For perspective projection lenses, it only takes a few sampling points to know the direction and magnitude of the entire "image offset"; in addition to having a wide and dense sampling range, the fisheye lens also has to cope with the non-stationary "movement" of the subject. Image", resulting in erroneous sampling. Therefore, the Image Signal Processor (ISP) in US9,800,780, which is responsible for image acquisition and processing, in addition to a circuit specifically for calculating subject motion vectors, also has a circuit for counting the divergence points and divergence points of these motion vectors. Assemblage point circuit.

If the vibration of the fisheye lens is not corrected, the consequences are not just shaky image quality. For example: WO98/27718A1 utilizes the characteristic that the peripheral image is compressed to a higher degree than the center to perform electronic zoom without reducing the resolution; however, as the zoom ratio is high, the vibration of the fisheye lens will cause image distortion. Causes resolution degradation. Another example: US5,185,667 cuts the image in the hemispheric field of view obtained through the fisheye lens and performs translation, tilt, rotation and magnification; the "image offset" caused by the vibration of the lens will make the cut image dim. It suddenly becomes brighter, warmer (yellowish), colder (bluer), or even distorted lines appear.

With the rise of electric vehicles and the Internet of Vehicles, around-view monitoring (AVM), one of the driver assistance systems, has also received active research. It directly provides the driver with the surrounding environment of the vehicle through multiple lenses installed in the Internet of Vehicles, and can even Analyze surrounding parking spaces, lanes or foreign objects on the road and issue warnings. Taking four lenses as an example, as shown in "Picture 3", the front lens Lf will cover part of the left and part of the right image, just making up for the road surface not illuminated by the left lens Lt and the right lens Lr of the car; similarly, The rear lens Lb can also make up for the parts of the left lens Lt and the right lens Lr that are blocked by the wheels. As long as you make some angle corrections to the image, you can create the feeling of shooting the vehicle and the surrounding environment directly above the vehicle. If you want to further analyze the lane conditions or even start the artificial intelligence parking process, the overlapping images of these four lenses need to be stitched together first. Since the car body will vibrate when the engine is running or due to uneven road surfaces, and the linkage relationship between each lens and the car body is different, the "image offset" produced by each lens will of course be different; it is easy to occur during the stitching and collage process. Poor splicing causes the bird's-eye view provided to the driver (such as Figure 4) to be interrupted from time to time, causing trouble to the driver.

Lens vibration will also affect the 3A (auto white balance/auto exposure/auto focus) processing in the image signal processor (ISP), especially the Auto White Balance (AWB) algorithm. Although each major brand has its own unique AWB algorithm, it is generally based on the logic of counting the RBG color temperature distribution in pixels to adjust the RGB gain. If there are subjects with different color temperatures in the field of view of the lens, for example, parking lot lights and various car lights have different color temperatures, the ISP can select the color temperature of most pixels, or select a color temperature biased towards the central subject. However, the "image shift" caused by lens vibration may accelerate the instability of color temperature statistics, causing the AWB algorithm to be repeatedly checked and slowing down the overall speed of the ISP. In the end, the image may appear to flicker between yellow and blue. Therefore, US9,105,105 proposes a method of simultaneously monitoring the brightness histogram (histogram) to select the brightness Turn off AWB correction when it is too low to avoid color temperature correction distortion.

It can be seen from the above that the impact caused by lens vibration is not small. Especially for panoramic images with a large difference in magnification ratio between the central area and the edge area, it is difficult to solve the problem with a single translation amount, for example: US7,511,756. In the square image converted back in this way, the change from the center area to the edge area is too great, which causes the edge area to be blurred even more due to correction. Therefore, US7,834,907 reveals a translational and slightly smaller spherical image range, in an attempt to reduce the huge gap between the central area and the edge area by reducing the size.

"Image shift" in this manual refers to the movement of the subject in the image, which is caused by changes in the imaging position of the subject caused by hand-holding, wind blowing or mechanical linkage; and "movement" (movement) image) refers to the movement of the subject in the image, which is the change in the imaging position of the subject caused by the movement of the subject relative to the lens. The vibration (shake) in this manual refers to the alternating action (act) in a short period of time, such as: the back and forth movement of the human body to maintain balance, the swing caused by strong wind, and the shaking caused by people and vehicles passing on the road; and vibration ( Vibration refers to periodic motion, which roughly has a balance point, for example: a lens assembly or a fixed part linked to the lens moves back and forth due to sudden external force.

A panoramic image processing method, stored in a non-transitory computer-readable medium, uses a main vibration vector provided by a camera device and a first image to convert a second image, wherein, The central area of the first image and the second image both have greater image magnification than the edge area. The method includes the following steps: a distribution function operation to calculate the projection value of each pixel of the first image to the main vibration vector to determine the distribution value corresponding to the projection value; and a displacement conversion procedure to obtain the first image, and based on The distribution value and the value of the main vibration vector calculate the corresponding correction displacement to convert the second image. The main vibration vector can be expressed as a Euclidean vector, falling on the imaging plane where the camera device photoelectrically converts the first image, and the "image offset" caused by the vibration of the camera device Related, and the optical axis of the imaging device passes through the origin of the imaging plane. The projection value and the distribution value roughly form a Gaussian distribution (Gauss Distribution). The distribution value can be obtained by using mathematical formulas, or the distribution value can be obtained by inputting the projection value using a look-up table.

Also, a panoramic image processing method is stored in a non-transitory computer-readable medium, using a main vibration vector provided by a camera device and a first image to convert a third image, Wherein, the central area of the first image has a greater image magnification than the edge area, and the image magnification of the central area and the edge area of the third image is approximately the same. The method includes the following steps: a distribution function operation to calculate the projection value of each pixel of the first image to the main vibration vector to determine the distribution value corresponding to the projection value; a displacement conversion program to obtain the first image, and Calculate the corresponding correction displacement according to the distribution value and the value of the main vibration vector to convert a second image, and the central area of the second image has a greater image magnification than the edge area; and a projection conversion process, according to the The first image is converted into a perspective projection image by converting the first image or the second image into the third image. The main vibration vector can be expressed as a Euclidean vector, falling on the imaging plane where the camera device photoelectrically converts the first image, and the "image offset" caused by the vibration of the camera device Relevantly, the optical axis of the camera device passes through the origin of the imaging plane. The projection value and the distribution value roughly form a Gaussian distribution (Gauss Distribution). You can use mathematical formulas to obtain the distribution value, or you can use a look-up table. table) input the projection value to obtain the distribution value.

The projection conversion program determines to use the first image or the second image to convert the third image based on the first component.

A panoramic camera device used to capture fish-eye images within a field of view, which includes: a panoramic image capture element that uses photoelectrically conversion to convert the fish-eye image, wherein the center area of the fish-eye image is larger than the edge area With greater image magnification; a vibration detection block to obtain vibration information about the panoramic camera equipment and analyze a main vibration vector; a displacement operation block to calculate each pixel from the direction of the main vibration vector The projection value, and determines the correction displacement of each pixel according to the value of the main vibration vector and the projection value; and a conversion block can convert the fisheye image into an output image with reference to the correction displacement. The main vibration vector is related to the "image offset" caused by vibration of the panoramic camera equipment, and the projection value and the correction displacement generally present a Gaussian distribution (Gauss Distribution).

Gb: car rear floor image

Gf: car front floor image

Gr: Car right floor image

Gt: car left floor image

Hrd: read horizontally

f: effective focal length of lens group

i: read serial number horizontally

j: read serial number vertically

k ₁ : first-order term coefficient

k ₂ : third-order term coefficient

k ₃ : fifth-order term coefficient

k ₄ : seventh-order term coefficient

k ₅ : ninth-order term coefficient

Lb: car rear shot

Lf: car front lens

Lr: Car right lens

Lt: Car left lens

O: origin

r:radius

Vrd: vertical reading

β: refraction angle

Φ: azimuth angle

θ: co-latitude angle

100:Optical axis

101: Lens set

102: Two-dimensional image sensor

103: Panoramic camera element

104: Vibration detection block

105:Conversion block

106: Displacement operation block

107:Memory control interface

108: First memory area

109: Second memory area

110: Video output block

62: Projection value

63: Distribution value

69:Gaussian distribution function

81: The first image with an "image shift" of 1.6° to the right

82: The first image of "Image Shift" 16° to the right

91: The second image shifted to the right by 1.6° "image offset"

92: The second image with an "image shift" of 16° to the right

Figure 1 is a three-dimensional schematic diagram of the imaging plane and lens group of the camera equipment

Figure 2 is a schematic diagram of a spherical image using equidistant projection, planar projection and solid angle projection.

Figure 3 is the configuration diagram of the AVM system used in this creation.

Picture 4 is a bird’s eye view of this creation applied to vehicles

Figure 5 is the result of conventionally using equal correction displacement within a lateral field of view of 80°.

Figure 6 shows the distribution of the degree of correction that should be corrected for the "image offset" of this creation to the right.

Figure 7 is a demonstration of the use of this creative lookup table.

Picture 8 is the first and second images of this creation with an "image shift" of 1.6° to the right.

Picture 9 is the first and second images of this creation with an "image shift" of 16° to the right.

Picture 10 is the third image of this creation that is "image shifted" to the right.

Figure 11 is a schematic diagram of the circuit block of the panoramic camera equipment of this creation

120°)攝像元件「影像偏移」的演算法，所述攝像元件係由鏡片組101與二維影像感測器102組合，再由一影像訊號處理器(ISP)轉換該二維影像感測器102上的類比訊號為數位訊號。本創作得以儲存於非暫態電腦可讀取媒體中，並從ISP取得數位訊號，以及從運動向量檢測電路取得所述攝像元件的振動資訊，以計算「影像偏移」的校正位移。所述攝像元件的ISP以及所述運動向量檢測電路常整合成一塊電路板或單晶片，並與所述攝像元件組合成一攝像設備。本創作一樣可整合至所述電路板或單晶片上，或是建置在與所述攝像設備連接之電腦或手機的應用程式記憶體裡。 This product is a corrected wide-angle field of view (FOV)

120°) algorithm for "image offset" of the imaging element. The imaging element is composed of a lens group 101 and a two-dimensional image sensor 102, and then an image signal processor (ISP) converts the two-dimensional image sensing The analog signal on the device 102 is a digital signal. This creation can be stored in a non-transitory computer-readable medium, obtain digital signals from the ISP, and obtain the vibration information of the imaging element from the motion vector detection circuit to calculate the correction displacement of the "image offset". The ISP of the imaging element and the motion vector detection circuit are often integrated into a circuit board or a single chip, and combined with the imaging element to form an imaging device. The invention can also be integrated into the circuit board or single chip, or built into the application memory of a computer or mobile phone connected to the camera device.

Please refer to "Figure 5", the conventionally used equal correction displacement within 80° of the lateral field of view is the results before correction (left) and after correction (right) (solid line). The upper two pictures show the "image shift" results of about 1.6° to the right; the lower two pictures show the "image shift" results of about 16° to the right; and the dotted lines in the pictures are used to mark the results without vibration. Obviously, the conventional electronic anti-shake method that corrects displacement has reached the correction limit at FOV 60° (dotted line frame). As mentioned in the previous technology, the magnification ratio of the image captured by the large-angle camera element is very different between the center area and the edge area, so the "image offset" will be enlarged in the center area and reduced in the edge area. Overall The image size in the center area will become smaller. This is why US7,834,907 calculates the reduced size of the central area, and then enlarges it back to the previous size.

Please refer to "Picture 6". In this creation, the distribution of the degree of correction required for "image offset" skewed to the right. The farther the pixel is to the right or the left, the smaller the degree of correction required. On the contrary, the farther to the center, the smaller the degree of correction required. The more you have to move to the left (negative value). The upper picture shows the distribution of the degree of correction for an "image shift" of about 1.6° to the right; the lower picture shows the distribution of the degree of correction for an "image shift" of about 16° to the right. The scales of the degree of correction have been separated. Use 1.6° and 16° to normalize. This creation reveals that when used in practical applications, the "image offset" is not a linear relationship as envisioned in US7,834,907, but is roughly a Gaussian distribution (Gauss Distribution), and after normalizing by the value of the main vibration direction, there is not much difference between them. .

During implementation, the invention is stored in a non-transitory computer-readable medium, and obtains the first image and the main vibration vector from the camera element, which includes the following steps: a distribution function operation to calculate each of the first images The projection value of the pixel to the main vibration vector is used to determine the distribution value corresponding to the projection value; and a displacement conversion program calculates the corresponding correction displacement according to the distribution value and the value of the main vibration vector to convert the second image. The direction of the main vibration vector can be expressed as two components on the first image. By doing the inner product with the first image position vector (rcosΦ, rsinΦ), the projection value 62 can be obtained; then use the set For the look-up table, please refer to "Figure 7" to find the distribution value 63 corresponding to the projection value 62. The look-up table (LUT) used in practical applications is a transistor logic circuit, which can be regarded as a memory element. As long as the corresponding value is burned in advance, the value after complex calculation can be quickly accessed. Or directly bring the projection value 62 into the Gaussian distribution function 69. For example, first calculate the square of the projection value 62 divided by a set parameter and then bring it into the exponential function table (Exponential Function table) to obtain the distribution value. 63. "Figure 6" also shows that the correction displacement is roughly proportional to the value of the main vibration vector.

The value of the main vibration vector is related to the "image offset" caused by vibration of the camera equipment. Most of the camera equipment has a motion vector detection circuit, such as three-axis acceleration. accelerator and its analysis circuit, a helical accelerator and its analysis circuit, and a compound eye detector (please refer to FIG.10 of US7,511,756). The former two directly measure the acceleration signal of the camera equipment, so they are particularly suitable for use in low light conditions, but require coordinate conversion; although the latter is affected by light, it is the most intuitive result and can directly analyze the subject. Vibration vector. The conversion of the first two mainly includes: moving one unit length to the left from the measurement signal, and then dividing it by the distance from the subject to the camera device or its arctangent function solution to obtain the original optical axis directly in front The angle at which the subject shifts to the right; and then the angle of the right shift is brought into the projection relationship of the camera equipment (Figures 1 and 2) to obtain the radius r, which is the value of the main vibration vector; and The direction of the main vibration vector can be directly obtained from measurement signals such as a three-axis accelerometer. The main vibration vector is only related to the degree of "image offset" directly in front of the optical axis, and there is no need to calculate the average result from other angles.

Please refer to "Picture 8", the first image 81 and the second image 91 of this creation are "image shifted" 1.6° to the right. In the displacement conversion process, the second image is converted according to the corrected displacement corresponding to each pixel of the first image. At a low "image shift" range, this creation can not only correct lenses with an FOV of more than 60°, but can even be applied to lenses with an FOV of 140° without any problem.

Please refer to "Picture 9". This creation shows the first image 82 and the second image 92 with an "image offset" of 16° to the right. In the displacement conversion process, the second image is converted according to the corrected displacement corresponding to each pixel of the first image. Since the 16° "image offset" has reached a severe vibration amplitude, just like running, the degree of image distortion is quite large. However, this creation only uses single-axis parameters to fit, and it can still be applied to longitudinal FOV60° and horizontal FOV140 ° lens, that is, the degree of correction along the main vibration vector is quite high.

Another embodiment of the invention further includes: a projection conversion program that converts the second image into a perspective projection image in a manner that converts the first image into a perspective projection image. Replace it with the third image. Please refer to "Picture 10". The image magnifications of the central area and the edge area of the third image are roughly the same. The upper left picture shows the third image with an "image shift" of 1.6° to the right, and the lower left picture shows the third image with an "image shift" of 1.6° to the right. For this third image with 16° "image offset", the dotted line is the image without vibration (only the FOV is about 82°). It shows that the "image offset" correction of this creation can maintain the stability of the main field of view even in the face of intense vibration, even when it is converted back to a perspective image. In the face of common small vibrations, such as hand shake, wind shake or car body vibration, this kind of swing can still maintain the stability of the picture at large angles. As shown in the picture on the right, the third image (FOV140°) with an "image shift" of 1.6° to the right almost overlaps with the image without vibration (dashed line).

Please refer to "Figure 11" for a schematic diagram of the circuit block of the panoramic camera equipment of this creation. The panoramic camera device images the light collected by the lens group 101 on the two-dimensional image sensor 102 through a panoramic image capturing element 103. The analog signal through photoelectrically conversion by the image signal processor is: Digital signal (A/D), automatic gain correction (AGC), automatic white balance (AWB), RGB color light switching, etc. The processed digital signal, which is the fisheye image in the front field of view, is stored through a memory control interface 107 in the first memory area 108. The central area of the fisheye image has a greater image magnification than the edge area and cannot be processed by optical or non-optical traditional image stabilization circuits (image stabilization). However, it can still be combined with a conventional motion detection circuit and an image offset. The analysis circuit forms a vibration detection block 104 and calculates a main vibration vector for a displacement operation block 106 to determine the correction displacement of the fisheye image. It uses an inner product circuit to calculate the projection value of each pixel to the main vibration vector, and finds the correction displacement through a Gaussian lookup table. A conversion block 105 may refer to the correction displacement to convert the fisheye image into an output image and store it in a second memory area 109. Finally, a video output block 110 generates a stable video source according to the first memory area 108 and the second memory area 109 .

The output image is obtained by the conversion block 105 converting the pixel positions of the fisheye image according to the correction displacement. The central area has a greater image magnification than the edge area, and is only corrected for "image offset"; also According to the method of converting the fisheye image into a perspective projection image, a distortion-free lookup table can be burned in the conversion block 105 in advance, and then the pixel position of the fisheye image can be converted accordingly; the pixel position of the fisheye image can also be converted into a perspective projection image. The distortion-free lookup table is coupled with the correction displacement superposition circuit to convert the pixel position of the fisheye image.

In one embodiment, the main vibration vector is represented by a component of the coordinates of the two-dimensional image sensor 102, and the image offset analysis circuit can set a threshold to output the main vibration vector with a value of 0. The video output block 110 may use a multiplexer to determine whether to output the fisheye image of the first memory area 108 or the output image of the second memory area 109 when the main vibration vector is 0. Similarly, the displacement operation block 106 may also use a multiplexer to decide not to execute the inner product circuit and search operations when the main vibration vector is 0. The invention may further include: a central processing unit (CPU) to control the operation and bus control of the vibration detection block 104, the displacement calculation block 106 and the conversion block 105, for example: receiving the user to switch the displacement. The operations of the perspective projection of the computing block 106 and the conversion block 105, and setting the threshold of the vibration detection block 104 according to the system statistics, etc.

In addition, the panoramic image capturing element 103 can also adjust or switch parameters such as automatic exposure, automatic white balance or automatic gain correction according to whether the vibration detection block 104 detects "image offset" or "moving image". ; It can also be determined whether to restore the parameters of the previous image based on whether the displacement operation block 106 and the conversion block 105 are executed.

Please refer to "Picture 3" and "Picture 4". This creation is applied to the AVM system configuration diagram and bird's-eye view. Due to the structure of the car body, most of them use the front, rear, right and right sides of the car. Car left lens (Lf, Lb, Lr, Lt) to complete. In order to capture a panoramic picture as much as possible, large-angle lenses are often used. After the conversion block 105 converts the perspective projection of the front, rear, right and left floor images (Gf, Gb, Gr, Gt) of the car, the AVM The system's processor performs the tiling. Due to the vibration of the car body, the AWB will be interfered with by other car lights or mirror reflections from time to time. If the "image offset" is not processed, the picture will become yellowish and sometimes bluish. Applying the AVM system of this invention, the "image offset" of vehicle body shake can be automatically corrected, and when it is determined that the correction displacement is indeed applied to the conversion block 105, the AWB of the fisheye image can be turned off or a fixed white balance parameter can be used. , providing clear and stable output images.

The light collection range of the fisheye lens is too wide to use the traditional optical anti-shake method, which controls the position of one lens of the lens group 101 based on the vibration measurement signal. The optical anti-shake method of the fisheye lens cannot only adjust the position of one lens, but the entire lens group 101 must be moved together, as shown by the dotted line in the panoramic camera element 103 in Figure 11. However, the motor required to move the entire lens assembly 101 takes up a lot of space, and its adjustable range is also limited. Using the panoramic camera device or method of this invention, the space of the motor can be saved so that the panoramic camera element can completely fit into the thin housing; when paired with the optical anti-shake, it can supplement the large-movement vibration that the optical anti-shake cannot handle. Correction.

This creation provides a fast and space-saving lookup table "image offset" correction. The number of input and output terminals of the Gaussian lookup table is the width of the spherical image (unit: pixels), which is about two thousand to four thousand, regardless of the main vibration. Any vector size is applicable. Since the LUT space used is small enough and the inner product circuit is simple, the displacement calculation block 106 can be completed using a field programmable array (FPGA); even if the projection method of the panoramic camera element is changed, for example: from type I If you switch to Type II, you can simply replace the variable parameters in the FPGA and continue to use it. Although this manual is based on horizontal Directional vibration is an example, but any vibration parallel to the imaging plane or vibration in the imaging plane caused by the inclination of the optical axis does not deviate from the scope of this creation.

To sum up, the panoramic image processing method of this creation has indeed met the requirements for patent application, and the patent application can be filed in accordance with the law. However, the above are only preferred embodiments of this invention and should not be used to limit the scope of implementation of this invention; therefore, any simple equivalent changes and modifications made based on the patent scope of this invention and the content of the specification are It should still be within the scope of this creative patent.

101: Lens set

102: Two-dimensional image sensor

103: Panoramic camera element

104: Vibration detection block

105:Conversion block

106: Displacement operation block

107:Memory control interface

108: First memory area

109: Second memory area

110: Video output block

Claims (10)

A panoramic image processing method, stored in a non-transitory computer-readable medium, using a main vibration vector provided by a camera device and a first image to convert a second image, wherein the Both the central area of the first image and the second image have greater image magnification than the edge area, which includes the following steps: A distribution function operation calculates the projection value of each pixel of the first image to the main vibration vector to determine the distribution value corresponding to the projection value, and the projection value and the distribution value approximately form a Gaussian distribution (Gauss Distribution); and A displacement conversion program obtains the first image, and calculates the corresponding corrected displacement according to the distribution value and the value of the main vibration vector to convert the second image, where the main vibration vector can be expressed as a Euclidean The vector (euclidean vector) falls on the imaging plane where the camera device photoelectrically converts the first image, and is related to the "image offset" caused by vibration of the camera device, and the optical axis of the camera device passes through The origin of the imaging plane. In the distribution function operation described in claim 1, the meaning of the calculation is based on the inner product of the origin of each pixel position of the first image on the imaging plane and the unit direction of the main vibration vector; and The meaning of the determination refers to using a look-up table to obtain the distribution value based on the projection value. A panoramic image processing method, stored in a non-transitory computer-readable medium, using a main vibration vector provided by a camera device and a first image to convert a third image, wherein the The central area of the first image has greater image magnification than the edge area, which includes the following steps: A distribution function operation to calculate the projection value of each pixel of the first image to the main vibration vector to determine the distribution value corresponding to the projection value, and the projection value and the distribution value approximately form a Gaussian distribution (Gauss Distribution); A displacement conversion program obtains the first image, and calculates the corresponding correction displacement according to the distribution value and the value of the main vibration vector to convert the second image, and the central area of the second image has a larger value than the edge area. image magnification; and A projection conversion program that converts the first image or the second image into the third image in a manner that converts the first image into a perspective projection image, and the center area and edge of the third image The image magnification of the area is approximately the same, where the main vibration vector can be expressed as a Euclidean vector, falling on the imaging plane where the camera device photoelectrically converts the first image, and the same as the camera It is related to the "image offset" caused by vibration of the equipment, and the optical axis of the imaging equipment passes through the origin of the imaging plane. In the distribution function operation described in claim 3, the meaning of the calculation is based on the inner product of the origin of each pixel position of the first image on the imaging plane and the unit direction of the main vibration vector; and The meaning of the determination refers to using a look-up table to obtain the distribution value based on the projection value. A panoramic camera device used to capture fisheye images within the field of view, which includes: A panoramic image capturing element uses photoelectrically conversion to convert the fisheye image, wherein the central area of the fisheye image has a greater image magnification than the edge area; A vibration detection block to obtain vibration information about the panoramic camera equipment and analyze it Get a main vibration vector; A displacement operation block calculates the projection value of each pixel from the direction of the main vibration vector, and determines the corrected displacement of each pixel based on the value of the main vibration vector and the projection value; and A conversion block converts the fisheye image into an output image according to the corrected displacements, wherein the main vibration vector can be expressed as a Euclidean vector falling on the panoramic image capturing element The photoelectric conversion of the fisheye image on the imaging plane is related to the "image offset" caused by vibration of the panoramic camera equipment, and the optical axis of the panoramic image capturing element passes through the origin of the imaging plane. The conversion block as described in claim 5 further includes: converting the output image in conjunction with the correction displacements according to a conversion method of converting the fisheye image into a perspective projection image. The displacement calculation block as described in claim 5 further includes: a Gaussian lookup table having the projection value input terminal and the main vibration vector value input terminal for outputting the corrected displacement. The displacement calculation block of claim 6 further includes: a distortion-free lookup table having the correction displacement input terminal and the pixel input terminal for outputting the distortion-free conversion displacement and converting the output image accordingly. The panoramic camera device as described in claim 5 further includes: a central controller that controls the vibration detection block and the displacement calculation block to cause the main vibration when the vibration information is insufficient to constitute an image offset. The value of the vector is 0, and the displacement operation block is inactive to output the corrected displacement as 0. The panoramic camera device as described in claim 6 further includes: a central controller that controls the conversion block for receiving an external signal to switch the output image into a spherical image or a square image, wherein, When the external signal is designated to output a spherical image, the central controller instructs the conversion block to convert the fisheye image to the output image only based on the correction displacements; and when the external signal is designated to output a square image, the central controller instructs the The conversion block converts the fisheye image into an output image according to the correction displacements and the perspective projection conversion method.

Images