CN117237237A - Luminosity balancing method and device for vehicle-mounted 360-degree panoramic image - Google Patents
Luminosity balancing method and device for vehicle-mounted 360-degree panoramic image Download PDFInfo
- Publication number
- CN117237237A CN117237237A CN202311501652.9A CN202311501652A CN117237237A CN 117237237 A CN117237237 A CN 117237237A CN 202311501652 A CN202311501652 A CN 202311501652A CN 117237237 A CN117237237 A CN 117237237A
- Authority
- CN
- China
- Prior art keywords
- image
- view
- overlapping
- vehicle
- cost function
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 235000004522 Pentaglottis sempervirens Nutrition 0.000 claims abstract description 36
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 27
- 238000006243 chemical reaction Methods 0.000 claims abstract description 24
- 230000004927 fusion Effects 0.000 claims abstract description 21
- 230000006870 function Effects 0.000 claims description 64
- 238000000354 decomposition reaction Methods 0.000 claims description 9
- 239000011159 matrix material Substances 0.000 claims description 8
- 230000000007 visual effect Effects 0.000 claims description 5
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 4
- 240000004050 Pentaglottis sempervirens Species 0.000 description 15
- 230000008569 process Effects 0.000 description 7
- 239000003086 colorant Substances 0.000 description 6
- 238000004590 computer program Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 238000005286 illumination Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 241000905137 Veronica schmidtiana Species 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 238000003786 synthesis reaction Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Image Processing (AREA)
- Studio Devices (AREA)
Abstract
The invention discloses a luminosity balancing method and device for a vehicle-mounted 360-degree panoramic image, comprising the following steps: acquiring a multi-view camera image, and performing view conversion on the multi-view camera image to obtain a bird's eye view image; selecting an overlapping region in the aerial view image, and establishing a cost function according to the overlapping region; solving the cost function to obtain compensation parameters corresponding to each overlapping region; and compensating the corresponding overlapping regions according to the obtained compensation parameters, fusing all the overlapping regions according to a region fusion algorithm, and outputting fused images. The invention solves the problem that obvious boundaries exist between adjacent views due to unbalance of luminosity among the views in the panoramic synthetic image.
Description
Technical Field
The invention relates to the technical field of auxiliary driving, in particular to a luminosity balancing method and device for a vehicle-mounted 360-degree panoramic image.
Background
Panoramic camera systems are one of the most commonly used technologies of Advanced Driving Assistance Systems (ADAS) for automobiles, and passenger cars equipped with the systems are almost equipped with the systems at present, and the systems provide 360-degree surrounding environment of the vehicle for drivers, so that the drivers can be helped to park and steer more safely. A panoramic camera system typically consists of four to six wide angle cameras, each facing in one direction, mounted around the vehicle. Through the input of the cameras, the system can synthesize and display a bird's eye view with a certain height centering on the own vehicle to the driver in real time, thereby assisting the driver to complete various operations.
In the panoramic synthesis mode of the existing panoramic camera system, as different cameras face different illumination conditions and different Automatic Exposure (AE) and Automatic White Balance (AWB) settings, obvious boundaries exist between adjacent views of a synthesized image, and the unbalance of luminosity between the boundaries and each view can seriously influence user experience, so that a seamless splicing method between overlook projection views of images of the multi-view cameras becomes very important.
Accordingly, there is a need in the art for improvement.
Disclosure of Invention
The invention aims to solve the technical problem that aiming at the defects of the prior art, the invention provides a luminosity balancing method and device for a vehicle-mounted 360-degree panoramic image, so as to solve the problem that obvious boundaries exist between adjacent views due to unbalance of luminosity among various views in the existing panoramic synthetic image.
The technical scheme adopted for solving the technical problems is as follows:
in a first aspect, the present invention provides a method for photometric equalization of a vehicle-mounted 360-degree panoramic image, including:
acquiring a multi-view camera image, and performing view conversion on the multi-view camera image to obtain a bird's eye view image;
selecting an overlapping region in the aerial view image, and establishing a cost function according to the overlapping region;
solving the cost function to obtain compensation parameters corresponding to each overlapping region;
and compensating the corresponding overlapping regions according to the obtained compensation parameters, fusing all the overlapping regions according to a region fusion algorithm, and outputting fused images.
In one implementation manner, the performing the view conversion on the multi-view camera image to obtain a bird's eye view image includes:
establishing a virtual camera with customized inner parameters and outer parameters, and projecting the multi-view camera image to the view angle of the virtual camera;
or according to the size of the defined aerial view angle picture, calculating an affine matrix by an algorithm for solving the 3D-2D point-to-motion, and converting each pixel of the multi-view camera image into the aerial view angle.
In one implementation, the selecting the overlapping area in the aerial image includes, before:
and performing image format conversion on the aerial view image to obtain a converted aerial view image.
In one implementation, the selecting an overlapping area in the aerial view image and establishing a cost function according to the overlapping area includes:
selecting an overlapping region in the converted aerial view image, and determining an interested region according to the overlapping region; wherein the region of interest is the whole or part of the overlapping region;
and calculating the mean and variance of the first channel and the second channel of the color space of each region of interest, and establishing the cost function according to the calculated mean and variance.
In one implementation manner, the selecting an overlapping area in the aerial view image, and establishing a cost function according to the overlapping area, further includes:
selecting an overlapping region in the aerial view image in the source format, and determining an interested region according to the overlapping region;
and calculating the mean and variance of the third channel of the color space of each region of interest, and establishing the cost function according to the calculated mean and variance.
In one implementation, the compensation parameters include: gain and bias;
the step of solving the cost function to obtain compensation parameters corresponding to each overlapping area comprises the following steps:
the cost functions are combined and converted into a matrix form;
solving the gain according to the singular value decomposition mode, and substituting the solved gain into a cost function with deviation;
and solving according to the singular value decomposition mode to obtain deviation.
In one implementation manner, the compensating the corresponding overlapping area according to the obtained compensation parameter, and fusing all the overlapping areas according to an area fusion algorithm, and outputting a fused image, including:
compensating the corresponding overlapping region according to the gain and the deviation obtained by solving;
fusing all the overlapped areas with different weights according to the area fusion algorithm;
and splicing the aerial views under all the visual angles, and outputting 360-degree looking-around images.
In a second aspect, the present invention provides a light equalizing device for a vehicle-mounted 360-degree panoramic image, including:
the aerial view module is used for acquiring a multi-view camera image, and performing view conversion on the multi-view camera image to obtain an aerial view image;
the cost function module is used for selecting an overlapping area in the aerial view image and establishing a cost function according to the overlapping area;
the compensation parameter module is used for solving the cost function to obtain compensation parameters corresponding to each overlapping region;
and the compensation and output module is used for compensating the corresponding overlapping areas according to the obtained compensation parameters, fusing all the overlapping areas according to the area fusion algorithm and outputting the fused image.
In a third aspect, the present invention provides a terminal comprising: the system comprises a processor and a memory, wherein the memory stores a luminosity balancing program of the vehicle-mounted 360-degree panoramic image, and the luminosity balancing program of the vehicle-mounted 360-degree panoramic image is used for realizing the operation of the luminosity balancing method of the vehicle-mounted 360-degree panoramic image according to the first aspect when being executed by the processor.
In a fourth aspect, the present invention further provides a medium, where the medium is a computer readable storage medium, where the medium stores a luminosity balancing program of a vehicle-mounted 360-degree panoramic image, where the luminosity balancing program of the vehicle-mounted 360-degree panoramic image is executed by a processor to implement the operations of the luminosity balancing method of the vehicle-mounted 360-degree panoramic image according to the first aspect.
The technical scheme adopted by the invention has the following effects:
according to the invention, by selecting the overlapping areas in the aerial view image and establishing the cost function according to the overlapping areas, the compensation parameters corresponding to the overlapping areas can be obtained by solving the cost function, so that the overlapping areas can be compensated according to the obtained compensation parameters, all the overlapping areas are fused according to the area fusion algorithm, and the fused image is output; the invention can balance the luminosity among all views in the panoramic synthetic image under the condition that the colors of the panoramic synthetic image are not distorted, and avoid obvious boundaries between adjacent views.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to the structures shown in these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a method for photometric equalization of a vehicle-mounted 360-degree panoramic image in one implementation of the present invention.
FIG. 2 is a flow chart of the overall scheme in one implementation of the invention.
FIG. 3 is a schematic diagram of an overlap region in one implementation of the invention.
Fig. 4 is a functional schematic of a terminal in one implementation of the invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more clear and clear, the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Exemplary method
In the panoramic synthesis mode of the existing panoramic camera system, one scheme is to directly splice the aerial views obtained under all view angles according to a certain geometric relationship, and the direct hard splicing scheme can cause inadaptation in visual observation of users because the hard splicing can generate an obvious dividing line; another approach is tone map based, which has the disadvantage that the gain and bias of the generated look-up table are locally optimal solutions at the calibration time, and often cannot cover various complex illumination environments faced by the actual product when in use, for example, the illumination of the light source of the calibration room is relatively average, so that the gain and bias obtained are not applicable in the scene of strong sunlight illumination in daytime or the scene at night, and some colors of the view are changed, resulting in color distortion of the picture.
Aiming at the technical problems, the embodiment of the invention provides a luminosity balancing method of a vehicle-mounted 360-degree panoramic image, which comprises the steps of selecting an overlapping region in a bird's eye view image, establishing a cost function according to the overlapping region, obtaining compensation parameters corresponding to each overlapping region by solving the cost function, compensating the overlapping region according to the obtained compensation parameters, fusing all the overlapping regions according to a region fusion algorithm, and outputting a fused image; therefore, the embodiment of the invention can balance the luminosity among all views in the panoramic synthetic image under the condition that the colors of the panoramic synthetic image are not distorted, and avoid obvious boundaries between adjacent views.
As shown in fig. 1, an embodiment of the present invention provides a luminosity balancing method for a vehicle-mounted 360-degree panoramic image, including the following steps:
step S100, a multi-view camera image is obtained, and the multi-view camera image is subjected to view conversion to obtain a bird' S-eye view image.
In one implementation manner of this embodiment, an image of the multi-view camera is used as input, the image is converted from RGB (i.e., a color standard, where RGB represents colors of three channels of red, green and blue) to HSV (i.e., a color space, where HSV color space refers to a visible photon set in a H, S, V three-dimensional color space), an overlapping area under the aerial view of each view is found, a simpler and interpretable cost function is established with the mean and variance of the S channel and the V channel of the overlapping area under each view as luminosity attributes, the corresponding gains and deviations are obtained, the gains and deviations are applied to the S channel and the V channel of each pixel of the aerial view original image under each view, and then after the HSV image is converted back to the RGB image, a pixel fusion operation is performed on the overlapping area of the aerial view under each view to complete luminosity equalization.
In another implementation manner of this embodiment, an image of a multi-view camera in YUV (YUV is a type of compiled color space) format is used as input, an overlapping region under an aerial view of each view is found, a mean value and a variance of a Y channel of the aerial view overlapping region under each view are used as brightness attributes, a cost function is established, corresponding gains and deviations are obtained, the gains and the deviations are applied to each pixel point of an aerial view original image under each view, and a pixel point fusion operation is performed on the overlapping region of the aerial view under each view, so that an image balancing process is completed. Of course, for images of multi-view cameras in other formats, the process of image equalization of the 360-degree panoramic aerial view can be completed by establishing a corresponding cost function and obtaining corresponding gains and deviations.
Specifically, in one implementation of the present embodiment, step S100 includes the steps of:
step S101, establishing a virtual camera with customized inner parameters and outer parameters, and projecting the multi-view camera image to the view angle of the virtual camera;
step S102, or according to the size of the defined aerial view picture, calculating an affine matrix by using an algorithm for solving the 3D-to-2D point-to-motion, and converting each pixel of the multi-view camera image into the aerial view.
In this embodiment, the input of the vehicle-mounted 360-degree panoramic image is a multi-view camera image; the multi-view camera image is an original image of four or six fish-eye cameras or a panoramic camera, and is an image in RGB format, or can be an image in YUV format or an image in any other storage format; it should be noted that the fisheye camera is slightly different from the FOV (field angle) and the de-distortion algorithm of the panoramic camera, and a different manner is required for the viewing angle conversion process of the fisheye camera and the panoramic camera.
In this embodiment, the viewing angle of the original pictures of the four or six fisheye cameras or the original pictures of the peri-view cameras is required to be converted into the aerial view by a virtual camera or an aerial view image; as shown in fig. 2, the original pictures of the four fisheye cameras or the panoramic camera can be converted into four aerial views by means of view angle conversion; the virtual camera is used for generating a virtual environment by using a computer and taking pictures by using the virtual environment; in the interface for establishing the virtual camera, the required internal parameters and external parameters of the virtual camera can be set in a self-defined mode, and the internal parameters and the external parameters set in the self-defined mode can be used for carrying out visual angle conversion in the follow-up mode.
In this embodiment, the viewing angle conversion step is essentially a projective transformation process of a picture, and may be accomplished in any manner; as an example, a custom virtual camera with internal and external parameters may be used to project the original pictures of multiple views to the virtual camera view angle, or a calibrated scheme may be adopted to define the size of the aerial view picture, an affine matrix may be obtained by using an algorithm for solving the 3D to 2D point-to-motion (i.e., PNP algorithm), and each pixel on the original pictures of each view angle is converted to the aerial view angle, where the result of this step is that the original pictures of four or six fisheye cameras or panoramic cameras are projected to the aerial view angle.
As shown in fig. 1, in an implementation manner of the embodiment of the present invention, the photometric equalizing method for the vehicle-mounted 360-degree panoramic image further includes the following steps:
step S200, selecting an overlapping area in the aerial view image, and establishing a cost function according to the overlapping area.
In this embodiment, if the input image is not an HSV format image, the bird's eye view image obtained in the previous step needs to be converted into an HSV format image, so as to complete the conversion process of the image format; of course, for the original image with the image format of YUV, the overlapping area in the aerial view image obtained in the last step can be directly selected without the conversion process of the image format.
Specifically, in one implementation of the present embodiment, the step S200 includes the following steps before:
and step S201a, performing image format conversion on the aerial view image to obtain a converted aerial view image.
In this embodiment, when the bird's-eye view image is subjected to image format conversion, an image in which the converted bird's-eye view image is in HSV format is obtained; as one example, the image may be converted from RGB onto HSV channels directly by a corresponding conversion tool or program, or other formats onto HSV channels.
For the above-mentioned HSV format bird 'S eye view obtained by conversion in step S201a, in this embodiment, an overlapping area of the HSV format bird' S eye view is selected to establish a cost function corresponding to the HSV format, so as to perform photometric equalization according to the cost function.
Specifically, in one implementation of the present embodiment, step S200 includes the steps of:
step S201, selecting an overlapping region in the converted aerial view image, and determining a region of interest according to the overlapping region; wherein the region of interest is the whole or part of the overlapping region;
step S202, calculating the mean and variance of the first channel and the second channel of the color space of each region of interest, and establishing the cost function according to the calculated mean and variance.
In this embodiment, after the completion of the viewing angle conversion, the obtained bird's eye view at each viewing angle is a bird's eye view having an overlapping region.
In this embodiment, the overlapping area is found out, and a region of interest (ROI) is selected, where the region of interest may be the overlapping area itself directly or may be a part of the overlapping area; when a part of the overlapping area is selected as the interested area, the method can be set according to empirical data, for example, the method can be selected according to a section from 100% overlapping area to 50% overlapping area, sampling is performed in a mode of equal interval of 10%, and the optimal interested area selection ratio corresponding to different panoramic camera systems can be determined according to sampling samples and experimental results.
In this embodiment, if the original image has four viewing angles in total, 8 such overlapping areas are generated, because two overlapping areas, that is, overlapping areas of the front view and the left view, are generated between every two views, and overlapping areas of the left view and the front view; similarly, for an artwork at six viewing angles, 12 overlapping areas would result.
As an example, as shown in fig. 3, taking four views as an example in the present embodiment, the overlapping area may be expressed as: overlay_fl (left front first Overlap region), overlay_lf (left front second Overlap region), overlay_lb (left rear first Overlap region), overlay_bl (left rear second Overlap region), overlay_br (right rear first Overlap region), overlay_rb (right rear second Overlap region), overlay_rf (right front first Overlap region), overlay_fr (right front second Overlap region); where f in the subscript denotes front (i.e., front view), b denotes back (i.e., rear view), l denotes left (i.e., left view), and r denotes right (i.e., right view).
For the HSV format bird 'S-eye views converted in step S201a, in this embodiment, the photometric properties of the bird' S-eye views are calculated first, and the photometric properties used in this embodiment are the mean and variance of the S-channel (i.e., the first channel) and the V-channel (the second channel) of the region of interest of each overlapping region.
After the luminosity attribute is obtained through calculation, a cost function is established, the following formula is adopted as the cost function, the cost function is essentially that the difference between the mean value and the variance of the S channel and the V channel of the aerial view under each view angle is expected to be minimum, and the four view angles of the front left view, the front right view, the rear left view and the rear right view are taken as examples, wherein the cost function is as follows:
，
，
where g represents gain, b represents bias, and m represents mean, where is the mean of the S-channel and V-channel of the bird' S eye view at each view angle; similarly, δ represents the standard deviation of the S channel and the V channel of the bird' S eye view under each viewing angle, the subscripts i, j represent each viewing angle, and taking four viewing angles as an example, i refers to four front, rear, left and right viewing angles, i.e., F (front, right), B (back, rear), L (left, left viewing angle), FR (right front viewing angle), RB (right rear viewing angle), BL (left rear viewing angle), LF (left front viewing angle) are viewing angle pairs, i.e., the front view forms an overlapping region with the left and right views, and the rear view also forms an overlapping region with the left and right views.
In this embodiment, for the bird's-eye view image not subjected to image format conversion, if the bird's-eye view image is in YUV format, an overlapping region in the bird's-eye view image in the source format may be directly selected, and the region of interest may be determined according to the overlapping region.
Specifically, in another implementation manner of the present embodiment, step S200 includes the following steps:
step S203, selecting an overlapping region in the aerial view image in the source format, and determining an interested region according to the overlapping region;
step S204, calculating the mean and variance of the third channel of the color space of each region of interest, and establishing the cost function according to the calculated mean and variance.
For the bird's eye view in YUV format, the method of selecting the region of interest (ROI) is the same as the method of the bird's eye view in HSV format; therefore, for the selected region of interest, the mean and variance of the Y channel (i.e., the third channel) are calculated, and then a corresponding cost function can be established according to the mean and variance of the Y channel.
As shown in fig. 1, in an implementation manner of the embodiment of the present invention, the photometric equalizing method for the vehicle-mounted 360-degree panoramic image further includes the following steps:
and step S300, solving the cost function to obtain compensation parameters corresponding to each overlapping region.
In this embodiment, the compensation parameters corresponding to each overlapping region include gain and deviation; for the established cost function of the HSV format aerial view or the cost function of the YUV format aerial view, the gain and the deviation can be solved by using a singular value decomposition mode, so that luminosity compensation is carried out on the aerial view under each view angle according to the solved gain and deviation.
Specifically, in one implementation of the present embodiment, step S300 includes the steps of:
step S301, the cost functions are combined and converted into a matrix form;
step S302, solving the gain according to the singular value decomposition mode, and substituting the solved gain into a cost function with deviation;
and step S303, solving and obtaining deviation according to the singular value decomposition mode.
In this embodiment, the cost function of the established HSV format aerial view or the cost function of the YUV format aerial view are combined and converted into a matrix form; then, the gain (gain) can be first solved using SVD (singular value decomposition), and then the gain is substituted into the cost function with the bias (bias), and the bias can be solved using SVD (singular value decomposition).
As shown in fig. 1, in an implementation manner of the embodiment of the present invention, the photometric equalizing method for the vehicle-mounted 360-degree panoramic image further includes the following steps:
and step S400, compensating the corresponding overlapping areas according to the obtained compensation parameters, fusing all the overlapping areas according to an area fusion algorithm, and outputting fused images.
Specifically, in one implementation of the present embodiment, step S400 includes the following steps:
step S401, compensating the corresponding overlapping area according to the gain and the deviation obtained by solving;
step S402, fusing all the overlapped areas with different weights according to the area fusion algorithm;
step S403, splicing the aerial views under each view angle, and outputting 360-degree looking-around images.
In this embodiment, after solving the cost function, luminosity compensation is performed on the aerial view under each view angle: that is, after the cost function of the aerial view of the HSV format is solved, four gains (taking four view angles as an example) and deviations can be obtained, and the gains and deviations respectively act on the whole aerial view of the HSV format under each view angle. And for the cost function of the bird's-eye view in YUV format, the obtained gain and deviation are respectively acted on the whole bird's-eye view in each view angle of the YUV format.
For example: front=front x gain_front+bias_front; where Front represents the Front view, gain_front represents the Front view gain, bias_front represents the Front view bias.
For the bird's eye views in HSV format, each bird's eye view needs to be converted back to RGB or any previous image format (if the image format is not HSV at the outset); in the case of a bird's eye view in YUV format, there is no need to convert the image format.
After luminosity compensation is carried out on the aerial view under each view angle, all overlapping areas can be fused; in the process of fusing the overlapping regions, an algorithm for fusing any region can be used, for example, a weighted average method based on distance or an alpha fusion algorithm (an image fusion algorithm), and each overlapping region can be overlapped together with different weights through any algorithm.
After the overlapping areas are fused, the bird's eye views under all the visual angles are spliced together, and then 360-degree looking-around images can be output.
The operations in the photometric equalizing method in this embodiment are operations performed for each frame of original image, so that various complex scenes can be dealt with; meanwhile, as the cost function is simpler, the solving speed is high, the requirement on calculation force is not high, and the processing efficiency of luminosity balance can be improved; finally, the phenomenon of color distortion does not occur due to the photometric equalization operation completed in the S and V channels.
It is worth mentioning that, for the bird' S eye view of the converted HSV format, the above photometric equalization operation is mainly completed on the S-channel and the V-channel; for the bird's eye view in YUV format, the luminosity balancing operation is mainly completed on the Y channel, and the luminosity attribute of the image can be adjusted and can be used without affecting the luminosity attribute of the color (without distortion); meanwhile, the cost function of the HSV format or the cost function of the YUV format can be adaptively changed according to requirements, for example, only one gain can be solved to save calculation force.
The following technical effects are achieved through the technical scheme:
according to the embodiment, by selecting the overlapping areas in the aerial view image and establishing a cost function according to the overlapping areas, compensation parameters corresponding to the overlapping areas can be obtained by solving the cost function, so that the overlapping areas can be compensated according to the obtained compensation parameters, all the overlapping areas are fused according to an area fusion algorithm, and the fused image is output; according to the embodiment, under the condition that the colors of the panoramic synthetic image are not distorted, the luminosity among all views in the panoramic synthetic image can be balanced, and obvious boundaries between adjacent views are avoided.
Exemplary apparatus
Based on the above embodiment, the present invention further provides a luminosity balancing device for a vehicle-mounted 360-degree panoramic image, including:
the aerial view module is used for acquiring a multi-view camera image, and performing view conversion on the multi-view camera image to obtain an aerial view image;
the cost function module is used for selecting an overlapping area in the aerial view image and establishing a cost function according to the overlapping area;
the compensation parameter module is used for solving the cost function to obtain compensation parameters corresponding to each overlapping region;
and the compensation and output module is used for compensating the corresponding overlapping areas according to the obtained compensation parameters, fusing all the overlapping areas according to the area fusion algorithm and outputting the fused image.
The following technical effects are achieved through the technical scheme:
according to the embodiment, by selecting the overlapping areas in the aerial view image and establishing a cost function according to the overlapping areas, compensation parameters corresponding to the overlapping areas can be obtained by solving the cost function, so that the overlapping areas can be compensated according to the obtained compensation parameters, all the overlapping areas are fused according to an area fusion algorithm, and the fused image is output; according to the embodiment, under the condition that the colors of the panoramic synthetic image are not distorted, the luminosity among all views in the panoramic synthetic image can be balanced, and obvious boundaries between adjacent views are avoided.
Based on the above embodiment, the present invention also provides a terminal, and a functional block diagram thereof may be shown in fig. 4.
The terminal comprises: the system comprises a processor, a memory, an interface, a display screen and a communication module which are connected through a system bus; wherein the processor of the terminal is configured to provide computing and control capabilities; the memory of the terminal comprises a storage medium and an internal memory; the storage medium stores an operating system and a computer program; the internal memory provides an environment for the operation of the operating system and computer programs in the storage medium; the interface is used for connecting with external equipment; the display screen is used for displaying corresponding information; the communication module is used for communicating with a cloud server or other devices.
The computer program when executed by the processor is configured to perform operations of a method for photometric equalization of a 360 degree panoramic image on board a vehicle.
It will be appreciated by those skilled in the art that the functional block diagram shown in fig. 4 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the terminal to which the present inventive arrangements may be applied, and that a particular terminal may include more or less components than those shown, or may combine some of the components, or have a different arrangement of components.
In one embodiment, a terminal is provided, including: the system comprises a processor and a memory, wherein the memory stores a luminosity balancing program of the vehicle-mounted 360-degree panoramic image, and the luminosity balancing program of the vehicle-mounted 360-degree panoramic image is used for realizing the operation of the luminosity balancing method of the vehicle-mounted 360-degree panoramic image when being executed by the processor.
In one embodiment, a storage medium is provided, wherein the storage medium stores a photometric equalization program for an on-board 360-degree panoramic image, which when executed by a processor is operative to implement the photometric equalization method for an on-board 360-degree panoramic image as above.
Those skilled in the art will appreciate that implementing all or part of the above-described methods may be accomplished by way of a computer program comprising instructions for the relevant hardware, the computer program being stored on a non-volatile storage medium, the computer program when executed comprising the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory.
In summary, the invention provides a method and a device for equalizing luminosity of a vehicle-mounted 360-degree panoramic image, wherein the method comprises the following steps: acquiring a multi-view camera image, and performing view conversion on the multi-view camera image to obtain a bird's eye view image; selecting an overlapping region in the aerial view image, and establishing a cost function according to the overlapping region; solving the cost function to obtain compensation parameters corresponding to each overlapping region; and compensating the corresponding overlapping regions according to the obtained compensation parameters, fusing all the overlapping regions according to a region fusion algorithm, and outputting fused images. The invention solves the problem that obvious boundaries exist between adjacent views due to unbalance of luminosity among the views in the panoramic synthetic image.
It is to be understood that the invention is not limited in its application to the examples described above, but is capable of modification and variation in light of the above teachings by those skilled in the art, and that all such modifications and variations are intended to be included within the scope of the appended claims.
Claims (10)
1. A luminosity balancing method of a vehicle-mounted 360-degree panoramic image is characterized by comprising the following steps:
acquiring a multi-view camera image, and performing view conversion on the multi-view camera image to obtain a bird's eye view image;
selecting an overlapping region in the aerial view image, and establishing a cost function according to the overlapping region;
solving the cost function to obtain compensation parameters corresponding to each overlapping region;
and compensating the corresponding overlapping regions according to the obtained compensation parameters, fusing all the overlapping regions according to a region fusion algorithm, and outputting fused images.
2. The method for luminance balancing of a vehicle-mounted 360-degree panoramic image according to claim 1, wherein performing view conversion on the multi-view camera image to obtain a bird's eye view image comprises:
establishing a virtual camera with customized inner parameters and outer parameters, and projecting the multi-view camera image to the view angle of the virtual camera;
or according to the size of the defined aerial view angle picture, calculating an affine matrix by an algorithm for solving the 3D-2D point-to-motion, and converting each pixel of the multi-view camera image into the aerial view angle.
3. The method for photometric equalization of a vehicle-mounted 360-degree panoramic image according to claim 1, wherein said selecting an overlapping region in said bird's eye view image comprises, before:
and performing image format conversion on the aerial view image to obtain a converted aerial view image.
4. The method for photometric equalization for a vehicle-mounted 360-degree panoramic image according to claim 3, wherein said selecting an overlapping region in said bird's eye view image and establishing a cost function according to said overlapping region comprises:
selecting an overlapping region in the converted aerial view image, and determining an interested region according to the overlapping region; wherein the region of interest is the whole or part of the overlapping region;
and calculating the mean and variance of the first channel and the second channel of the color space of each region of interest, and establishing the cost function according to the calculated mean and variance.
5. The method for photometric equalization of a vehicle-mounted 360-degree panoramic image according to claim 1, wherein said selecting an overlapping region in said bird's eye view image and establishing a cost function according to said overlapping region further comprises:
selecting an overlapping region in the aerial view image in the source format, and determining an interested region according to the overlapping region;
and calculating the mean and variance of the third channel of the color space of each region of interest, and establishing the cost function according to the calculated mean and variance.
6. The method for photometric equalization of a vehicle-mounted 360-degree panoramic image of claim 1, wherein said compensation parameters comprise: gain and bias;
the step of solving the cost function to obtain compensation parameters corresponding to each overlapping area comprises the following steps:
the cost functions are combined and converted into a matrix form;
solving the gain according to the singular value decomposition mode, and substituting the solved gain into a cost function with deviation;
and solving according to the singular value decomposition mode to obtain deviation.
7. The method for photometric equalization of a vehicle-mounted 360-degree panoramic image according to claim 6, wherein compensating the corresponding overlapping regions according to the obtained compensation parameters, fusing all the overlapping regions according to a region fusion algorithm, and outputting the fused image, comprises:
compensating the corresponding overlapping region according to the gain and the deviation obtained by solving;
fusing all the overlapped areas with different weights according to the area fusion algorithm;
and splicing the aerial views under all the visual angles, and outputting 360-degree looking-around images.
8. The utility model provides a on-vehicle 360 degrees panoramic image's luminosity equalizer, its characterized in that includes:
the aerial view module is used for acquiring a multi-view camera image, and performing view conversion on the multi-view camera image to obtain an aerial view image;
the cost function module is used for selecting an overlapping area in the aerial view image and establishing a cost function according to the overlapping area;
the compensation parameter module is used for solving the cost function to obtain compensation parameters corresponding to each overlapping region;
and the compensation and output module is used for compensating the corresponding overlapping areas according to the obtained compensation parameters, fusing all the overlapping areas according to the area fusion algorithm and outputting the fused image.
9. A terminal, comprising: the system comprises a processor and a memory, wherein the memory stores a luminosity balancing program of the vehicle-mounted 360-degree panoramic image, and the luminosity balancing program of the vehicle-mounted 360-degree panoramic image is used for realizing the operation of the luminosity balancing method of the vehicle-mounted 360-degree panoramic image according to any one of claims 1-7 when being executed by the processor.
10. A computer readable storage medium, wherein the computer readable storage medium stores a luminosity balancing program of a vehicle-mounted 360-degree panoramic image, which when executed by a processor is configured to implement operations of the luminosity balancing method of a vehicle-mounted 360-degree panoramic image as claimed in any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311501652.9A CN117237237B (en) | 2023-11-13 | 2023-11-13 | Luminosity balancing method and device for vehicle-mounted 360-degree panoramic image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311501652.9A CN117237237B (en) | 2023-11-13 | 2023-11-13 | Luminosity balancing method and device for vehicle-mounted 360-degree panoramic image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117237237A true CN117237237A (en) | 2023-12-15 |
| CN117237237B CN117237237B (en) | 2024-07-05 |
Family
ID=89093262
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311501652.9A Active CN117237237B (en) | 2023-11-13 | 2023-11-13 | Luminosity balancing method and device for vehicle-mounted 360-degree panoramic image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117237237B (en) |
Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130215269A1 (en) * | 2012-02-22 | 2013-08-22 | Yi-Lin Hsieh | Bird view system and compensating method thereof |
| US20150138312A1 (en) * | 2013-11-18 | 2015-05-21 | Texas Instruments Incorporated | Method and apparatus for a surround view camera system photometric alignment |
| US20160379074A1 (en) * | 2015-06-25 | 2016-12-29 | Appropolis Inc. | System and a method for tracking mobile objects using cameras and tag devices |
| CN106373091A (en) * | 2016-09-05 | 2017-02-01 | 山东省科学院自动化研究所 | Automatic panorama parking aerial view image splicing method, system and vehicle |
| US20170195560A1 (en) * | 2015-12-31 | 2017-07-06 | Nokia Technologies Oy | Method and apparatus for generating a panoramic view with regions of different dimensionality |
| CN109948398A (en) * | 2017-12-20 | 2019-06-28 | 深圳开阳电子股份有限公司 | The image processing method and panorama parking apparatus that panorama is parked |
| CN110288527A (en) * | 2019-06-24 | 2019-09-27 | 北京智行者科技有限公司 | The vehicle-mounted camera panorama of looking around of one kind gets a bird's eye view drawing generating method |
| WO2020029877A1 (en) * | 2018-08-10 | 2020-02-13 | 深圳岚锋创视网络科技有限公司 | Brightness adjustment method for multi-camera mosaicking, and portable terminal |
| US20200280684A1 (en) * | 2017-10-26 | 2020-09-03 | Harman International Industries, Incorporated | Method and system of fast image blending for overlapping region in surround view |
| CN113421183A (en) * | 2021-05-31 | 2021-09-21 | 中汽数据(天津)有限公司 | Method, device and equipment for generating vehicle panoramic view and storage medium |
| CN113658058A (en) * | 2021-07-22 | 2021-11-16 | 武汉极目智能技术有限公司 | Brightness balancing method and system in vehicle-mounted all-round system |
| CN113689368A (en) * | 2020-05-18 | 2021-11-23 | 上海赫千电子科技有限公司 | Automatic illumination consistency adjusting method applied to vehicle-mounted all-around image |
| CN113781317A (en) * | 2021-08-02 | 2021-12-10 | 华南理工大学 | Brightness equalization method of panoramic all-around system |
| CN115170689A (en) * | 2022-07-27 | 2022-10-11 | 斑马网络技术有限公司 | Image color adjusting method and device, electronic equipment and medium |
| CN115456898A (en) * | 2022-09-09 | 2022-12-09 | 雄狮汽车科技(南京)有限公司 | Method and device for building image of parking lot, vehicle and storage medium |
| CN115861080A (en) * | 2023-02-24 | 2023-03-28 | 杭州枕石智能科技有限公司 | Splicing method of panoramic transparent vehicle bottom images and terminal equipment |
| CN116630915A (en) * | 2023-04-28 | 2023-08-22 | 浙江大学 | Reverse order layered aerial view reasoning method with relationship-enhanced attention |
-
2023
- 2023-11-13 CN CN202311501652.9A patent/CN117237237B/en active Active
Patent Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130215269A1 (en) * | 2012-02-22 | 2013-08-22 | Yi-Lin Hsieh | Bird view system and compensating method thereof |
| US20150138312A1 (en) * | 2013-11-18 | 2015-05-21 | Texas Instruments Incorporated | Method and apparatus for a surround view camera system photometric alignment |
| US20160379074A1 (en) * | 2015-06-25 | 2016-12-29 | Appropolis Inc. | System and a method for tracking mobile objects using cameras and tag devices |
| US20170195560A1 (en) * | 2015-12-31 | 2017-07-06 | Nokia Technologies Oy | Method and apparatus for generating a panoramic view with regions of different dimensionality |
| CN106373091A (en) * | 2016-09-05 | 2017-02-01 | 山东省科学院自动化研究所 | Automatic panorama parking aerial view image splicing method, system and vehicle |
| US20200280684A1 (en) * | 2017-10-26 | 2020-09-03 | Harman International Industries, Incorporated | Method and system of fast image blending for overlapping region in surround view |
| CN109948398A (en) * | 2017-12-20 | 2019-06-28 | 深圳开阳电子股份有限公司 | The image processing method and panorama parking apparatus that panorama is parked |
| WO2020029877A1 (en) * | 2018-08-10 | 2020-02-13 | 深圳岚锋创视网络科技有限公司 | Brightness adjustment method for multi-camera mosaicking, and portable terminal |
| CN110288527A (en) * | 2019-06-24 | 2019-09-27 | 北京智行者科技有限公司 | The vehicle-mounted camera panorama of looking around of one kind gets a bird's eye view drawing generating method |
| CN113689368A (en) * | 2020-05-18 | 2021-11-23 | 上海赫千电子科技有限公司 | Automatic illumination consistency adjusting method applied to vehicle-mounted all-around image |
| CN113421183A (en) * | 2021-05-31 | 2021-09-21 | 中汽数据(天津)有限公司 | Method, device and equipment for generating vehicle panoramic view and storage medium |
| CN113658058A (en) * | 2021-07-22 | 2021-11-16 | 武汉极目智能技术有限公司 | Brightness balancing method and system in vehicle-mounted all-round system |
| CN113781317A (en) * | 2021-08-02 | 2021-12-10 | 华南理工大学 | Brightness equalization method of panoramic all-around system |
| CN115170689A (en) * | 2022-07-27 | 2022-10-11 | 斑马网络技术有限公司 | Image color adjusting method and device, electronic equipment and medium |
| CN115456898A (en) * | 2022-09-09 | 2022-12-09 | 雄狮汽车科技(南京)有限公司 | Method and device for building image of parking lot, vehicle and storage medium |
| CN115861080A (en) * | 2023-02-24 | 2023-03-28 | 杭州枕石智能科技有限公司 | Splicing method of panoramic transparent vehicle bottom images and terminal equipment |
| CN116630915A (en) * | 2023-04-28 | 2023-08-22 | 浙江大学 | Reverse order layered aerial view reasoning method with relationship-enhanced attention |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117237237B (en) | 2024-07-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10523865B2 (en) | Three dimensional rendering for surround view using predetermined viewpoint lookup tables | |
| US7570280B2 (en) | Image providing method and device | |
| US10007853B2 (en) | Image generation device for monitoring surroundings of vehicle | |
| JP5239326B2 (en) | Image signal processing apparatus, image signal processing method, image projection system, image projection method and program | |
| CN109246416A (en) | The panorama mosaic method of vehicle-mounted six road camera | |
| CN112224132A (en) | Vehicle panoramic all-around obstacle early warning method | |
| CN102045546A (en) | Panoramic parking assist system | |
| US20200388005A1 (en) | Enhanced rendering of surround view images | |
| CN113421183B (en) | Method, device and equipment for generating vehicle panoramic view and storage medium | |
| CN112348741A (en) | Panoramic image splicing method, panoramic image splicing equipment, storage medium, display method and display system | |
| CN115063455A (en) | Method and device for calibrating imaging of double-optical camera and double-optical camera | |
| CN111798375A (en) | All-round-view image splicing method, system, device, storage medium and mixer truck | |
| CN111294580B (en) | Camera video projection method, device and equipment based on GPU and storage medium | |
| TWI505203B (en) | Image processing method and image processing apparatus for generating vehicular image | |
| CN113542463B (en) | Video shooting device and method based on folding screen, storage medium and mobile terminal | |
| CN110428361A (en) | A kind of multiplex image acquisition method based on artificial intelligence | |
| CN117237237B (en) | Luminosity balancing method and device for vehicle-mounted 360-degree panoramic image | |
| CN111263951A (en) | Method and system for fast image fusion of overlapping regions in panoramic views | |
| CN116894773A (en) | Image processing method, device, electronic equipment and storage medium | |
| US12080075B2 (en) | Method for generating an image of vehicle surroundings, and apparatus for generating an image of vehicle surroundings | |
| Kim et al. | Hemispherical 3D around view monitoring algorithm using image synthesis of multi-channel cameras | |
| US20050030380A1 (en) | Image providing apparatus, field-of-view changing method, and computer program product for changing field-of-view | |
| CN113016011A (en) | Augmented reality system and method for substrates, coated articles, insulated glass units, and the like | |
| US20220215668A1 (en) | Method for generating an image of vehicle surroundings, and apparatus for generating an image of vehicle surroundings | |
| CN112698717B (en) | Local image processing method and device, vehicle-mounted system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |