Disclosure of Invention
In view of the above, the present invention provides an image defogging method and system, which not only can simplify the complexity of the image defogging operation process, but also can fully exert the computing advantages of hardware resources to achieve a better defogging effect.
In a first aspect, an embodiment of the present invention provides an image defogging method, which is applied to an image defogging system and includes an image processing chip and a data processor, where the data processor is configured with an image defogging algorithm, and the method includes: reading a previous frame image corresponding to an image to be processed through the image processing chip, and generating a gray image and a dark channel image corresponding to the previous frame image; calculating, by the data processor, a first defogging parameter based on the image defogging algorithm, the grayscale image, and the dark channel image; and reading the first defogging parameter and the image to be processed through the image processing chip, and performing defogging processing on the image to be processed according to the first defogging parameter to obtain a target image.
In one embodiment, the step of generating the grayscale image and the dark channel image corresponding to the previous frame image further includes: starting a first thread to perform first down-sampling processing on the previous frame image to obtain a first down-sampled image, and converting the first down-sampled image into a gray image; and starting a second thread to carry out second down-sampling processing on the previous frame image to obtain a second down-sampled image, and calculating a dark channel image based on the second down-sampled image.
In one embodiment, the image defogging system further comprises a caching unit; the method further comprises the following steps: and storing the gray-scale image and the dark channel image to the cache unit through the image processing chip.
In one embodiment, the step of calculating a first defogging parameter based on the image defogging algorithm, the grayscale image and the dark channel image comprises: reading the gray image and the dark channel image from the buffer unit; calculating a grayscale mapping value based on the image defogging algorithm and the grayscale image, and calculating a dark channel mapping value based on the image defogging algorithm and the dark channel image; wherein the first defogging parameter includes a grayscale mapping value and a dark channel mapping value.
In one embodiment, the method further comprises: storing, by the data processor, the first defogging parameter to the cache unit; the step of reading the first defogging parameter through the image processing chip includes: and reading the first defogging parameter from the cache unit through the image processing chip.
In an embodiment, the step of performing defogging processing on the image to be processed according to the first defogging parameter to obtain a target image includes: calculating a second defogging parameter corresponding to each pixel point in the image to be processed according to the first defogging parameter; and carrying out defogging processing on the image to be processed according to the second defogging parameter corresponding to each pixel point to obtain a target image.
In one embodiment, the step of calculating a second defogging parameter corresponding to each pixel point in the image to be processed according to the first defogging parameter includes: carrying out up-sampling processing on the first defogging parameters, and determining the first defogging parameter corresponding to each pixel point in the image to be processed; and searching a second defogging parameter corresponding to each pixel point according to the first defogging parameter corresponding to each pixel point.
In an embodiment, the step of performing defogging processing on the image to be processed according to the second defogging parameter corresponding to each pixel point to obtain the target image includes: acquiring a fog intensity parameter corresponding to the image to be processed; and carrying out defogging treatment on the image to be processed based on the fog intensity parameter and a second defogging parameter corresponding to each pixel point to obtain a target image.
In one embodiment, before searching for the second defogging parameter corresponding to each pixel point according to the first defogging parameter corresponding to each pixel point, the method further includes: and carrying out alignment processing on the image to be processed and the first defogging parameter.
In a second aspect, an embodiment of the present invention further provides an image defogging system, including an image processing chip and a data processor, where the data processor is configured with an image defogging algorithm; the image processing chip is used for reading a previous frame image corresponding to an image to be processed and generating a gray image and a dark channel image corresponding to the previous frame image; the data processor is used for calculating a first defogging parameter based on the image defogging algorithm, the grayscale image and the dark channel image; the image processing chip is further used for reading the first defogging parameter and the image to be processed, and performing defogging processing on the image to be processed according to the first defogging parameter to obtain a target image.
The image defogging method and the image defogging system are applied to an image defogging system and comprise an image processing chip and a data processor, wherein the data processor is configured with an image defogging algorithm, firstly, a previous frame image corresponding to an image to be processed is read through the image processing chip, a gray level image and a dark channel image corresponding to the previous frame image are generated, then, a first defogging parameter is calculated through the data processor based on the image defogging algorithm, the gray level image and the dark channel image, finally, the first defogging parameter and the image to be processed are read through the image processing chip, and defogging processing is carried out on the image to be processed according to the first defogging parameter, so that a target image is obtained. The method provides a method for realizing image defogging by combining hardware and software, the first defogging parameter is calculated by utilizing an image defogging algorithm configured by a data processor, and then the image processing chip is utilized to exert the hardware advantage and perform defogging treatment on the image to be processed based on the first defogging parameter, so that a target image with better defogging effect can be obtained.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the embodiments, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
At present, the image defogging scheme based on the software algorithm has the problem of complex operation process, and the image defogging scheme based on the hardware has the problem of high image defogging cost.
To facilitate understanding of the present embodiment, first, a detailed description is given to an image defogging method disclosed in the present embodiment, where the method is applied to an image defogging system including an image processing chip and a data processor, the image processing chip may be referred to as a hardware side for short, the data processor (CPU) may be referred to as a software side for short, and the data processor is configured with an image defogging algorithm, and referring to a flow diagram of the image defogging method shown in fig. 1, the method mainly includes the following steps S102 to S106:
step S102, reading a previous frame image corresponding to the image to be processed through an image processing chip, and generating a gray image and a dark channel image corresponding to the previous frame image. Optionally, the video acquired in the haze environment may be used as a video to be processed, and each frame of image in the video may be used as an image to be processed. The gray image is used for representing gray information of a previous frame image, and the dark channel image is used for representing fog concentration information of the previous frame image. In an embodiment, the hardware side may perform two-way down-sampling processing on a previous frame image to obtain a grayscale image and a dark channel image, respectively, so as to reduce the amount of data read and written by the back-level cache.
Step S104, calculating a first defogging parameter based on the image defogging algorithm, the grayscale image and the dark channel image through the data processor. The first defogging parameter, that is, the Map mapping result, may include a grayscale mapping value and a dark channel mapping value. In one embodiment, the software side may perform iterative operations on the grayscale image and the dark channel image respectively using an image defogging algorithm to calculate a grayscale mapping value and a dark channel mapping value.
And S106, reading the first defogging parameter and the image to be processed through the image processing chip, and performing defogging processing on the image to be processed according to the first defogging parameter to obtain a target image. In an embodiment, the hardware side may perform two-way upsampling on the grayscale mapping value and the dark channel mapping value to restore the grayscale mapping value and the dark channel mapping value to a size consistent with a size of an image to be processed, so as to ensure image accuracy, then calculate a second defogging parameter (i.e., a defogging coefficient) by using the grayscale mapping value and the dark channel mapping value after the upsampling, and finally perform defogging on the image to be processed according to the second defogging parameter.
The image defogging method provided by the embodiment of the invention provides a method for realizing image defogging by combining hardware and software, wherein a first defogging parameter is calculated by utilizing an image defogging algorithm configured by a data processor, and then an image processing chip is utilized to exert the hardware advantage and perform defogging treatment on an image to be processed based on the first defogging parameter, so that a target image with better defogging effect can be obtained.
As for the foregoing step S102, an embodiment of the present invention provides an implementation manner for generating a grayscale image and a dark channel image corresponding to a previous frame of image, (1) starting a first thread to perform a first downsampling process on the previous frame of image to obtain a first downsampled image, and converting the first downsampled image into a grayscale image; (2) and starting a second thread to perform second down-sampling processing on the previous frame image to obtain a second down-sampled image, and calculating a dark channel image based on the second down-sampled image. Wherein, the down-sampling process may adopt window filtering. In one embodiment, the sampling coefficients may be pre-configured, such as filtered in a 4 × 4 window, to obtain down-sampled images, and then the first down-sampled image is converted into a grayscale image by a method such as an averaging method, a maximum-minimum averaging method, a weighted average method, a binary image method, etc., and dark channel information is extracted from the second down-sampled image and a dark channel image is generated.
In practical application, the image defogging system further comprises a cache unit, the cache unit can adopt memory cache, the image processing chip can store the grayscale image and the dark channel image into the cache unit in real time, and meanwhile, the software side interacts with the cache unit to read the grayscale image and the dark channel image.
Based on this, embodiments of the present invention provide an implementation for calculating a first defogging parameter based on an image defogging algorithm, a grayscale image and a dark channel image, which may read the grayscale image and the dark channel image from a cache unit, then calculate a grayscale mapping value based on the image defogging algorithm and the grayscale image, and calculate a dark channel mapping value based on the image defogging algorithm and the dark channel image. The image defogging algorithm may adopt a guided filter algorithm (guided filtering algorithm). In one embodiment, the software side may perform an iterative operation on the grayscale image and the dark channel image respectively by using an image defogging algorithm, so as to obtain the grayscale mapping value and the dark channel mapping value.
In a specific implementation, the software side may store the first defogging parameter in the cache unit, and optionally, may store the grayscale mapping value and the dark channel mapping value in a fixed location of the memory cache, and wait for the hardware side to read the grayscale mapping value and the dark channel mapping value from the fixed location. The hardware side can read the first defogging parameter from the cache unit when reading the image to be processed.
For the foregoing step S106, an embodiment of the present invention provides an implementation manner of performing defogging processing on an image to be processed according to a first defogging parameter to obtain a target image, where reference is made to the following steps 1 to 2:
step 1, calculating a second defogging parameter corresponding to each pixel point in the image to be processed according to the first defogging parameter. In one embodiment, the second defogging parameter corresponding to each pixel point can be determined according to the following steps 1.1 to 1.3:
step 1.1, carrying out up-sampling processing on the first defogging parameters, and determining the first defogging parameters corresponding to each pixel point in the image to be processed. In one embodiment, an upsampling parameter may be set, so that the grayscale map value and the dark channel map value are respectively upsampled according to the upsampling parameter, so that both the grayscale map value and the dark channel map value are restored to the size of the image to be processed.
And 1.2, aligning the image to be processed and the first defogging parameter. In consideration of misalignment deviation which may be caused by delay in practical application, the image to be processed and the first defogging parameter are subjected to alignment processing to eliminate the misalignment deviation. Optionally, the alignment processing may be performed on the image to be processed and the first defogging parameter based on the pixel point analysis.
And step 1.3, searching a second defogging parameter corresponding to each pixel point according to the first defogging parameter corresponding to each pixel point. In one embodiment, each pixel point corresponds to a gray mapping value and a dark channel mapping value, and the second defogging parameter corresponding to the gray mapping value and the dark channel mapping value is searched by a table lookup method. The Look-Up Table may be a Look-Up Table (Look-Up-Table) in a Look-Up Table.
And 2, carrying out defogging treatment on the image to be treated according to the second defogging parameter corresponding to each pixel point to obtain a target image. In one embodiment, the target image may be obtained by performing defogging processing on the image to be processed according to the following steps 2.1 to 2.2:
and 2.1, acquiring a fog intensity parameter corresponding to the image to be processed. In one embodiment, the fog intensity parameter may be preset.
And 2.2, carrying out defogging treatment on the image to be treated based on the fog intensity parameter and the second defogging parameter corresponding to each pixel point to obtain a target image. In one embodiment, the target image may be obtained by subtracting the fog intensity parameter from the image to be processed to obtain a difference value, multiplying the difference value by the second defogging parameter to obtain a product, and summing the product and the fog intensity parameter to obtain a sum value.
To facilitate understanding of the image defogging method provided in the above embodiment, an application example of the image defogging method is also provided in the embodiment of the present invention, and with reference to an image defogging frame shown in fig. 2, the image defogging frame includes: (1) an image down-sampling section: the image acquisition device is used for acquiring a gray image and a dark channel image of a previous frame image; (2) an image buffer section: the image processing device is used for storing a gray image and a dark channel image of a previous frame image; (3) the software calculation part: calculating the Map mapping result; (4) an image up-sampling section: the Map mapping value is used for restoring the Map mapping result to the mapping value with the size corresponding to the original image; (5) a cache alignment part: the method is used for eliminating the misalignment between the image to be processed and the coefficient; (6) the coefficient calculating section: used for running water treatment; (7) a defogging treatment part: for performing defogging processing to obtain a defogged image (i.e., the target image).
Based on the foregoing fig. 2, an embodiment of the present invention provides a schematic flow chart of another image defogging method as shown in fig. 3, which mainly includes the following steps S302 to S314:
step S302, the hardware side respectively obtains a gray image picture0 and a dark channel image picture1 after two-way down-sampling is carried out on the previous frame image picture before defogging. According to the embodiment of the invention, the down-sampling processing is carried out on the image picture of the previous frame, so that the data volume read and written by the whole rear-stage cache can be obviously reduced.
Step S304, the hardware side stores the grayscale image picture0 and the dark channel image picture1 into the memory cache in real time, and the software side reads the grayscale image picture0 and the dark channel image picture1 from the memory cache.
And S306, performing parameter operation on the grayscale image picture0 and the dark channel image picture1 by the software side to obtain two paths of Map mapping results, storing the Map mapping results to a fixed position of a memory cache, and waiting for the hardware side to read the Map mapping results. The parameter operation mainly comprises iterative operation, and the Map mapping result comprises a gray mapping value a [ i ] and a dark channel mapping value b [ i ].
Step S308, the hardware side reads the gray mapping value a [ i ] and the dark channel mapping value b [ i ] from the memory cache in the next frame, and respectively carries out up-sampling processing on the gray mapping value a [ i ] and the dark channel mapping value b [ i ] to obtain the gray mapping value a [ i-1] and the dark channel mapping value b [ i-1] corresponding to the image pic [ i-1] of the next frame. By up-sampling the gray mapping value a [ i ] and the dark channel mapping value b [ i ], the gray mapping value a [ i ] and the dark channel mapping value b [ i ] can be restored to the size consistent with the size of the original image, and therefore the image defogging precision is improved.
And S310, caching the image to be processed, the gray mapping value a [ i-1] and the dark channel mapping value b [ i-1] by the hardware side through cache, and aligning the image to be processed, the gray mapping value a [ i-1] and the dark channel mapping value b [ i-1 ]. Misalignment deviations caused by delays can be eliminated by the alignment process.
In step S312, the hardware side calculates the defogging coefficient 1/t (x) of the image to be processed in real time by using a hardware pipeline operation mode. Wherein the defogging coefficient is also the second defogging parameter. In one embodiment, 1/t (x) gray a [ i-1] + b [ i-1], gray represents a gray scale value, and the defogging coefficient 1/t (x) is determined by a LUT table look-up method in consideration of resource waste caused by calculation and comparison in hardware.
And step S314, the hardware side carries out defogging processing on the image to be processed by utilizing the defogging coefficient to obtain a target image. In one embodiment, the defogging process may be performed according to the following formula:
wherein J (x) represents a target image, I (x) represents an image to be processed, and A represents a fog intensity parameter.
In summary, the hardware-implemented image defogging architecture provided in the embodiment of the present invention performs defogging on an image to be processed in an image defogging parameter calculation manner combining hardware and software, so that the advantages of hardware parallel calculation can be fully exerted while complex iterative operation is implemented, and the image/video defogging processing is performed in a pipeline manner, thereby obtaining a better defogging effect.
As for the image defogging method provided in the foregoing embodiment, an embodiment of the present invention provides an image defogging system, referring to a schematic structural diagram of the image defogging system shown in fig. 4, where the system includes: an image processing chip 100 and a data processor 200, the data processor 200 being configured with an image defogging algorithm.
The image processing chip 100 is configured to read a previous frame image corresponding to an image to be processed, and generate a grayscale image and a dark channel image corresponding to the previous frame image;
the data processor 200 is configured to calculate a first defogging parameter based on the image defogging algorithm, the grayscale image and the dark channel image;
the image processing chip 100 is further configured to read the first defogging parameter and the image to be processed, and perform defogging processing on the image to be processed according to the first defogging parameter to obtain a target image.
The image defogging system provided by the embodiment of the invention provides a hardware and software combined image defogging method, the image defogging method is characterized in that a first defogging parameter is calculated by utilizing an image defogging algorithm configured by a data processor, and an image processing chip is utilized to exert the hardware advantage and perform defogging treatment on an image to be processed based on the first defogging parameter, so that a target image with a better defogging effect can be obtained.
In one embodiment, the image processing chip 100 is further configured to: starting a first thread to perform first down-sampling processing on a previous frame image to obtain a first down-sampled image, and converting the first down-sampled image into a gray image; and starting a second thread to carry out second down-sampling processing on the previous frame image to obtain a second down-sampled image, and calculating a dark channel image based on the second down-sampled image.
In one embodiment, the image defogging system further comprises a caching unit; the image processing chip 100 is further configured to: and storing the gray-scale image and the dark channel image into a buffer unit.
In one embodiment, the data processor 200 is further configured to: reading a gray image and a dark channel image from a buffer unit; calculating a gray mapping value based on an image defogging algorithm and a gray image, and calculating a dark channel mapping value based on the image defogging algorithm and a dark channel image; wherein the first defogging parameter includes a grayscale mapping value and a dark channel mapping value.
In one embodiment, the data processor 200 is further configured to: storing the first defogging parameter to a cache unit; the image processing chip 100 is further configured to: and reading the first defogging parameter from the cache unit.
In one embodiment, the image processing chip 100 is further configured to: calculating a second defogging parameter corresponding to each pixel point in the image to be processed according to the first defogging parameter; and carrying out defogging treatment on the image to be treated according to the second defogging parameter corresponding to each pixel point to obtain a target image.
In one embodiment, the image processing chip 100 is further configured to: carrying out up-sampling processing on the first defogging parameters, and determining the first defogging parameters corresponding to each pixel point in the image to be processed; and searching a second defogging parameter corresponding to each pixel point according to the first defogging parameter corresponding to each pixel point.
In one embodiment, the image processing chip 100 is further configured to: acquiring a fog intensity parameter corresponding to an image to be processed; and carrying out defogging treatment on the image to be treated based on the fog intensity parameter and the second defogging parameter corresponding to each pixel point to obtain a target image.
In one embodiment, the image processing chip 100 is further configured to: and carrying out alignment processing on the image to be processed and the first defogging parameter.
The system provided by the embodiment of the present invention has the same implementation principle and technical effect as the foregoing method embodiment, and for the sake of brief description, no mention is made in the system embodiment, and reference may be made to the corresponding contents in the foregoing method embodiment.
The embodiment of the invention provides electronic equipment, which particularly comprises a processor and a storage device; the storage means has stored thereon a computer program which, when executed by the processor, performs the method of any of the above described embodiments.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.