CN111435972B

CN111435972B - Image processing method and device

Info

Publication number: CN111435972B
Application number: CN201910034934.XA
Authority: CN
Inventors: 杨朝伟
Original assignee: Hangzhou Hikvision Digital Technology Co Ltd
Current assignee: Hangzhou Hikvision Digital Technology Co Ltd
Priority date: 2019-01-15
Filing date: 2019-01-15
Publication date: 2021-03-23
Anticipated expiration: 2039-01-15
Also published as: CN111435972A

Abstract

The application provides an image processing method and device, which are applied to a camera, wherein the method comprises the following steps: based on a first original preview image acquired by a color sensor, carrying out exposure adjustment on the color image sensor; based on the first original preview image and a second original preview image acquired by the black-and-white sensor at the same time, carrying out exposure adjustment on the black-and-white image sensor through an infrared light supplementing device; respectively acquiring a first effective preview image and a second effective preview image which are acquired by a color image sensor and a black-and-white image sensor after exposure adjustment, wherein the first effective preview image clearly shows road environment information, and the second effective preview image clearly shows license plate information of vehicles running on a road; and synthesizing the first effective preview image and the second effective preview image to obtain a target image for output, thereby realizing complete monitoring of a road monitoring area and improving the monitoring effect of the camera.

Description

Image processing method and device

Technical Field

The present application relates to the field of data processing technologies, and in particular, to a method and an apparatus for image processing.

Background

At present, in the field of intelligent transportation, in order to guarantee that a snapshot image can clearly identify information such as vehicle profiles and license plates, a road monitoring camera is generally matched with light supplementing equipment such as a light supplementing lamp, a strobe lamp or an exposure flashing lamp for use. With the aid of various supplementary lighting devices, the traffic camera can capture the vehicles on the road to obtain clear images.

The existing road monitoring camera mainly uses an exposure flash light supplement lamp in a visible light wave band to supplement light to perform image capture, or uses a normally-on light supplement lamp in a visible light wave band range to supplement light to a license plate so as to enable an image to obtain clear license plate or other vehicle information. However, whether the flash light supplement lamp or the highlight light supplement lamp is used, especially under the condition that the ambient brightness is dark such as an expressway, the interference on the sight line can be brought to the drivers of the motor vehicles driving in front, even the transient highlight blind influence is caused, and great driving potential safety hazards can be brought.

Disclosure of Invention

In view of the above, the present application provides a method and an apparatus for image processing.

Specifically, the method is realized through the following technical scheme:

in a first aspect, the present application provides an image processing method, which is applied to a camera, where the camera includes a black and white sensor and a color sensor, the black and white sensor is used for collecting license plate information of a vehicle running in a road monitoring area, and the color sensor is used for collecting road environment information in the road monitoring area;

the method comprises the following steps:

carrying out exposure adjustment on the color image sensor based on a first original preview image acquired by the color sensor;

based on the first original preview image and a second original preview image acquired by the black-and-white sensor at the same time, carrying out exposure adjustment on the black-and-white image sensor through an infrared light supplementing device;

respectively acquiring a first effective preview image and a second effective preview image which are acquired by the color image sensor and the black-and-white image sensor after exposure adjustment, wherein the first effective preview image clearly displays road environment information, and the second effective preview image clearly displays license plate information of vehicles running on a road;

and synthesizing the first effective preview image and the second effective preview image to obtain a target image for output.

Preferably, the exposure adjustment of the color image sensor based on the first original preview image acquired by the color sensor includes:

blocking the first original preview image according to a preset image blocking strategy to obtain a first blocking set, wherein the first blocking set comprises M first blocking images;

acquiring the brightness value of each first block image, wherein the brightness value of the first block image is the average value of the brightness values of all pixel points in the first block image;

calculating an average value of brightness values of the first block images in the first block set to obtain a brightness value of the first original preview image;

and comparing the brightness value of the first original preview image with a preset brightness range, and carrying out exposure adjustment on the color image sensor according to the comparison result.

Preferably, the performing exposure adjustment on the black-and-white image sensor through an infrared light supplement device based on the first original preview image and a second original preview image acquired by the black-and-white sensor at the same time includes:

blocking the second original preview image according to the image blocking strategy to obtain a second blocking set, wherein the second blocking set comprises M second blocking images;

acquiring the brightness value of each second block image, wherein the brightness value of each second block image is the average value of the brightness values of all pixel points in the second block image;

identifying a target license plate area of a target vehicle from the second block set according to the brightness value of each first block image and the brightness value of each second block image;

and acquiring the brightness value of the target license plate area, and controlling the infrared light supplementing device to supplement the infrared light to the black-and-white image sensor according to the brightness value of the target license plate area, wherein the brightness value of the target license plate area is the average value of the brightness values of the second block images contained in the target license plate area.

Preferably, the identifying a target license plate region of a target vehicle from the second block set according to the brightness value of each first block image and the brightness value of each second block image includes:

determining a brightness reference value according to the brightness value of the first original preview image;

for the first block set, determining the brightness value of the first block image with the brightness value smaller than the brightness reference value as a preset minimum brightness value, and determining the brightness value of the first block image with the brightness value larger than or equal to the brightness reference value as a preset maximum brightness value;

tracking the first block image with the brightness value of the preset maximum brightness value, determining the first block image with the position change and the brightness value of the preset maximum brightness value as a target first block image, and taking the area surrounded by the target first block images as a license plate area;

and identifying a second block image mapped with the first block image contained in the license plate region in a second block set of a second original preview image corresponding to the first block set and having the same first original preview image acquisition time, so as to obtain a target license plate region of the target vehicle.

Preferably, before the obtaining of the first effective preview image and the second effective preview image simultaneously acquired by the color image sensor and the black-and-white image sensor after the exposure adjustment, the method further includes:

checking whether the color image sensor outputs a first effective preview image, if not, continuing to execute the step of exposure adjustment on the color image sensor based on the first original preview image acquired by the color sensor;

and checking whether the black-and-white image sensor outputs a second effective preview image, if not, continuing to execute the step of carrying out exposure adjustment on the black-and-white image sensor through an infrared light supplementing device based on the first original preview image and the second original preview image acquired by the black-and-white sensor at the same moment.

Preferably, the synthesizing the first effective preview image and the second effective preview image to obtain a target image for output includes:

performing enhancement processing on the second effective preview image to obtain enhancement information;

and overlaying the enhanced information to the first effective preview image to generate a target image.

Preferably, the enhancement information comprises a luminance enhancement matrix;

the enhancing the second effective preview image to obtain enhanced information includes:

forming a brightness matrix by the brightness value of each pixel point in the second effective preview image;

and determining the brightness value which is greater than or equal to a preset brightness threshold value in the brightness matrix as a first brightness value, and determining the brightness value which is smaller than the preset brightness threshold value as a second brightness value to obtain a brightness enhancement matrix.

Preferably, the superimposing the enhancement information to the first effective preview image to generate a target image includes:

calculating the product of each brightness value in the brightness enhancement matrix and a first preset coefficient to obtain a first brightness enhancement matrix;

calculating the product of each brightness value in the brightness enhancement matrix and a second preset coefficient to obtain a second brightness enhancement matrix; taking the first brightness enhancement matrix as a weight matrix, and mapping and stretching the brightness value of each pixel point in the first effective preview image to obtain an intermediate image;

and mapping and superposing the second brightness enhancement matrix and the brightness value of each pixel point of the intermediate image to obtain the brightness value of each pixel point of a target image, multiplexing the chromatic value and the concentration value of each pixel point of the first effective preview image, and outputting the target image in a YUV format.

Preferably, the color image sensor monitors road environment information by using a visible waveband, and the black-and-white image sensor monitors license plate information by using an infrared waveband; and the shutter speed of the color image sensor during exposure is less than the shutter speed of the black-and-white image sensor during exposure.

In a second aspect, the present application provides an image processing apparatus, which is applied to a camera, where the camera includes a black and white sensor and a color sensor, the black and white sensor is used to collect license plate information of vehicles running in a road monitoring area, and the color sensor is used to collect road environment information in the road monitoring area;

the device comprises:

the first exposure adjusting module is used for carrying out exposure adjustment on the color image sensor based on a first original preview image acquired by the color sensor;

the second exposure adjusting module is used for carrying out exposure adjustment on the black-and-white image sensor through an infrared light supplementing device based on the first original preview image and a second original preview image acquired by the black-and-white sensor at the same moment;

the image acquisition module is used for respectively acquiring a first effective preview image and a second effective preview image which are acquired by the color image sensor and the black-and-white image sensor after exposure adjustment, wherein the first effective preview image clearly displays road environment information, and the second effective preview image clearly displays license plate information of vehicles running on a road;

and the image synthesis module is used for synthesizing the first effective preview image and the second effective preview image to obtain a target image for output.

The technical scheme provided by the embodiment of the application can have the following beneficial effects:

in the embodiment of the application, the camera may include a color sensor for acquiring road environment information in a road monitoring area and a black-and-white sensor for acquiring license plate information of a vehicle running in the road monitoring area, and the exposure adjustment of the black-and-white image sensor may be performed by combining an infrared light supplement device with a first original preview image acquired by the color sensor and a second original preview image acquired by the black-and-white sensor at the same time.

After exposure adjustment is carried out on the infrared sensor, the road environment information can be clearly displayed on a first effective preview image acquired by the infrared sensor; after exposure adjustment is carried out on the black-and-white sensor, the second effective preview image acquired by the black-and-white sensor can clearly display the license plate information of vehicles running on the road, and the obtained target image can clearly display the road environment information and the license plate information through synthesis of the first effective preview image and the second effective preview image, so that complete monitoring of a road monitoring area is realized, and the monitoring effect of the camera is improved.

Drawings

FIG. 1 is a flow chart illustrating steps of a method embodiment of image processing according to an exemplary embodiment of the present application;

fig. 2 is a schematic diagram of a block set obtained after binarization adjustment is performed on each first block image in a first block set when exposure adjustment is performed on a black-and-white image sensor in the step flowchart of fig. 1 according to an exemplary embodiment of the present application;

fig. 3 is a schematic diagram of a first block set after binarization processing corresponding to two frames of first original preview images determined in the manner shown in fig. 2 according to an exemplary embodiment of the application;

fig. 4 is a schematic diagram illustrating a license plate region surrounded by the first block images of the target identified in the last first block set after binarization processing in fig. 3 according to an exemplary embodiment of the application;

fig. 5 is a schematic diagram illustrating a target license plate region identified from a second block set of a second original preview image corresponding to the first block set at the same acquisition time as the first original preview image according to the license plate region in fig. 4 in an exemplary embodiment of the present application;

FIG. 6 is a schematic diagram of a first valid preview image output after exposure adjustment of a color image sensor in the flowchart of steps in FIG. 1, as shown in an exemplary embodiment of the present application;

FIG. 7 is a schematic diagram of a second effective preview image output after exposure adjustment for a black and white image sensor according to an exemplary embodiment of the present application in conjunction with FIGS. 2-5;

fig. 8 is a schematic diagram of a target image, which is obtained by superimposing license plate information in the second effective preview image shown in fig. 7 onto a license plate region of the first effective preview image shown in fig. 6 and clearly displays both road environment information and license plate information, according to an exemplary embodiment of the present application;

FIG. 9 is a hardware block diagram of the device in which the apparatus of the present application is located;

FIG. 10 is a block diagram illustrating an embodiment of an image processing apparatus corresponding to the embodiment of the method of FIG. 1, according to an exemplary embodiment of the present application;

fig. 11 is a block diagram illustrating an embodiment of an electronic device corresponding to the embodiment of the apparatus in fig. 10 according to an exemplary embodiment of the present application.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.

Referring to FIG. 1, a flowchart illustrating steps of one embodiment of a method of image processing is shown in an exemplary embodiment of the present application. The embodiment of the application can be applied to cameras, and in specific implementation, the application of the cameras on road scenes is mainly divided into two main categories according to functions: one is scene monitoring, which is mainly used for monitoring road environment under the conditions of vehicle passing and no vehicle; the other type is vehicle monitoring, which is mainly used for monitoring vehicle information such as license plates and the like when a road passes.

In the embodiment of the present application, the camera may include a black and white sensor and a color sensor, the black and white sensor is used for collecting license plate information of a vehicle running in the road monitoring area, and the color sensor is used for collecting road environment information in the road monitoring area, where the road environment information may include information of a vehicle profile and other information on a currently monitored road.

Aiming at the same road monitoring area, the black-white sensor and the color sensor can simultaneously acquire images to obtain two corresponding paths of imaging data. Naturally, before that, the black-and-white sensor and the color sensor need to be calibrated so that the size of the images shot by the black-and-white sensor and the size of the images shot by the color sensor are the same, the shot areas are the same, and the same coordinate system is used.

In one embodiment, the color image sensor monitors road environment information by using a visible waveband, and the black-and-white image sensor monitors license plate information by using an infrared waveband; and the shutter speed of the color image sensor during exposure is less than the shutter speed of the black-and-white image sensor during exposure.

Specifically, the following differences may exist between a black-and-white sensor and a color sensor:

(1) the black-and-white image sensor is used for collecting license plate information of the vehicle, and exposure is carried out according to brightness statistics of a license plate area to obtain an image with proper license plate exposure; the color image sensor is used for collecting road environment information, and exposure is carried out according to the overall brightness statistics of the road, and the license plate is overexposed at the moment.

(2) The color image sensor monitors road environment information using a visible band, and the black-and-white image sensor monitors license plate information using an infrared band (e.g., 730 nm).

(3) Color image sensors may raise the shutter to 1/M (e.g., 1/25) when performing an exposure; black and white image sensors may raise the shutter to 1/N (e.g., 1/200) when performing an exposure.

Of course, the embodiments of the present application are not limited to the above-mentioned differences, and other differences that a person skilled in the art can design two image sensors according to actual requirements are all possible.

The embodiment of the application specifically comprises the following steps:

101, performing exposure adjustment on the color image sensor based on a first original preview image acquired by the color sensor;

in the embodiment of the present application, for a color image sensor, exposure adjustment may be directly performed according to a first original preview image acquired by the color image sensor.

In a preferred embodiment of the present application, step 101 may further include the following sub-steps:

substep S11, performing blocking processing on the first original preview image according to a preset image blocking strategy to obtain a first blocking set, where the first blocking set includes M first blocking images;

in a specific implementation, the first raw preview image may be an image in Bayer format, which is a raw picture inside the camera, with the suffix of.raw.

The first original preview image may be subjected to a blocking process according to a preset image blocking policy (for example, the size of each block image is 32 × 32 pixels), so as to obtain M first block images, where the M first block images form a first block set.

Substep S12, obtaining a brightness value of each first block image, where the brightness value of the first block image is an average value of brightness values of each pixel point in the first block image;

for each first block image, the luminance value of each pixel point in the block may be averaged (i.e. the sum of the luminance values of each pixel point/the number of pixel points in the block) to obtain the luminance value of the first block image.

Substep S13, calculating an average value of luminance values of each first block image in the first block set to obtain a luminance value of the first original preview image;

after the luminance values of the M first block images in the first block set are calculated, the M first block images and the luminance values thereof may constitute luminance statistical information of the first original preview image.

Calculating an average value of the luminance values of the M first block images (i.e., a sum/M of the luminance values of the M first block images), a luminance value of the first original preview image, i.e., Luma _ Avr, may be obtained.

And a substep S14 of comparing the brightness value of the first original preview image with a preset brightness range, and performing exposure adjustment on the color image sensor according to the comparison result.

The exposure adjustment means that the brightness of the image frame is automatically adjusted and maintained at a certain brightness by some methods.

In a specific implementation, a brightness range may be preset (may be specifically set according to actual requirements), and exposure adjustment may be performed on the color image sensor by comparing the brightness value of the first original preview image with the preset brightness range.

One way of exposure adjustment may be: when the brightness value of the first original preview image exceeds the preset brightness range, the color sensor can be considered to be in an overexposure state currently, so that the exposure target value of the color image sensor can be reduced, the brightness of the picture acquired by the color image sensor is reduced, and the brightness is prevented from being overexposed. When the brightness value of the first original preview image is lower than the preset brightness range, the color sensor can be considered to be in an underexposure state currently, so that the exposure target value of the color image sensor can be improved, the brightness of the picture acquired by the color image sensor is increased, and the brightness is not underexposed.

102, carrying out exposure adjustment on the black-and-white image sensor through an infrared light supplementing device based on the first original preview image and a second original preview image acquired by the black-and-white sensor at the same moment;

according to the embodiment of the application, the infrared light supplementing device can be arranged for the black-and-white sensor, and the infrared light supplementing device can be controlled to supplement light to the black-and-white sensor by combining the first original preview image and the second original preview image acquired by the black-and-white sensor at the same time, so that exposure adjustment of the black-and-white image sensor is realized.

In a preferred embodiment of the present application, the step 102 may further include the following sub-steps:

substep S21, performing blocking processing on the second original preview image according to the image blocking policy to obtain a second blocking set, where the second blocking set includes M second blocking images;

in a specific implementation, the same image blocking policy as that of the first original preview image may be used to perform blocking processing on the second original preview image to obtain M second blocked images, where the M second blocked images form a second blocked set.

Because the color sensor and the black-and-white sensor are calibrated, and the sizes of the images acquired by the color sensor and the black-and-white sensor are the same, after the same image blocking strategy is adopted for image blocking processing, each second blocking image has a mapping between the corresponding first blocking image with the same position and one of the first blocking images.

Substep S22, obtaining a brightness value of each second block image, where the brightness value of the second block image is an average value of brightness values of each pixel point in the second block image;

for each second block image, the luminance value of each pixel point in the block may be averaged (i.e. the sum of the luminance values of each pixel point/the number of pixel points in the block) to obtain the luminance value of the second block image.

A substep S23, identifying a target license plate region of the target vehicle from the second block set according to the brightness value of each first block image and the brightness value of each second block image;

specifically, after the brightness values of the second block images are obtained, the target license plate area of the target vehicle may be determined by combining the brightness values of the corresponding first block images and the brightness value of the first original preview image.

In a preferred embodiment of the present application, the sub-step S23 further includes the following sub-steps:

substep S231, determining a brightness reference value according to the brightness value of the first original preview image;

in one implementation, the luminance value Luma _ Avr of the first original preview image may be multiplied by a preset adjustment coefficient K _ Luma (K _ Luma may be an empirical value), and the resultant value is used as the luminance reference value.

Substep S232, for the first block set, determining a luminance value of the first block image having a luminance value smaller than the luminance reference value as a preset minimum luminance value, and determining a luminance value of the first block image having a luminance value greater than or equal to the luminance reference value as a preset maximum luminance value;

after obtaining the luminance reference value, the luminance value of each first block image in the first block set may be compared with the luminance reference value, and the luminance value of the first block image whose luminance value is greater than or equal to the luminance reference value is determined as a preset maximum luminance value (for example, the preset maximum luminance value is 1023); the luminance value of the first block image with the luminance value smaller than the luminance reference value is determined as a preset minimum luminance value (for example, the preset minimum luminance value is 0), and blocks of a brighter part in the first block set can be highlighted after the binarization processing. For example, as shown in fig. 2, the luminance value of the first block image in white is a preset maximum luminance value, and the luminance values of the other first block images are preset minimum luminance values.

In the substep S233, tracking the first block image with the brightness value being the preset maximum brightness value, so as to determine the first block image with the changed position and the brightness value being the preset maximum brightness value as a target first block image, and taking a region surrounded by each target first block image as a license plate region;

in one embodiment, one tracking method may be: and comparing the first block sets which correspond to the first original preview images of two or more continuous frames and are subjected to the binarization processing to judge whether the positions of the first block images with the brightness values of the preset maximum brightness values are changed or not, and taking the first block images with the brightness values of the changed positions of the preset maximum brightness values as the first block images in a moving state, namely the target first block images (moving _ res).

In practice, in a vehicle monitoring scene in a road, a first block image in a moving state may be regarded as a region where vehicle lamps are located, and a region surrounded by the regions where the two vehicle lamps are located generally includes a license plate region. After the target first block images are identified, the region surrounded by the target first block images can be used as a license plate region.

For example, in fig. 3, a first binarized set of blocks corresponding to two consecutive frames of a first original preview image is shown, comparing the position change of the first block image of which each brightness value is a preset maximum brightness value in the first block set after the binarization processing of the front frame and the back frame, it can be found that the positions of the first block image included in the area a (i.e. the first block image having the luminance value of the preset maximum luminance value in the first block set) and the first block image included in the area B are changed, while the positions of the first block image contained in the area C and the first block image contained in the area D are not changed, at this time, the areas a and B can be regarded as the positions of the moving light sources (i.e., lamps), namely, the first block image contained in the area a and the first block image contained in the area B in the first original preview image of the previous frame or the next frame are the target first block images.

If the first original preview image corresponding to the second first block set in fig. 3 is taken as the current first original preview image, a region enclosed by the region a and the region B in the first block set is taken as the license plate region, as shown in the region a1 in fig. 4.

In the substep S234, in a second block set of a second original preview image corresponding to the first block set and having the same acquisition time as the first original preview image, identifying a second block image mapped with the first block image included in the license plate region to obtain a target license plate region of the target vehicle;

for a current first original preview image, after a license plate region is determined from a first block set which is subjected to binarization processing and corresponds to the first original preview image, a second original preview image which is acquired at the same time as the first original preview image can be acquired, and a second block image which is mapped with a first block image contained in the license plate region is identified from a second block set which corresponds to the second original preview image, so that a target license plate region of a target vehicle is obtained.

For example, for each first block image contained in the a1 area in fig. 4, a second block set of the second original preview image acquired at the same time is shown in fig. 5, and each second block image mapped to each first block image contained in the a1 area is identified in the second block set of fig. 5, and then the target license plate area composed of the identified second block images is shown in the a2 area of fig. 5.

And a substep S24, acquiring the brightness value of the target license plate region, and controlling the infrared light supplementing device to supplement infrared light to the black-and-white image sensor according to the brightness value of the target license plate region.

And the brightness value of the target license plate region is the average value of the brightness values of the second block images contained in the target license plate region. Specifically, after the target license plate region is determined, an average operation may be performed according to the acquired brightness values of the second block images included in the target license plate region, so as to obtain the brightness value of the target license plate region.

In a specific implementation, after the brightness value of the target license plate region is obtained, the brightness value of the target license plate region can be compared with a preset license plate brightness range (which can be set according to actual requirements), and according to a comparison result, the infrared light supplementing device is controlled to supplement infrared light to the black-and-white image sensor, so that the brightness value of the target license plate region is within the license plate brightness range, and exposure adjustment of the black-and-white image sensor is achieved.

The embodiment of the application adopts the infrared light supplementing device to supplement the infrared light to the black-and-white image sensor, belongs to soft light supplementing, and avoids visual interference brought to a driver.

103, respectively acquiring a first effective preview image and a second effective preview image which are acquired by the color image sensor and the black-and-white image sensor after exposure adjustment, wherein the first effective preview image clearly displays road environment information, and the second effective preview image clearly displays license plate information of vehicles running on a road;

it should be noted that, the exposure adjustment for the color image sensor and the black-and-white image sensor in steps 101 and 102 may be performed more than once, and after each execution, it is checked whether the color image sensor outputs the first valid preview image, and whether the black-and-white image sensor outputs the second valid preview image, and if not, the corresponding steps are returned to be continuously performed, that is, if the color image sensor does not output the first valid preview image, step 101 is continuously performed, and if the black-and-white image sensor does not output the second valid preview image, step 102 is continuously performed until the color image sensor outputs the first valid preview image, and the black-and-white image sensor outputs the second valid preview image.

The first effective preview image can clearly display the road environment information, and at this time, the distinction degree between the license plate information and the background environment is not obvious, as shown in fig. 6, and fig. 6 shows the result in the Y domain (i.e., Y component). The second effective preview image can clearly display the license plate information of the vehicles running on the road, and due to the reasons of infrared supplementary lighting, license plate reflection and the like, the effect achieved by the second effective preview image basically only keeps the license plate information and the imaging of objects with strong reflection of the supplementary lighting, as shown in fig. 7.

The degree of clarity can be determined empirically by the developer.

And 104, synthesizing the first effective preview image and the second effective preview image to obtain a target image for output.

In this embodiment, a first effective preview image and a second effective preview image acquired by two image sensors at the same time can be fused, license plate information clearly displayed in the second effective preview image is superimposed on a license plate area of the first effective preview image, and a target image containing clear road environment information and clear license plate information is finally obtained. For example, after the license plate information of fig. 7 is superimposed on the license plate region of fig. 6, the target image of fig. 8 can be obtained.

In a specific implementation, before the first effective preview image and the second effective preview image are subjected to the combining processing, the first effective preview image and the second effective preview image may be subjected to preprocessing such as noise reduction, respectively.

In a preferred embodiment of the present invention, the step 104 further comprises the following sub-steps:

a substep S31 of performing enhancement processing on the second effective preview image to obtain enhancement information;

in a specific implementation, in order to enhance the difference between words and a background on a license plate, enhancement processing may be performed on the second effective preview image, and enhancement information after the enhancement processing is obtained, for example, the enhancement information may include brightness enhancement information, such as a brightness enhancement matrix.

In a preferred embodiment of the present application, the sub-step S31 further includes the following sub-steps: substep S311, forming a brightness matrix from the brightness values of each pixel point in the second effective preview image;

specifically, for a second effective preview image acquired by the black-and-white sensor, the brightness values of the pixels in the second effective preview image can be acquired, and all the acquired brightness values form a brightness matrix, which can also be referred to as a Y component matrix.

Substep S312, determining a brightness value greater than or equal to a preset brightness threshold in the brightness matrix as a first brightness value, and determining a brightness value smaller than the preset brightness threshold as a second brightness value, to obtain a brightness enhancement matrix;

the present embodiment can perform binarization processing on the luminance matrix. Specifically, since the brightness of words in the license plate information is generally greater than the brightness of the background, a brightness threshold (which may be an empirical value) may be set according to the difference, then each brightness value in the brightness matrix is compared with the brightness threshold, the brightness value in the brightness matrix greater than or equal to the brightness threshold is determined as a first brightness value, and the brightness value in the brightness matrix smaller than the brightness threshold is determined as a second brightness value, so as to obtain a brightness enhancement matrix, where the second brightness value is smaller than the first brightness value.

For example, the preset brightness threshold is 25, and then the brightness value greater than or equal to 25 in the brightness matrix is set to 100, and the brightness value less than 25 in the brightness matrix is set to 10 or 5, so as to finally obtain the brightness enhancement matrix a (m, n), thereby enhancing the difference between words and background in the license plate.

And a substep S32 of superimposing the enhancement information on the first effective preview image to generate a target image.

In a specific implementation, the enhancement information may be enhancement information of a license plate region, and the enhancement information may be superimposed on the first effective preview image to generate a target image.

In a preferred embodiment of the present application, the sub-step S32 further includes the following sub-steps:

substep S321, calculating a product of each brightness value in the brightness enhancement matrix and a first preset coefficient to obtain a first brightness enhancement matrix;

and a substep S322 of calculating a product of each luminance value in the luminance enhancement matrix and a second preset coefficient to obtain a second luminance enhancement matrix.

Specifically, the luminance enhancement matrix may be constrained by using a preset coefficient, where the preset coefficient is a value greater than a value 0 and smaller than a value 1. In practice, the preset coefficients are different, and the strength of the constraint is different, as an example, the preset coefficients may include a first preset coefficient and a second preset coefficient, and if the first preset coefficient is k _ a, the obtained first brightness enhancement matrix is k _ a × a (m, n), where 0< k _ a < 1; if the second predetermined coefficient is k _ b, the obtained second luminance enhancement matrix is k _ b × a (m, n), where 0< k _ b < 1.

The substep S323, using the first brightness enhancement matrix as a weight matrix, and mapping and stretching the brightness value of each pixel point in the first effective preview image to obtain an intermediate image;

in this embodiment, the first effective preview image and the second effective preview image acquired at the same time are registered images, and the first effective preview image and the second effective preview image are in pixel level one-to-one correspondence, so that mapping and stretching of corresponding pixel points can be performed on the corresponding first effective preview image acquired at the same time according to the brightness enhancement matrix obtained by stretching the second effective preview image.

In implementation, a luminance matrix composed of luminance values of pixels of the first effective preview image may be denoted as B (m, n). Then, the first luminance enhancement matrix k _ a × a (m, n) is used as a weighting matrix of B (m, n), and weighting processing is performed on each luminance value in B (m, n) by using a corresponding weight, so as to obtain an intermediate image, i.e., the intermediate image is k _ a × a (m, n) × B (m, n).

In practice, the brightness enhancement matrix a (m, n) is obtained by performing enhancement processing on the words in the license plate, so that when performing weighting processing, the brightness enhancement matrix a (m, n) is actually obtained by performing enhancement processing on the words in the license plate region in B (m, n), that is, the intermediate image is an image obtained by performing mapping and stretching on the words in the license plate region.

And a substep S324 of mapping and superposing the second brightness enhancement matrix and the brightness value of each pixel point of the intermediate image to obtain the brightness value of each pixel point of the target image, multiplexing the chromatic value and the concentration value of each pixel point of the first effective preview image, and outputting the target image in a YUV format.

In order to display the license plate information more clearly and achieve a better license plate word enhancement effect, the license plate information can be further superimposed into the intermediate image in the embodiment, wherein in the scenario of one embodiment, the license plate information can be directly superimposed into the intermediate image to obtain the target image if no license plate information exists in the intermediate image.

An exemplary stacking approach is: and mapping and superposing the second brightness enhancement matrix and the brightness value of each pixel point of the intermediate image to obtain the brightness value of each pixel point of the target image, namely the brightness matrix C (m, n) of the target image is k _ a (m, n) B (m, n) + k _ B A (m, n).

In a specific implementation, in order to prevent the synthesized data from exceeding the data bit width of the image processing, the data may be limited to a range of 0 to 4095, that is:

C(m,n)＝clip(C(m,n),0,4095)。

after the brightness matrix of the target image is obtained, post-processing such as noise reduction and sharpening can be performed on the brightness matrix of the target image, then the chromatic values and the density values of all pixel points of the first effective preview image are multiplexed, and the target image is output in a YUV format.

Corresponding to the embodiment of the method, the application also provides an embodiment of the image processing device.

The image processing device embodiment of the application can be applied to terminal equipment. The device embodiments may be implemented by software, or by hardware, or by a combination of hardware and software. The software implementation is taken as an example, and is formed by reading corresponding computer program instructions in the nonvolatile memory into the memory for operation through the processor of the device where the software implementation is located as a logical means. From a hardware aspect, as shown in fig. 9, the hardware structure diagram of the device in the present application is a hardware structure diagram of an apparatus, except for the processor, the memory, the network interface, and the nonvolatile memory shown in fig. 9, the apparatus in the embodiment may also include other hardware according to an actual function of the device, which is not described again.

Referring to fig. 10, a block diagram of an embodiment of an image processing apparatus according to an exemplary embodiment of the present application is shown, where the apparatus is applicable to a camera, the camera includes a black and white sensor for collecting license plate information of a vehicle traveling in a road monitoring area, and a color sensor for collecting road environment information in the road monitoring area;

the method and the device comprise the following steps:

a first exposure adjustment module 1001, configured to perform exposure adjustment on the color image sensor based on a first original preview image acquired by the color sensor;

a second exposure adjustment module 1002, configured to perform exposure adjustment on the black-and-white image sensor through an infrared light supplement device based on the first original preview image and a second original preview image acquired by the black-and-white sensor at the same time;

an image obtaining module 1003, configured to obtain a first effective preview image and a second effective preview image, which are acquired by the color image sensor and the black-and-white image sensor after exposure adjustment, respectively, where the first effective preview image clearly shows road environment information, and the second effective preview image clearly shows license plate information of a vehicle driving on a road;

an image synthesizing module 1004, configured to synthesize the first effective preview image and the second effective preview image to obtain a target image for output.

In a preferred embodiment of the present application, the first exposure adjusting module 1001 includes:

the first blocking submodule is used for carrying out blocking processing on the first original preview image according to a preset image blocking strategy to obtain a first blocking set, and the first blocking set comprises M first blocking images;

a first block brightness value obtaining submodule, configured to obtain a brightness value of each first block image, where the brightness value of the first block image is an average value of brightness values of each pixel point in the first block image;

an image brightness value obtaining submodule, configured to calculate an average value of brightness values of the first block images in the first block set, so as to obtain a brightness value of the first original preview image;

and the exposure sub-module is used for comparing the brightness value of the first original preview image with a preset brightness range and carrying out exposure adjustment on the color image sensor according to the comparison result.

In a preferred embodiment of the present application, the second exposure adjusting module 1002 includes:

the second partitioning submodule is used for partitioning the second original preview image according to the image partitioning strategy to obtain a second partitioning set, and the second partitioning set comprises M second partitioning images;

a second block brightness value obtaining submodule, configured to obtain a brightness value of each second block image, where the brightness value of the second block image is an average value of brightness values of each pixel point in the second block image;

the target license plate region determining submodule is used for identifying a target license plate region of a target vehicle from the second block set according to the brightness value of each first block image and the brightness value of each second block image;

and the infrared light supplementing control sub-module is used for acquiring the brightness value of the target license plate region and controlling the infrared light supplementing device to supplement the infrared light to the black-and-white image sensor according to the brightness value of the target license plate region, wherein the brightness value of the target license plate region is the average value of the brightness values of the second block images contained in the target license plate region.

In a preferred embodiment of the present application, the target license plate region determining sub-module is specifically configured to:

In a preferred embodiment of the embodiments of the present application, the apparatus further comprises:

a first validity checking module, configured to check whether the color image sensor outputs a first valid preview image, and if not, invoke a first exposure adjusting module 1001;

and a second validity checking module, configured to check whether the black-and-white image sensor outputs a second valid preview image, and if not, invoke the second exposure adjusting module 1002.

In a preferred embodiment of the present application, the image synthesis module 1004 includes:

the enhancement information acquisition submodule is used for carrying out enhancement processing on the second effective preview image so as to obtain enhancement information;

and the target image generation submodule is used for superposing the enhancement information to the first effective preview image to generate a target image.

In a preferred embodiment of the present application, the enhancement information includes a luminance enhancement matrix;

the enhanced information obtaining sub-module is specifically configured to:

In a preferred embodiment of the present application, the target image generation sub-module is specifically configured to:

In a preferred embodiment of the embodiments of the present application,

the color image sensor monitors road environment information by adopting a visible waveband, and the black-and-white image sensor monitors license plate information by adopting an infrared waveband; and the shutter speed of the color image sensor during exposure is less than the shutter speed of the black-and-white image sensor during exposure.

Referring to fig. 11, based on the image processing apparatus in fig. 10, a block diagram of an embodiment of an electronic device shown in an exemplary embodiment of the present application is shown, where the electronic device includes a color image sensor 1005, a black-and-white image sensor 1006, a first exposure adjusting module 1001 shown in fig. 10 and connected to the color image sensor 1005, a second exposure adjusting module 1002 shown in fig. 10 and connected to the black-and-white image sensor 1006, an infrared light supplementing device 1007 connected to the second exposure adjusting module 1002, an image obtaining module 1003 shown in fig. 10, and an image synthesizing module 1004 shown in fig. 10.

For the device and electronic apparatus embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points.

The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the application. One of ordinary skill in the art can understand and implement it without inventive effort.

Embodiments of the present application also provide a computer-readable storage medium, on which a computer program is stored, which when executed by a processor, implements the steps of the above-described method embodiments.

The embodiment of the present application further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above method embodiments when executing the program.

Embodiments of the subject matter and the functional operations described in this specification can be implemented in: digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a tangible, non-transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode and transmit information to suitable receiver apparatus for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

Computers suitable for executing computer programs include, for example, general and/or special purpose microprocessors, or any other type of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory and/or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer does not necessarily have such a device. Further, the computer may be embedded in another device, e.g., a vehicle-mounted terminal, a mobile telephone, a Personal Digital Assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device such as a Universal Serial Bus (USB) flash drive, to name a few.

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., an internal hard disk or a removable disk), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. In other instances, features described in connection with one embodiment may be implemented as discrete components or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Further, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.

The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. The method is characterized by being applied to a camera, wherein the camera comprises a black-and-white sensor and a color sensor, the black-and-white sensor is used for collecting license plate information of vehicles running in a road monitoring area, and the color sensor is used for collecting road environment information in the road monitoring area;

the method comprises the following steps:

2. The method of claim 1, wherein the exposure adjustment of the color image sensor based on the first raw preview image captured by the color sensor comprises:

3. The method according to claim 2, wherein the adjusting exposure of the black-and-white image sensor by an infrared light supplement device based on the first original preview image and a second original preview image acquired by the black-and-white sensor at the same time comprises:

4. The method of claim 3, wherein identifying a target license plate region of a target vehicle from the second set of blocks according to the luminance values of the first block images and the luminance values of the second block images comprises:

5. The method according to any of claims 1-4, further comprising, before the respectively acquiring the first and second effective preview images simultaneously acquired by the color image sensor and the black-and-white image sensor after exposure adjustment, the steps of:

6. The method according to any of claims 1-4, wherein said combining the first active preview image and the second active preview image to obtain a target image for output comprises:

7. The method of claim 6, wherein the enhancement information comprises a luminance enhancement matrix;

8. The method of claim 7, wherein superimposing the enhancement information into the first effective preview image generates a target image comprising:

9. The method of claim 1, wherein the color image sensor monitors road environment information using a visible band, and the black and white image sensor monitors license plate information using an infrared band; and the shutter speed of the color image sensor during exposure is less than the shutter speed of the black-and-white image sensor during exposure.

10. An image processing device is characterized in that the device is applied to a camera, the camera comprises a black and white sensor and a color sensor, the black and white sensor is used for collecting license plate information of vehicles running in a road monitoring area, and the color sensor is used for collecting road environment information in the road monitoring area;

the device comprises: