TW202213269A - Image processing method and apparatus, computer device, and storage medium - Google Patents

Abstract

An image processing method and apparatus, a computer device, and a storage medium. The method comprises: obtaining a reference color image and at least one real-time light intensity change amount corresponding to the reference color image; and generating a fused color image according to the reference color image and each real-time light intensity change amount.

Description

Image processing method, device, computer equipment and storage medium

Embodiments of the present invention relate to image processing technologies, and in particular, to an image processing method, device, computer equipment, and storage medium.

With the continuous development of computer vision technology, people's requirements for the quality of color images captured by traditional shooting equipment are also increasing. By capturing absolute light intensity information and color information, traditional photographic equipment can form color images with high image reproduction.

At present, traditional shooting equipment is widely used in home entertainment electronic equipment. However, because traditional shooting equipment needs to collect and output the pixel value of each pixel in the image every time it performs image acquisition and output, the The acquisition and output interval time is generally long, which cannot be effectively used in the industrial control field that requires high image acquisition speed.

The embodiments of the present invention provide an image processing method, device, computer equipment and storage medium, and provide a new dual-modal image fusion technology to improve the image accuracy of fused color images.

In a first aspect, an embodiment of the present invention provides an image processing method, the method includes: acquiring a reference color image, and at least one instant light intensity variation corresponding to the reference color image; Strong variation, resulting in fused color images.

In a second aspect, an embodiment of the present invention further provides an image processing method, the method includes: acquiring a plurality of color images by using a color image sensing circuit in a dual-modal visual sensor; The light intensity change amount sensing circuit in the device obtains the real-time light intensity change amount; according to each of the color images and each of the real-time light intensity change amounts, the image processing method according to any one of the embodiments of the present invention is used to generate the image processing method for At least one fused color image inserted between two consecutive color images.

In a third aspect, an embodiment of the present invention also provides an image processing device, the device includes: a fusion feature acquisition module, configured to acquire a reference color image and at least one instant light intensity change corresponding to the reference color image; fusion The image generating module is used for generating a fused color image according to the reference color image and each of the real-time light intensity changes.

In a fourth aspect, an embodiment of the present invention further provides an image processing device, the device comprising: a color image acquisition module for acquiring a plurality of color images by using a color image sensing circuit in a dual-modal visual sensor; The real-time light intensity change acquisition module is used for obtaining the real-time light intensity change by the light intensity change sensing circuit in the dual-mode visual sensor; the fusion color image generation module is used for according to each color image and For each of the instantaneous light intensity changes, the image processing method according to any embodiment of the present invention is used to generate at least one fused color image for insertion between two consecutive color images.

In a fifth aspect, an embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements any of the present invention when executing the program. The image processing method described in the embodiment.

In a sixth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, wherein the program is executed by a processor to implement the image processing method according to any embodiment of the present invention .

The technical solution of the embodiment of the present invention is to obtain a reference color image and at least one real-time light intensity variation corresponding to the reference color image; and generate a fusion color image technology according to the reference color image and each of the real-time light intensity changes The scheme proposes a new dual-modal image fusion technology that combines low-speed color images and high-speed light intensity changes. The feature of high-speed instant light intensity changes is added to the fusion color image, which improves the image fusion of color images. Accuracy and image quality.

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention. In addition, it should be noted that, for the convenience of description, the drawings only show some but not all structures related to the present invention.

In addition, it should be noted that, for the convenience of description, the drawings only show some but not all of the contents related to the present invention. Before discussing the exemplary embodiments in greater detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart depicts various operations (or steps) as a sequential process, many of the operations may be performed in parallel, concurrently, or concurrently. Additionally, the order of operations can be rearranged. The process may be terminated when its operation is complete, but may also have additional steps not included in the figures. The process may correspond to a method, function, procedure, subroutine, subroutine, and the like.

In some related technologies, one or more interpolated images can be inserted between two consecutively acquired color images by a set interpolation algorithm, so as to reduce the time interval between adjacent color images finally obtained. However, since the related art still uses color images collected by traditional shooting equipment for interpolation, no additional image information is substantially introduced. Therefore, the interpolation effect is poor, and the image precision and quality of the interpolated images are low.

In the first aspect, FIG. 1 is a flowchart of an image processing method provided by an embodiment of the present invention. This embodiment can be applied to the case where image signals of two different modalities are fused to obtain a fused color image. Executed by an image processing device, which can be implemented by software and/or hardware, and can generally be integrated into a terminal with image processing function, or directly integrated into a dual-mode imager for acquiring image signals of two modalities. in the modal vision sensor. The method of the embodiment of the present invention specifically includes the following steps.

S110. Acquire a reference color image and at least one instant light intensity variation corresponding to the reference color image.

In this embodiment, the reference color image specifically refers to a reference image used for fusion based on it to obtain a fused color image, and the reference color image is composed of a matrix combination composed of a plurality of rows and columns of pixels. Each pixel has a set pixel value, and the pixel value can reflect the color information of the location of the pixel. The color information of the above-mentioned pixel points can be represented by using RGB color space, YUV color space or YCbCr color space.

The instant light intensity change specifically refers to the light intensity change of a certain pixel in the reference color image in a certain period of time, or it can also be expressed as a relative gray value (brightness value) Change amount, the instant light intensity change amount represents the change amount between the current brightness value of the pixel point and the historical brightness value at a previous moment. Optionally, there may be one or more instant light intensity changes obtained at a certain moment. If there are multiple instant light intensity changes, the multiple instant light intensity changes correspond to different ones in the reference color image. pixel location.

It is understandable that when each color image is generated, the pixel value of each pixel must be generated. Therefore, the generation speed of the color image will be relatively slow, while the brightness change value of a pixel can be captured at high speed (for example, , captured by a dynamic vision sensor). Therefore, in this embodiment, the inventor creatively proposes to use the real-time light intensity variation of pixels obtained by high-speed real-time acquisition to fuse with the reference color image to obtain a fused color image.

In this embodiment, the at least one instant light intensity change amount corresponding to the reference color image specifically refers to, for each pixel in the reference color image, at one or more times after the acquisition time of the reference color image , one or more instant light intensity changes obtained by the collection. Generally speaking, whenever the pixel value of a certain pixel in the reference color image changes, it can be captured at high speed and in real time to generate the real-time light intensity change.

Optionally, after a reference color image is acquired for a certain shooting area, the acquisition time of the reference color image can be recorded, and then, for the same shooting area, a period of time (for example, 5ms, 10ms, etc.) from the acquisition time can be obtained. ) of the instantaneous light intensity change of one or more pixels in the .

In a specific example, a static image sensor and a dynamic visual sensor can be used to obtain a reference color image and at least one instant light intensity variation corresponding to the reference color image, respectively.

Optionally, a static image sensor and a dynamic vision sensor can be used to capture images in the same shooting area, and ensure that each pixel of the image captured by the static image sensor is the same as the image captured by the dynamic vision sensor. Each pixel point collected has a one-to-one correspondence, and then one or more real-time light intensity changes obtained by the dynamic visual sensor can be obtained at high speed and in real time during the time gap between two pictures taken by the static image sensor, which is accurate. The fusion results in a high-quality fused color image whose acquisition time lies between the two images.

Optionally, the still image sensor includes an APS sensor, a CCD sensor, and the like.

In another specific example, a dual-modality visual sensor can be used to simultaneously acquire the reference color image and at least one instantaneous light intensity change corresponding to the reference color image.

Optionally, the dual-modal visual sensor can simultaneously acquire a reference color image and an instantaneous change in light intensity for the same shooting area, and then can collect one or more images obtained after the acquisition time of the reference color image. Instantaneous light intensity change amount, as the instant light intensity change amount corresponding to the reference color image.

S120. Generate a fused color image according to the reference color image and each of the real-time light intensity changes.

In this embodiment, the fused color image is a color image that simultaneously includes the information of the reference color image and the information of the real-time light intensity variation.

Wherein, according to the reference color image and each of the real-time light intensity changes, the method of generating the fusion color image may be: according to the order of the acquisition time of each real-time light intensity change in a first-to-last order, sequentially acquiring a pair of real-time light intensity changes The pixel value (for example, the brightness value of the pixel, or the R value, B value, G value, etc.), so as to update the pixel value of at least one pixel in the reference color image once according to each instantaneous light intensity change, and finally obtain a fused color image.

The technical solution of the embodiment of the present invention is to obtain a reference color image and at least one real-time light intensity variation corresponding to the reference color image; and generate a fusion color image technology according to the reference color image and each of the real-time light intensity changes The scheme proposes a dual-modal image fusion technology that fuses low-speed color images and high-speed light intensity changes. The feature of high-speed instant light intensity changes is added to the fused color images to improve the image accuracy and image quality of the fused color images. quality.

On the basis of the above embodiments, generating a fusion color image according to the reference color image and each of the instantaneous light intensity changes may be: Pixel value adjustment is performed on the corresponding fused region to generate the fused color image.

As mentioned above, the instantaneous light intensity change reflects the brightness change of a certain pixel in the reference color image. Therefore, after obtaining the instantaneous light intensity change, it is possible to The brightness value of a pixel is adjusted. However, such adjustment will cause sudden change in brightness between the pixel and other pixels, which in turn makes the image display effect of the fused color image more abrupt and the fusion effect is unsatisfactory.

Based on this, in this embodiment, the inventor proposes to determine a fusion region (a set of pixel points, including a plurality of pixel points) in the entire reference color image based on the pixel position of each instantaneous light intensity change, and to determine the fusion region The pixel value of each pixel in the area is adjusted as a whole to further improve the image display effect of the fused color image.

Wherein, when determining the fusion area, the pixel position that matches the instantaneous light intensity change can be determined as the center point, and a set shape (for example, a rectangle or circle, etc.) and a set size (for example, 10 pixels) can be determined. *10 pixels, 100 pixels * 100 pixels, or the pixel area with the set number of pixels as the radius), as the fusion area. The advantage of this setting is that the brightness transition of the pixel values of each pixel in the fused color image is smoother, and the display effect of the fused color image is improved.

FIG. 2 shows a flowchart of another image processing method in an embodiment of the present invention. This embodiment is embodied on the basis of the above-mentioned embodiment. In this embodiment, in the reference color image, The pixel value adjustment is performed on the fusion regions corresponding to the pixel positions of the instantaneous light intensity variation respectively, which is embodied as: determining the target pixel position corresponding to the instantaneous light intensity variation of the target currently being processed; The target fusion area with matching pixel positions; adjust the pixel value of the target fusion area according to the instantaneous light intensity change of the target.

Correspondingly, as shown in FIG. 2 , the method of this embodiment may include the following steps.

S210. Acquire a reference color image and at least one instant light intensity variation corresponding to the reference color image.

S220: Obtain one of the at least one instant light intensity change corresponding to the reference color image in sequence as the target instant light intensity change currently being processed.

In this embodiment, each real-time light intensity change has a generation time, and then the generation time of the reference acquisition image can be used as the starting point. The at least one real-time light intensity change amount corresponding to the reference color image is sorted, and the sorting result reflects the luminance change order of pixel values of each pixel in the reference color image. Further, according to the sorting result, a real-time light intensity change amount can be sequentially obtained, and the pixel value of one or more pixel points in the reference color image can be adjusted.

S230. Determine the target pixel position corresponding to the current processing target instant light intensity variation.

As mentioned above, each pixel in the reference color image has a one-to-one correspondence with each pixel associated with the instantaneous light intensity change, that is, the instantaneous light intensity change corresponds to a certain one in the reference color image Pixel position (that is, the position of a pixel).

In an optional implementation manner of this embodiment, the instant light intensity variation obtained by the dynamic visual sensor or the dual-modal visual sensor may be in the form of (X, Y, P, T). Among them, "X, Y" is the event address, "P" is the 4-value event output (including the first sign bit), and "T" is the time when the event occurs.

Wherein, the event address corresponds to a pixel position in the reference color image. Optionally, "X, Y" can be the row and column positions in the reference color image, respectively, and "P" is the specific value of the instantaneous light intensity change. Numerical value, "T" is the generation time of the instant light intensity change.

Correspondingly, after obtaining the instantaneous light intensity change amount of the target currently processed, the target pixel position corresponding to the target instantaneous light intensity change amount can be further obtained.

S240. In the reference color image, obtain a target fusion region matching the target pixel position.

In this embodiment, in order to make the finally obtained fused color image smoother, the pixel value of each pixel in the target fusion region whose position of the target pixel is matched may be adjusted.

In an optional implementation of this embodiment, in the reference color image, acquiring a target fusion region matching the target pixel position may be: according to the pixel position and a preset extended fusion range, in the reference color image Determine the target fusion area in the color image.

Optionally, the extended fusion range may be a preset area range (for example, a rectangle or a circle, etc.) centered on the target pixel position, for example, 100 pixels * 100 centered on the target pixel position The area range of the pixel point; it may also be a dynamically changing learning value obtained by learning a neural network model, etc., which is not limited in this embodiment.

S250. Adjust the pixel value of the target fusion region according to the instantaneous light intensity change of the target.

In this embodiment, the target instant light intensity change amount may be a relative brightness value (also referred to as a grayscale value). Based on the instantaneous light intensity change of the target, the brightness value (gray value) or the color value of each color (for example, R value, B value or G value, or It can be understood as the adjustment of the brightness value or gray value of each color component.

In an optional implementation of this embodiment, according to the instantaneous light intensity change of the target, the method of adjusting the pixel value of the target fusion area may be: obtaining the brightness value of each pixel in the target fusion area; The target real-time light intensity change amount, and the brightness value of each pixel point is adjusted.

Optionally, if the pixel value of each pixel in the reference color image is represented by the RGB color space, you can first convert the pixel value of each pixel in the target fusion area to use the preset color space transformation formula. Represented in YUV color space, or using YCbCr color space.

After that, you can use the target instant light intensity change to adjust the pixel value of the Y channel (luminance component channel) in the pixel point of the YUV color space, or adjust the pixel value of the Y channel in the pixel point of the YCbCr color space. After the adjustment of the pixel values is completed, the inverse transformation of the above-mentioned color space transformation formula is used to re-convert the pixel values of each pixel in the target fusion region to be represented by the RGB color space.

Specifically, the instantaneous light intensity change amount of the target can be a positive value (representing an increase in the luminance value of a pixel), or a negative value (representing a decrease in the luminance value of a pixel). Furthermore, after directly superimposing the instantaneous light intensity change of the target and the Y channel pixel value of a pixel point, a new Y channel pixel value of the pixel point can be obtained.

Optionally, if the pixel value of each pixel in the reference color image is directly represented by the YUV color space, or directly represented by the YCbCr color space, the target instant light intensity change can be used to directly compare each YUV color in the target fusion area. The pixel value of the Y channel in the pixel point of the space is adjusted, or the pixel value of the Y channel in the pixel point of the YCbCr color space is adjusted to obtain a new pixel value of each pixel point in the target fusion area.

In an optional manner of this embodiment, the instantaneous light intensity change of the target may be used to uniformly adjust the brightness value of each pixel in the target fusion area, or the brightness value of each pixel in the target fusion area may be adjusted uniformly. Set a weight value for each point, and specify that the pixel point that is closer to the target pixel position has a larger weight value, and then according to the product of the weight value and the target instant light intensity change, the brightness value of each pixel in the target fusion area is calculated. Make matching personalized adjustments.

S260. Determine whether the processing of all the instantaneous light intensity changes is completed: if yes, execute S270; otherwise, return to execute S220.

S270. Use the currently adjusted reference color image as the fusion color image.

In the technical solution of the embodiment of the present invention, when the reference color image is adjusted by the real-time light intensity change, the target pixel position corresponding to the currently processed target real-time light intensity change is determined; in the reference color image, the The target fusion area matching the target pixel position; according to the target real-time light intensity change, the pixel value adjustment of the target fusion area is performed, and the image information contained in the target real-time light intensity change is added to the reference color image. At the same time, it also ensures the natural transition of the whole fusion process, makes the transition of pixel values of each pixel in the fusion color image smoother, and further improves the display effect of the fusion color image.

On the basis of the above embodiments, according to the instantaneous light intensity change of the target, the pixel value adjustment method of the target fusion area may also be: according to the target instantaneous light intensity change, calculate the average value of the light change; The average value of the illumination change, and the R value, G value and B value of each pixel in the target fusion area are adjusted respectively.

In this optional embodiment, when the reference color image is represented by the RGB color space, in the process of adjusting the pixel value of the target fusion region, an additional calculation amount of color space conversion is introduced. Therefore, a method of directly adjusting the R value, G value and B value of each pixel in the target fusion area by using the instantaneous light intensity change of the target is further proposed.

Specifically, considering which channel of the R value, G value and B value in each pixel in the target fusion area is adjusted by using the target instant light intensity change, each pixel in the target fusion area will produce chromatic aberration. Therefore, it can be considered to adjust the R value, G value and B value of each pixel at the same time according to the same amount of light intensity change, so as to ensure the display effect of the fusion color image under the premise of reducing the amount of calculation to the greatest extent. .

In this embodiment, since it is necessary to adjust the R value, G value and B value of each pixel at the same time by using the target instant light intensity change, it is necessary to divide the target instant light intensity change into each pixel. on the color channel. Therefore, according to the instantaneous light intensity change of the target, the average value of the light change can be calculated first.

In a specific example, for example, if the instantaneous light intensity change of the target is A, A/3 can be used as the average value of the light change, or K1*(A/3) can be used as the average value of the light change, and the K1 can be Adjust the scale factor for a preset. Of course, other methods may also be adopted to calculate the average value of the illumination change, which is not limited in this embodiment.

Wherein, when adjusting the R value, G value and B value of each pixel in the target fusion area according to the average value of the illumination change, the average value of the illumination change can be used to adjust each pixel in the target fusion area. The R value, G value and B value are adjusted uniformly. It is also possible to set a weight value for each pixel in the target fusion area, and specify that the pixel point closer to the target pixel position has a higher weight value. The R value, G value and B value of each pixel in the target fusion area can be adjusted according to the product of the weight value and the average value of the illumination change.

FIG. 3a shows a flowchart of another image processing method in the embodiment of the present invention. This embodiment is embodied on the basis of the above-mentioned embodiment. In this embodiment, at least one image corresponding to the reference color image is acquired. The operation of the instant light intensity change is embodied as: obtaining the image generation time of the reference color image; determining the light intensity change collection time period according to the image generation time and the accumulated time of the light intensity change; The instantaneous light intensity variation detected in the quantity collection time period is determined as at least one instantaneous light intensity variation corresponding to the reference color image.

Correspondingly, as shown in FIG. 3a, the method of the embodiment of the present invention may include the following steps.

S310. Acquire a reference color image.

In this embodiment, the reference color image obtained in S310 may refer to a color image collected by a static image sensor, or may refer to a color image collected by a dual-modal visual sensor, etc. The above reference The acquired image is the reference image for generating the fused color image.

S320. Obtain the image generation time of the reference color image.

In this embodiment, the reference color image described in S320 may refer to the reference color image obtained in S310, or may refer to a fused color image obtained by fusion in S370 (that is, the previous fusion image obtained by fusion can be used The color image is used as the reference color image to generate the latter fused color image).

Correspondingly, the image generation time of the reference color image may refer to the acquisition time of the reference color image acquired by the static image sensor or the dual-modal visual sensor, or may refer to the reference color image and at least one instant light. The image fusion time of a fused color image obtained by fusion of strong changes.

S330. Determine the light intensity change collection time period according to the image generation time and the accumulated time period of the light intensity change.

In this embodiment, the time interval for obtaining the reference color image collected by the static image sensor or the dual-modal visual sensor, and the quantifier of the fused color image inserted into the above two reference color images can be obtained in advance, based on The time interval and the quantifier determine the cumulative duration of the matched light intensity change.

In a specific example, if the time interval between the static image sensor or the dual-modal visual sensor to obtain the reference color image is 40ms, and it is preset to insert 3 reference color images into the two reference color images continuously collected by the above device After fusing the color images, the cumulative duration of the light intensity change can be determined to be 40ms/(3+1)=10ms. That is, each fused color image uses the reference color image acquired by the device 10ms ago, or the fused color image fused 10ms ago, and one or more real-time images acquired within 10ms. Variation in light intensity.

In a specific example, if it is determined that the image generation time is T1 and the cumulative duration of the light intensity change is Δt, then a light intensity change collection time period can be determined to be [T1, T1+Δt].

S340. Determine the instant light intensity change detected in the light intensity change collection time period as at least one instant light intensity change corresponding to the reference color image.

S350. Generate a fused color image according to the reference color image and each of the instantaneous light intensity changes.

S360: Determine whether the new reference color image acquisition condition is met: if yes, return to S310; otherwise, execute S370.

S370 , when the new reference color image acquisition condition is not met, update the reference color image to the currently generated fused color image, and return to S320 .

In this embodiment, the new reference color image acquisition condition specifically refers to the acquisition time of the next reference color image of the device used to acquire the reference color image in S310.

If it is determined that the new reference color image acquisition conditions are met, the reference color image newly acquired by the device is used as the reference color image, and new fusion color images are continuously generated; The obtained fused color image is the reference color image, and new fused color images are continuously generated.

It can be understood that the pixel value of each pixel in the reference color image collected by the device is the most accurate, and although one or more instant light intensity changes are introduced into each fused color image, during the fusion process, A certain fusion error will also be introduced. Therefore, if too many fused color images are introduced between the two reference color images collected by the device, the above-mentioned fusion errors will continue to accumulate and amplify. , the quantifier of the fused color image inserted into the two reference color images collected continuously.

3b shows a schematic diagram of a process of generating a fused color image to which the embodiment of the present invention is applicable. As shown in Fig. 3b, a reference color image 3110 is acquired by a static image sensor or a dual-modal visual sensor at time Tx, and another reference color image 3120 is acquired by the same device at time Ty. A total of three fused color images are inserted into the above two reference color images, that is, the first fused color image 3111 obtained by merging at time Ta, the second fused color image 3112 obtained by merging at time Tb, and the third fused color image 3112 obtained by merging at time Tc Fusion color image 3113.

The first fused color image 3111 is obtained by merging the reference color image 3110 and at least one real-time light intensity variation (discrete image points shown in FIG. 3b ) detected in the [Tx, Ta] time period; the second The fused color image 3112 is obtained by merging the first fused color image 3111 and at least one real-time light intensity change detected in the [Ta, Tb] time period; the third fused color image 3113 is obtained by merging the second fused color image 3112 and At least one instantaneous light intensity change detected in the [Tb, Tc] time period is fused.

The technical solution of the embodiment of the present invention can flexibly set the adjacent fusion color by selectively selecting the accumulated time of the light intensity change according to the accuracy requirements of the fusion color image, and then determining the collection time period of the light intensity change. The time interval between images and the image precision of each fused color image further expand the versatility of the technical solutions of the embodiments of the present invention.

FIG. 4a shows a flowchart of another image processing method in the embodiment of the present invention. This embodiment is embodied on the basis of the above-mentioned embodiment. In this embodiment, the reference color image and the real-time light The operation of generating a fused color image is embodied as: inputting the reference color image and each of the instantaneous light intensity changes into a pre-trained dual-modal fusion model, and obtaining the output of the dual-modal fusion model fused color image.

Correspondingly, as shown in FIG. 4a, the method of this embodiment may specifically include the following steps: S410. Acquire a reference color image and at least one instant light intensity variation corresponding to the reference color image. S420. Input the reference color image and each of the real-time light intensity changes into a pre-trained dual-modal fusion model, and obtain a fusion color image output by the dual-modal fusion model.

Wherein, the first input end of the dual-modal fusion model is used for receiving the reference color image, and the second input end of the dual-modal fusion model is used for receiving each of the instantaneous light intensity changes.

In this embodiment, in order to further reduce the real-time calculation amount in the process of generating the fusion image, a dual-modal fusion model can be obtained by pre-training. The parameters are respectively input into the dual-modal fusion model, and the matched fused color image can be output by the model in real time.

Specifically, as shown in FIG. 3b, assuming that the reference color image 3110 and at least one real-time light intensity change detected in the [Tx, Ta] time period are used to fuse the first fused color image 3111, the first fused color image 3111 can be obtained at the time Tx After reaching the reference color image 3110, firstly input the reference color image 3110 to the first input end of the dual-modal fusion model, and in the [Tx, Ta] time period, whenever a real-time light intensity change is detected, immediately The real-time light intensity change is input to the second input end of the dual-modal fusion model, so that the dual-modal fusion model performs cumulative fusion of the reference color image 3110 for each input real-time light intensity change Calculate, and when the determined time point reaches time Ta, obtain the fused color image output by the dual-modal fusion model, that is, the first fused color image 3111 .

Based on this, it is necessary to pre-train to obtain the dual-modal fusion model. Correspondingly, before acquiring the reference color image and at least one instant light intensity variation corresponding to the reference color image, the method may further include: acquiring a training sample set, each training sample includes: a standard color image, at least one light intensity The amount of change and the standard fusion color image; using each of the training samples in the training sample set, the set machine learning model is trained to obtain the dual-modal fusion model.

As mentioned above, due to the use of static image sensors in the related art, it is impossible to acquire two consecutive standard color images with a relatively close interval, and the standard color images included in each training sample and the standard fusion color images , the interval between the two needs to be much smaller than the interval between two consecutive standard color images acquired by the static image sensor.

Correspondingly, when constructing a training sample, an interpolated frame may be firstly performed on two consecutively acquired standard color images to obtain a post-interpolated color image. Since no additional image information is introduced in the above-mentioned interstitial frame process, the image quality of the color image after the interstitial frame is not high enough to be used as a training sample for the dual-modal fusion model, and some advanced image processing techniques are required. , performing image optimization on the color image after the interstitial frame to improve the image quality of the color image after the interstitial frame, and using the optimized color image after the interstitial frame as the standard to fuse the color image.

In this embodiment, the machine learning model trained to obtain the dual-modal fusion model may be a CNN (Convolutional Neural Networks, convolutional neural network) model, or an ANN (Artificial Neutral Network, artificial neural network) model, etc. A machine learning model, which is not limited in this embodiment.

In an optional implementation of this embodiment, considering that the dual-modal fusion model has dual inputs, the twin-modal convolutional network can be selected for training to obtain the dual-modal fusion model.

Optionally, as shown in Figure 4b, the twin convolutional network may include a first convolution module 4110, a second convolution module 4120, a fusion module 4130, and an output module 4140. The first convolution module 4110 includes the first input end, and the second convolution module 4120 includes the second input end; the output ends of the first convolution module 4110 and the second convolution module 4120 are respectively the same as the output ends of the fusion module 4130. The input end is connected, the output end of the fusion module 4130 is connected with the input end of the ASZD 4140, and the output end of the output module is used for outputting the fusion color image. Specifically, the network structures in the first convolution module 4110 and the second convolution module 4120 are the same, and may specifically include: an input convolution layer and a plurality of hidden layers. Wherein, the input convolution layer may include one or more convolution units, and each hidden layer may include a maximum pooling layer and one or more convolution units connected in sequence.

Generally speaking, the first convolution module 4110 and the second convolution module 4120 in the Siamese convolutional network can share weight values during the training process. However, considering that the technical solutions of the embodiments of the present invention need to use image signals of two different modes, that is, the reference color signal and the real-time light intensity change, in this embodiment, the twin convolutional network During the training process of the path, the first convolution module 4110 and the second convolution module do not share weight values.

The fusion module 4130 is the core module in the dual-modal fusion model, and is used to generate a fusion result according to the reference color image and at least one real-time light intensity change corresponding to the reference color image, and combine the fusion result Provided to output module 4140.

Optionally, based on the structure of the twin convolutional network, the output module 4140 can be designed as a single output or as a dual output. When the output module 4140 is designed as a single output, it can only output the fused color image, and when the output module 4140 is designed as a dual output, it can output the fused color image in one way, and the other output is related to the above at least one real-time light intensity change The corresponding cumulative image of light intensity change, in the cumulative image of light intensity change, the pixel position where the black (or a specific color) pixel value is located is the pixel position without light intensity change, and the non-black pixel value The pixel position is the pixel position where the light intensity change occurs, and the specific light intensity change amount corresponding to the pixel position is recorded by the non-black pixel value.

Optionally, when training a dual-modal fusion model, the L1-loss function can be used as the loss function, and at the same time, the optical flow (light intensity change) error can be obtained by using ADAM (A Method for Stochastic Optimization, stochastic optimization method) Trained together with the backpropagation algorithm.

The technical solution of the embodiment of the present invention uses a pre-trained dual-modal fusion model to fuse the reference color image and the real-time light intensity changes corresponding to the reference color image to obtain a fused color image, which can save the greatest extent. The computing power consumption in the fusion process is reduced, and the production efficiency and timeliness of the fusion color image are further improved.

On the basis of the above embodiments, the reference color image can be generated according to the voltage signal of the light intensity of the light signal collected by the color image sensing circuit in the dual-mode visual sensor; the real-time light intensity change can be based on the The current signal of the light intensity change of the light signal collected by the light intensity change amount sensing circuit in the dual-modal visual sensor is generated.

In an optional implementation manner of the embodiment of the present invention, the dual-modal visual sensor may specifically include: a first sensing circuit (also referred to as a light intensity variation sensing circuit) and a second sensing circuit ( It can also be referred to as a color image sensing circuit); the first sensing circuit is used to extract the optical signal of the first set wavelength band in the target optical signal, and output the light intensity change amount representing the optical signal of the first set wavelength band. The current signal; the second sensing circuit is used for extracting the optical signal of the second preset wavelength band in the target optical signal, and outputting a voltage signal representing the light intensity of the optical signal of the second preset wavelength band.

Optionally, the first sensing circuit includes a first excitation type photosensitive unit and a first inhibitory type photosensitive unit, and both the first excitation type photosensitive unit and the first inhibitory type photosensitive unit are used to extract the first in the target light signal. The optical signal of the set wavelength band is converted into a current signal; the first sensing circuit is also used for converting the current signal according to the first excitation type photosensitive unit and the first inhibition type photosensitive unit The difference between the two outputs a current signal representing the variation of the light intensity of the light signal in the first set wavelength band.

Optionally, the second sensing circuit includes at least one second photosensitive unit, and the second photosensitive unit is used to extract the optical signal of the second preset wavelength band in the target optical signal, and convert the optical signal of the second preset wavelength band into a current signal; the second sensing circuit is also used for outputting a voltage signal representing the light intensity of the light signal of the second set wavelength band according to the current signal converted by the second photosensitive unit.

The advantage of this setting is that by using the dual-mode visual sensor to generate the reference color image and the image signal of the two modes of the instantaneous light intensity change, the accurate alignment between the above two modes of signals can be ensured, so as to further Guarantees the image quality of fused color images.

In the second aspect, FIG. 5a is a flowchart of another image processing method provided by an embodiment of the present invention. This embodiment can be applied to the case where at least one fused color image is inserted between two color images obtained by continuous collection, The method can be performed by an image processing device, which can be implemented by software and/or hardware, and can generally be integrated in a terminal with image processing function, or directly integrated in a dual-modal visual sensor. The method of this embodiment specifically includes the following steps.

S510. Acquire a plurality of color images by using the color image sensing circuit in the dual-modal visual sensor.

In this embodiment, the dual-modality visual sensor acquires a plurality of color images at a set acquisition time interval.

S520. Acquire a real-time light intensity change by using the light intensity change sensing circuit in the dual-modal visual sensor.

S530. According to each of the color images and each of the real-time light intensity changes, use the method described in any embodiment of the present invention to generate at least one fused color image for insertion between two consecutive color images.

5b is a schematic diagram of a process of inserting and merging color images in continuous color images to which an embodiment of the present invention is applicable. As shown in Figure 5b, the color image sensing circuit in the dual-modal vision sensor samples the image signals such as color image 1, color image 2, and color image 3 at a set sampling interval. The light intensity change sensing circuit in the sensor obtains the instantaneous light intensity change, and can insert one or more (two in Figure 5b) fused color images between two consecutive color images.

The technical solution of the embodiment of the present invention breaks through the limitation of the color image acquisition time interval in the prior art, can obtain color images with a smaller acquisition time interval, and can ensure the image quality of the fused color image obtained by fusion to the greatest extent. The advantages of color images and real-time light intensity change signals can obtain image signals with different types of image feature information.

In a third aspect, FIG. 6 is a structural diagram of an image processing apparatus according to an embodiment of the present invention. As shown in FIG. 6 , the apparatus 600 includes a fusion feature acquisition module 610 and a fusion image generation module 620 .

The fusion feature acquisition module 610 is configured to acquire a reference color image and at least one real-time light intensity variation corresponding to the reference color image.

The fused image generating module 620 is used for generating a fused color image according to the reference color image and each of the real-time light intensity changes.

The technical solution of the embodiment of the present invention is to obtain a reference color image and at least one real-time light intensity variation corresponding to the reference color image; and generate a fusion color image technology according to the reference color image and each of the real-time light intensity changes The scheme proposes a new dual-modal image fusion technology that combines low-speed color images and high-speed light intensity changes. The feature of high-speed instant light intensity changes is added to the fusion color image, which improves the image fusion of color images. Accuracy and image quality.

On the basis of the above-mentioned embodiments, the fusion image generation module 620 may include: an area fusion unit, configured to perform pixel fusion on the fusion areas corresponding to the pixel positions of the instantaneous light intensity changes in the reference color image. value adjustment to produce the fused color image.

On the basis of the above embodiments, the area fusion unit may specifically include: a target pixel position determination subunit, which is used to determine the target pixel position corresponding to the instantaneous light intensity change of the currently processed target; a target fusion area acquisition subunit, In the reference color image, the target fusion area matching the target pixel position is obtained; the target fusion area adjustment subunit is used to adjust the pixel value of the target fusion area according to the instantaneous light intensity change of the target.

On the basis of the above embodiments, the target fusion area acquisition subunit may be specifically configured to: determine the target fusion area in the reference color image according to the pixel position and the preset extended fusion range.

On the basis of the above embodiments, the target fusion area adjustment subunit can be specifically used to: obtain the brightness value of each pixel in the target fusion area; value to adjust.

On the basis of the above-mentioned embodiments, the target fusion area adjustment subunit can be specifically used to: calculate the average value of the illumination change according to the real-time light intensity change of the target; The R value, G value and B value of each pixel are adjusted.

On the basis of the above embodiments, the fusion feature acquisition module 610 can be specifically used for: acquiring the image generation time of the reference color image; Collection time period; determine the instant light intensity change amount detected in the light intensity change amount collection time period as at least one instant light intensity change amount corresponding to the reference color image.

On the basis of the above-mentioned embodiments, it may further include, repeating the execution module, for: after the fusion color image is generated according to the reference color image and each of the instantaneous light intensity changes, if the acquisition of the new reference color image does not meet the requirements When conditions are met, the reference color image is updated to the currently generated fused color image, and the process returns to the step of acquiring at least one instant light intensity variation corresponding to the reference color image.

On the basis of the above-mentioned embodiments, the fusion image generation module 620 can be specifically used for: inputting the reference color image and each of the real-time light intensity changes into a pre-trained dual-modal fusion model, and obtaining the The fused color image output by the dual-modal fusion model; wherein, the first input of the dual-modal fusion model is used to receive the reference color image, and the second input of the dual-modal fusion model is used to receive the real-time light Strong variability.

On the basis of the above embodiments, it may further include: a model training module, configured to: obtain a training sample set before obtaining a reference color image and at least one instant light intensity variation corresponding to the reference color image, and train the The samples include: a standard color image, at least one light intensity variation, and a standard fusion color image; using each of the training samples in the training sample set, train the set machine learning model to obtain the dual-modal fusion model.

On the basis of the above-mentioned embodiments, the standard fusion color image is obtained by performing interstitial frame and image optimization processing on two consecutively acquired standard color images.

On the basis of the above embodiments, the set machine learning model is a twin convolution network; the twin convolution network includes a first convolution module, a second convolution module, a fusion module and an output module, The first convolution module includes the first input end, and the second convolution module includes the second input end; the output ends of the first convolution module and the second convolution module are respectively connected with the fusion module The input end of the fusion module is connected to the input end of the output module, and the output end of the output module is used to output the fusion color image; wherein, in the training process of the twin convolutional network , the first convolution module and the second convolution module do not share weight values.

On the basis of the above embodiments, the reference color image is generated according to the voltage signal of the light intensity of the light signal collected by the color image sensing circuit in the dual-mode visual sensor; the real-time light intensity variation is based on the dual-mode visual sensor. The current signal of the light intensity change of the light signal collected by the light intensity change amount sensing circuit in the state vision sensor is generated.

The image processing apparatus provided by the embodiment of the present invention can execute the image processing method provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

In the fourth aspect, FIG. 7 is a structural diagram of another image processing device provided by an embodiment of the present invention. As shown in FIG. 7 , the device 700 includes: a color image acquisition module 710 , a real-time light intensity change acquisition module 720 , and The fusion color image generation module 730 .

The color image acquisition module 710 is used for acquiring a plurality of color images through the color image sensing circuit in the dual-mode visual sensor.

The real-time light intensity change amount acquisition module 720 is used for obtaining the real-time light intensity change amount through the light intensity change amount sensing circuit in the dual-mode visual sensor.

The fusion color image generating module 730 is used for generating a method for inserting between two consecutive color images according to each of the color images and each of the real-time light intensity changes using the method described in any embodiment of the present invention at least one fused color image.

The technical solution of the embodiment of the present invention breaks through the limitation of the color image acquisition time interval in the prior art, can obtain color images with a smaller acquisition time interval, and can ensure the image quality of the fused color images obtained by fusion to the greatest extent, and at the same time integrates The advantages of color images and real-time light intensity change signals can obtain image signals with different types of image feature information.

The image processing apparatus provided by the embodiment of the present invention can execute the image processing method provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

In the fifth aspect, FIG. 8 is a schematic structural diagram of a computer device according to an embodiment of the present invention. As shown in FIG. 8 , the computer device includes a processor 80, a memory 81, and may also include an input device 82 and an output device 83; the computer The number of processors 80 in the device can be one or more, and one processor 80 is taken as an example in FIG. For connection in other ways, in FIG. 8 , the connection by bus bar is taken as an example.

As a computer-readable storage medium, the memory 81 can be used to store software programs, computer-executable programs and modules, such as modules corresponding to the image processing method in the embodiment of the present invention. The processor 80 executes various functional applications and data processing of the computer equipment by running the software programs, computer-executable programs and modules stored in the memory 81, that is, to realize the image processing method described in any embodiment of the present invention , the method includes: acquiring a reference color image and at least one instantaneous light intensity variation corresponding to the reference color image; and generating a fusion color image according to the reference color image and each of the instantaneous light intensity variations.

Alternatively, the processor 80 executes various functional applications and data processing of the computer equipment by running the software programs, computer-executable programs and modules stored in the memory 81 to realize another image according to any embodiment of the invention A processing method, the method comprising: acquiring a plurality of color images by using a color image sensing circuit in a dual-mode visual sensor; acquiring real-time light intensity by using a light intensity variation sensing circuit in the dual-mode visual sensor Variation; according to each of the color images and each of the real-time light intensity changes, the image processing method according to any embodiment of the present invention is used to generate at least one fusion for inserting between two consecutive color images Color image.

The memory 81 may mainly include a stored program area and a stored data area, wherein the stored program area can store an operating system and an application program required for at least one function; the stored data area can store data created according to terminal usage. In addition, the memory 81 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk memory device, flash memory device, or other non-volatile solid-state memory device. In some examples, the memory 81 may further include memory disposed remotely from the processor 80, and these remote storages may be connected to computer equipment via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The input device 82 can be used to receive input digital or character information and generate key signal input related to user settings and function control of the computer equipment. The output device 83 may include a display device such as a display screen.

In a sixth aspect, referring to FIG. 9 , an embodiment of the present invention also provides a computer-readable storage medium 900 containing computer-executable instructions (computer programs), and the computer-executable instructions, when executed by a computer processor, are used to execute any of the methods of the present invention. The image processing method according to the embodiment, the method includes: acquiring a reference color image and at least one instant light intensity change amount corresponding to the reference color image; generating a fusion according to the reference color image and each of the instant light intensity changes Color image.

Alternatively, the computer-executable instructions are executed by a computer processor to perform another image processing method according to any embodiment of the present invention, the method comprising: using a color image sensing circuit in a dual-modal visual sensor, Acquire a plurality of color images; obtain the real-time light intensity change by the light intensity change sensing circuit in the dual-modal visual sensor; according to each of the color images and each of the real-time light intensity changes, use any one of the present invention The image processing method described in the embodiment generates at least one fused color image for insertion between two consecutive color images.

Of course, a computer-readable storage medium 900 including computer-executable instructions (computer programs) provided by the embodiment of the present invention is not limited to the above-mentioned method operations, and can also execute the computer-executable instructions provided by any embodiment of the present invention. related operations in the method.

From the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by software and necessary general-purpose hardware, and of course can also be implemented by hardware, but in many cases the former is more best implementation. Based on such understanding, the technical solutions of the present invention can be embodied in the form of software products, which can be stored in computer-readable storage media, such as computer floppy disks. , read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), flash memory (FLASH), hard disk or CD, etc., including several instructions to make a computer A device (which may be a personal computer, a server, or a network device, etc.) executes the methods of the various embodiments of the present invention.

Note that the above are only preferred embodiments of the present invention and applied technical principles. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in detail by the above embodiments, the present invention is not limited to the above embodiments, and can also include more other equivalent embodiments without departing from the concept of the present invention. The scope of the invention is determined by the scope of the appended claims.

80: Processor 81: Memory 82: Input device 83: Output device Steps 600, 700: Device 610: Fusion feature acquisition module 620: Fusion image generation module 710: Color Image Acquisition Module 720: Real-time light intensity change acquisition module 730: Combined color image generation module 900: Computer-readable storage media 3110, 3120: Reference color image 3111: First Fusion Color Image 3112: Second Fusion Color Image 3113: Third Fusion Color Image 4110: The first convolution module 4120: The second convolution module 4130: Fusion Module 4140: Output module

FIG. 1 is a flowchart of an implementation of an image processing method in an embodiment of the present invention; Fig. 2 is the realization flow chart of another image processing method in the embodiment of the present invention; FIG. 3a is a flow chart of the realization of another image processing method in an embodiment of the present invention; 3b is a schematic diagram of a process for generating a fused color image to which an embodiment of the present invention is applicable; Fig. 4a is an implementation flowchart of another image processing method in an embodiment of the present invention; 4b is a schematic structural diagram of a dual-modal fusion model to which an embodiment of the present invention is applicable; Fig. 5a is an implementation flowchart of another image processing method in an embodiment of the present invention; 5b is a schematic diagram of a process of inserting and merging color images in a continuous color image to which an embodiment of the present invention is applicable; 6 is a structural diagram of an image processing apparatus according to an embodiment of the present invention; 7 is a structural diagram of another image processing apparatus in an embodiment of the present invention; FIG. 8 is a structural diagram of a computer device in an embodiment of the present invention. FIG. 9 is a structural diagram of a computer-readable storage medium according to an embodiment of the present invention.

S110, S120: Steps

Claims (18)

An image processing method, comprising: acquiring a reference color image and at least one real-time light intensity variation corresponding to the reference color image; According to the reference color image and each of the real-time light intensity changes, a fused color image is generated. The method of claim 1, wherein generating a fused color image according to the reference color image and each of the real-time light intensity changes, comprising: In the reference color image, pixel value adjustment is performed on the fusion regions corresponding to the pixel positions of each instantaneous light intensity variation, so as to generate the fusion color image. The method according to claim 2, wherein, in the reference color image, adjusting the pixel values of the fusion regions corresponding to the pixel positions of each instantaneous light intensity change amount respectively includes: determining a target pixel position corresponding to a target instant light intensity change currently being processed; In the reference color image, obtain a target fusion region matching the target pixel position; According to the instantaneous light intensity change of the target, the pixel value of the target fusion area is adjusted. The method of claim 3, wherein, in the reference color image, acquiring a target fusion region matching the target pixel position, comprising: According to the pixel position and the preset extended fusion range, the target fusion area is determined in the reference color image. The method according to claim 3, wherein adjusting the pixel value of the target fusion region according to the instantaneous light intensity change of the target, comprising: Obtain the brightness value of each pixel in the target fusion area; Adjust the brightness value of each pixel point according to the change amount of the real-time light intensity of the target. The method according to claim 3, wherein adjusting the pixel value of the target fusion region according to the instantaneous light intensity change of the target, comprising: According to the real-time light intensity change of the target, calculate a mean value of the light change; According to the average value of the illumination change, the R value, G value and B value of each pixel in the target fusion area are adjusted respectively. The method according to claim 1, wherein acquiring at least one instant light intensity variation corresponding to the reference color image comprises: obtaining an image generation time of the reference color image; According to the generation time of the image and the accumulated time of the light intensity change, determine the light intensity change collection time period; The instant light intensity change detected in the light intensity change collection time period is determined as at least one instant light intensity change corresponding to the reference color image. The method of claim 1, wherein after generating a fused color image according to the reference color image and each of the instant light intensity changes, further comprising: When the acquisition condition of the new reference color image is not satisfied, the reference color image is updated to the currently generated fused color image, and the step of acquiring at least one instant light intensity variation corresponding to the reference color image is returned to. The method of claim 1, wherein generating a fused color image according to the reference color image and each of the real-time light intensity changes, comprising: The reference color image and each of the real-time light intensity changes are respectively input into a pre-trained dual-modal fusion model, and the fusion color image output by the dual-modal fusion model is obtained; wherein, the dual-modal fusion model A first input terminal is used for receiving the reference color image, and a second input terminal of the dual-modal fusion model is used for receiving each of the instantaneous light intensity changes. The method according to claim 9, wherein before acquiring the reference color image and at least one instantaneous light intensity variation corresponding to the reference color image, further comprising: acquiring a training sample set, the training samples include: a standard color image, at least one light intensity variation and a standard fusion color image; Using each of the training samples in the training sample set, the set machine learning model is trained to obtain the dual-modal fusion model. The method according to claim 10, wherein the standard fused color image is obtained by performing interstitial frame and image optimization processing on two continuously acquired standard color images. The method of claim 10, wherein the set machine learning model is a twin convolutional network; The twin convolutional network includes a first convolution module, a second convolution module, a fusion module and an output module, the first convolution module includes the first input end, and the second convolution module The module includes the second input end; The output ends of the first convolution module and the second convolution module are respectively connected to the input end of the fusion module, the output end of the fusion module is connected to the input end of the output module, and the output module The output terminal is used to output the fused color image; Among them, in the training process of the twin convolutional network, the first convolution module and the second convolution module do not share weight values. The method of any one of claim 1 to claim 12, wherein: The reference color image is generated according to the voltage signal of the light intensity of the light signal collected by the color image sensing circuit in a dual-modal visual sensor; The real-time light intensity change amount is generated according to the current signal of the light intensity change amount of the light signal collected by the light intensity change amount sensing circuit in the dual-mode visual sensor. An image processing method, comprising: Acquiring a plurality of color images through the color image sensing circuit in the dual-modal vision sensor; Obtain the real-time light intensity change through the light intensity change sensing circuit in the dual-modal visual sensor; According to each of the color images and each of the real-time light intensity changes, adopt the method described in any one of claim 1 to claim 13 to generate at least one fusion for insertion between two consecutive color images Color image. An image processing device, comprising: a fusion feature acquisition module for acquiring a reference color image and at least one real-time light intensity variation corresponding to the reference color image; A fused image generating module is used for generating a fused color image according to the reference color image and each of the real-time light intensity changes. An image processing device, comprising: a color image acquisition module for acquiring a plurality of color images by using the color image sensing circuit in the dual-mode visual sensor; a real-time light intensity change acquisition module, used for obtaining the real-time light intensity change through the light intensity change sensing circuit in the dual-modal visual sensor; A fused color image generating module, for generating a module for inserting into two consecutive two images by adopting the method described in any one of claim 1 to claim 13 according to each of the color images and each of the real-time light intensity changes At least one fused color image between the color images. A computer device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that when the processor executes the computer program, items such as request item 1 to request item are implemented The image processing method described in any one of 13, or the image processing method described in claim 14 is implemented. A computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the image processing method as described in any one of request item 1 to request item 13 is realized, or, the request item is realized The image processing method described in 14.

Images