WO2023077431A1

WO2023077431A1 - Image processing method and apparatus, and image collection device and storage medium

Info

Publication number: WO2023077431A1
Application number: PCT/CN2021/129018
Authority: WO
Inventors: 吴伟霖; 王雯琪; 郑子翔
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2021-11-05
Filing date: 2021-11-05
Publication date: 2023-05-11

Abstract

An image processing method and apparatus, and an image collection device and a storage medium. The method comprises: for the same target photographing scene, switching a first camera of an image collection device to a second camera, so as to realize optical zoom (S202); performing, by using color information of a first image collected by the first camera, color processing on a second image collected by the second camera, so as to obtain a target image, wherein the color information is acquired on the basis of a target pixel area in the first image, and the target pixel area has the same field of view as all or some pixel areas of the second image (S204); and outputting the target image, wherein image areas having the same field of view in an image output before optical zoom by the first camera and the target image have the same color (S206). By means of the method, the effect of a field of view difference on an image color can be eliminated, thereby ensuring that identical scenes in images output before and after the switching of a camera have the same color.

Description

Image processing method, device, image acquisition device and storage medium

technical field

The present application relates to the technical field of image processing, and in particular, to an image processing method, device, image acquisition device, and storage medium.

Background technique

In order to realize that after the image acquisition device zooms with different magnifications, the obtained image is still relatively clear. At present, many image acquisition devices are equipped with multiple cameras with different focal lengths. When the zoom magnification reaches the corresponding zoom magnification of the next camera, the The camera switches from the current camera to the next camera, and uses the next camera to collect images to achieve optical zoom and obtain clear images. However, due to the differences between different cameras, the colors of the same scene in the images output by different cameras before and after camera switching are quite different, which affects user experience.

Contents of the invention

In view of this, the present application provides an image processing method, device, image acquisition device and storage medium.

According to a first aspect of the present application, an image processing method is provided, the method comprising:

For the same target shooting scene, switch the first camera of the image acquisition device to the second camera to achieve optical zoom;

Using the color information of the first image captured by the first camera before the optical zoom to perform color processing on the second image captured by the second camera after the optical zoom to obtain a target image, wherein the color information is based on the first Acquiring a target pixel area in the image, where the target pixel area has the same field of view as all or part of the pixel area of the second image;

Outputting the target image, wherein the image output by the first camera before the optical zoom is in the same color as the image area in the target image having the same field of view.

According to the second aspect of the present application, there is provided an image processing device, the device includes a processor, a memory, and a computer program stored in the memory for execution by the processor, when the processor executes the computer program The following steps can be implemented:

According to a third aspect of the present application, an image acquisition device is provided, and the image acquisition includes a first camera, a second camera, and the image processing device mentioned in the second aspect above.

According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed, the method mentioned in the above-mentioned first aspect is implemented .

Applying the solution provided by this application, in the scene where the camera is switched to achieve optical zoom, the target pixel area can be determined from the first image captured by the first camera, and the target pixel area and all or part of the pixels in the second image captured by the second camera The regions have the same field of view, and then use the color information obtained based on the target pixel region to perform color processing on the second image, and perform color processing on the second image by using the color information of the target pixel region that has the same field of view as the pixel region of the second image Color processing can eliminate the color difference of the images output by the two cameras caused by the difference in the field of view, so that the colors of the images output by the two cameras for the same scene can be consistent before and after switching.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

FIG. 1 is a schematic diagram of images output before and after camera switching according to an embodiment of the present application.

Fig. 2 is a schematic diagram of an image processing method according to an embodiment of the present application.

FIG. 3 is a schematic diagram of an expanded pixel area obtained by extending a target pixel area according to an embodiment of the present application.

Fig. 4(a) is a schematic diagram of an image processing method according to an embodiment of the present application.

Fig. 4(b) is a schematic diagram of an image processing method according to another embodiment of the present application.

Fig. 5 is a schematic diagram of a logical structure of an image processing device according to an embodiment of the present application.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

In order to solve the problem of poor image quality obtained by digital zooming, a multi-camera relay zoom technology is currently provided, that is, one image acquisition device can be equipped with multiple cameras, and the focal lengths of these cameras can be different. In the process of zooming the image acquisition device, the camera can be switched to the camera whose focal length matches the zoom ratio based on the zoom ratio, so as to use the switched camera for image acquisition. In this way, the image acquisition device can support larger The focal length range, and the images obtained during the zooming process have relatively high definition and good quality.

For camera switching in the optical zoom scene, since the focal lengths of the two cameras are different, the field of view is also different. During the zooming process, the field of view of the camera is constantly changing. For example, assume that the first camera is a short-focus camera with a larger field of view, and the second camera is a telephoto camera with a smaller field of view that is only a part of the field of view of the first camera. During the zooming process, digital zooming is first performed using images captured by the first camera, and when the zoom ratio matches the focal length of the second camera, the camera is switched to the second camera.

However, in the related art, when color processing the images output by the two cameras, the influence of the difference in the field of view of the cameras on the color of the output image is not considered, resulting in a large difference in the color of the same scene in the images output by the last two cameras. As shown in Figure 1, when the image acquisition device switches from camera 1 to camera 2, there is a large difference in the color of the image area with the same field of view in image 1 output by camera 1 and image 2 output by camera 2. Therefore, when the camera is switched, the color of the output image may jump, which seriously affects the user's experience.

In addition, due to the differences in the response characteristics of image sensors, filters, and lenses to light in different camera devices, there will be a large color deviation in the same field of view in the final image output by the camera.

Based on this, the embodiment of the present application provides an image processing method. In the scene where the camera is switched to realize optical zoom, the target pixel area can be determined from the first image captured by the first camera, and the target pixel area and the first image captured by the second camera can be determined. All or part of the pixel areas in the second image have the same field of view, and then use the color information obtained based on the target pixel area to perform color processing on the second image, by using the target pixel area that has the same field of view as the pixel area of the second image The color information of the second image is color processed, which can eliminate the influence of the field of view difference on the color of the image output by the two cameras, so as to avoid the large deviation in the color of the image output for the same scene before and after the switching of the two cameras .

The image processing method of the embodiment of the present application can be used to process images collected by various image acquisition devices equipped with at least two cameras, and the method can be executed by the image acquisition device or by a specialized image processing device.

The image acquisition device can be a mobile phone, a camera, a cloud platform, a drone, an unmanned car, and the like. In some embodiments, the image acquisition device may include a user interaction interface, and images acquired by the image acquisition device may be displayed on the user interaction interface for viewing by the user. In some embodiments, the image acquisition device may be communicatively connected with the control terminal, and the image acquisition device may send the acquired image to the control terminal to be displayed on the interactive interface of the control terminal for the user to view. For example, the image collected by the drone is sent to the user's mobile phone for the user to view.

The image acquisition device includes at least a first camera and a second camera, and the focal length supported by the first camera is not exactly the same as that supported by the second camera. Wherein, the first camera and the second camera can both be fixed-focus cameras, or both can be zoom cameras, or one can be a fixed-focus camera and the other can be a zoom camera. Not completely the same focal length means that the focal lengths supported by the first camera and the second camera are partly the same or completely different.

Specifically, as shown in Figure 2, the image processing method may include the following steps:

S202. For the same target shooting scene, switch the first camera of the image acquisition device to the second camera to achieve optical zoom;

In step S202, when using the image acquisition device to take pictures of the target shooting scene, the user may input a zoom command to perform zoom processing on the captured image. During the zooming process, the first image captured by the first camera can be digitally zoomed, and when the zoom ratio matches the focal length of the second camera, the first camera is switched to the second camera to realize optical zoom.

S204. Use the color information of the first image captured by the first camera before optical zoom to perform color processing on the second image captured by the second camera after optical zoom to obtain a target image, wherein the color information is based on the Obtaining a target pixel area in the first image, where the target pixel area has the same field of view as all or part of the pixel area in the second image;

In step S204, since the focal lengths of the first camera and the second camera are different, the fields of view are also different. Therefore, after the switching of the cameras is completed, the target pixel area can be determined from the first image captured by the first camera, and the target pixel area can have the same field of view as all or part of the pixel areas in the second image captured by the second camera , where having the same field of view means that the content of the scene displayed by the two image areas is the same, but the magnification is different. Then, color processing is performed on the second image by using the color information of the target pixel area to obtain the target image. Among them, the color processing can be white balance correction (AWB) processing on the image, or color correction (CCM) processing on the image, or color uniformity (Color Shading) processing on the image, etc. It is not difficult to understand that any Image processing to make the color of the image meet the needs of human eyes is within the protection scope of this application.

By using the color information of the target pixel area that has the same field of view as the pixel area in the second image to perform color processing on the second image, the color deviation of the image caused by the difference in the field of view of the two cameras can be eliminated, and the output image before and after camera switching can be guaranteed. The same scene in the same color is consistent.

Wherein, the first image may be the last frame of image taken by the first camera before zooming, the second image may be the first frame of image taken by the second camera after zooming, or the first image may be taken by the first camera before zooming For images other than the last image, the second image is the zoomed image of the first frame. Alternatively, the first image may be the last frame image captured by the first camera before zooming, and the second image may be all frame images captured by the second camera after zooming. As long as a pixel area having the same field of view as part or all of the pixel area of the second camera can be found in the first image, this embodiment of the present disclosure does not make a limitation.

S206. Output the target image, wherein the image output by the first camera before optical zooming is in the same color as the image area in the target image having the same field of view.

In step S206, after performing color processing on the second image to obtain the target image, the target image may be output, wherein the processed target image is consistent with the color of the image area having the same field of view in the image output by the first camera, Therefore, there is no jump in the color of images output by the two cameras for the same scene, thereby improving user experience.

Among them, the consistent color can mean that the colors of the image areas with the same field of view are exactly the same, or roughly the same, that is, there may be a certain deviation between the two, but the deviation is controlled within a small range to ensure the visual perception of the human eye. It seems that there is no color jump phenomenon.

In some embodiments, when using the color information of the first image captured by the first camera to perform color processing on the second image captured by the second camera, the white balance gain of the first image may be corrected based on the white balance gain of the first image. The second image undergoes white balance correction. Of course, it is also possible to use parameters for other color processing (such as color correction, color uniformity processing, etc.) The image can be processed more in line with the needs of the human eye.

In some embodiments, the focal length of the first camera may be shorter than the focal length of the second camera, and the angle of view of the first camera covers the angle of view of the second camera, that is, the second image shows the enlarged content of some pixel areas in the first image . Therefore, the target pixel area may have the same field of view as all pixel areas of the second image.

In some embodiments, the focal length of the first camera may be greater than the focal length of the second camera, and the angle of view of the second camera covers the angle of view of the first camera, that is, the first image is the enlarged content of some pixel areas in the second image. Therefore, The target pixel area may have the same field of view as the part of the pixel area in the second image.

Of course, the fields of view of the first camera and the second camera may not be exactly the same, that is, there is a certain overlapping field of view, and the target pixel area may be the image area corresponding to the overlapping field of view in the first image, which is different from the overlapping field of view in the second image. The corresponding image areas have the same field of view.

In some embodiments, the target pixel area may be determined from the first image based on the current zoom ratio and the relative positional relationship between the first camera and the second camera. As the zoom ratio changes, the content of the image output and displayed to the user will also change. Since the image captured by the second camera corresponds to the zoomed image, the content of the image captured by the second camera can be determined based on the zoom ratio. In addition, because there is a certain deviation in the center positions of the two cameras, the output content of the second camera is not completely the content of the center position of the first image, so it is necessary to determine the image captured by the second camera based on the deviation of the center position of the cameras. A corresponding area in the image to obtain the target pixel area.

In some embodiments, when performing white balance correction on the second image based on the white balance gain of the first image, the first white balance gain can be determined based on the target pixel area, for example, the target pixel area can be obtained The RGB value of the area, and the first white balance gain is determined based on the RGB value. Wherein, the method for determining the white balance gain may adopt a grayscale world method, a white balance algorithm based on machine learning, etc., which are not limited in this embodiment of the present disclosure.

Because there are certain differences in the response characteristics of the image sensors, filters, and lenses in the first camera and the second camera to light, for example, the sensitivity of the two cameras to red light is different, resulting in the same amount of red light incident on the two cameras. After the two cameras are integrated, the intensity of the light signals sensed by the image sensors of the two cameras will be different, resulting in color deviation in the images captured by the two cameras. In order to eliminate the color deviation caused by the difference of the light response characteristics of the two cameras, the color mapping relationship between the two cameras can be calibrated in advance. After determining the first white balance gain based on the target pixel area in the first image, the first white balance gain can be mapped to the color space of the second camera according to the color mapping relationship between the first camera and the second camera, and the second White balance gain. Then use the second white balance gain to perform white balance correction on the second image to obtain and output the target image.

Through the above method, it can not only eliminate the image color deviation caused by the difference in the field of view of the two cameras, but also eliminate the image color deviation caused by the difference in the light response characteristics of the two cameras, so that the same scene in the image output before and after the camera is switched The color is consistent.

In addition, in some embodiments, after the first white balance gain is determined based on the target pixel area, white balance correction processing may be performed on the first image based on the first white balance gain, and then the white balance processed first The image is digitally zoomed and output. In related technologies, when digital zooming, the change of field of view brought about by the change of zoom ratio is not considered, and the color of the scene has always maintained the overall color, and the output image during digital zooming is not its real field of view Corresponds to the true color. Compared with directly determining the white balance correction gain based on the full image of the first image and correcting the image output by the first camera, the embodiment of the present application can dynamically determine the target pixel area based on the magnification, and use the target pixel area to determine the first white The balance gain is used to correct the white balance of the image output by the first camera, so that the image output during the digital zoom process is more in line with the real color, ensuring that the perceived field of view is consistent before and after the camera is switched.

In some embodiments, since the shooting scene may be complicated, if the target pixel area consistent with the display content of the second image is determined in the first image directly according to the zoom ratio and the relative positional relationship of the two cameras, and based on the target pixel area Determine the first white balance gain. There may be scenes corresponding to the determined target pixel area that are not suitable for determining the white balance gain. For example, it is a white wall, or the determined target pixel area is not accurate enough, and the second image shows There is a certain deviation in the content, so the final white balance gain based on the target pixel area is not very accurate. In order to avoid the above problems as much as possible, as shown in Figure 3, when determining the white balance gain, based on the target pixel area, a certain range can be extended around the target pixel area according to the preset expansion ratio, and the surrounding area of the target pixel area The pixel area is expanded to obtain the expanded pixel area, and then the first white balance gain can be determined based on the RGB values of the expanded pixel area. Wherein, the expansion ratio can be set based on actual requirements, for example, it can be set to 5%, 7% and so on. The field of view of the expanded pixel area is relatively similar to that of the second image as a whole, and can ensure that the determined first white balance gain is more accurate.

Generally, at different color temperatures, the white balance gain for performing white balance correction on the image will be different, and the corresponding relationship between the color temperature and the white balance gain can be determined in advance. In some embodiments, when the second white balance gain is obtained based on the color mapping relationship between the first camera and the second camera and the first white balance gain, the first color temperature corresponding to the first white balance gain can be determined first, and then according to the first white balance gain The color mapping relationship between the first camera and the second camera maps the first color temperature to the color space of the second camera to obtain the second color temperature, and then determines the white balance gain corresponding to the second color temperature as the second white balance gain. The color mapping relationship is related to the spectral response characteristics of the two cameras, which can be established during the tuning process, or can be determined according to the spectral response deviations of the image sensors, lenses, and filters of the two cameras. Based on the established color mapping relationship, the photographing device can automatically look up a table to determine the second color temperature corresponding to the first color temperature.

In some embodiments, after determining the second white balance gain, the third white balance gain can be further determined based on the RGB value of the second image, and then the second white balance gain and the third white balance gain can be combined to adjust the second camera The multi-frame second images collected are output after white balance correction to obtain multi-frame target images, so that the color of the multi-frame target images output by the second camera changes from the color corrected by the second white balance gain to the color corrected by the third white balance gain. After the color transition.

For example, in order to avoid the color deviation of the two frames of images output before and after camera switching, and the phenomenon of color mutation, you can first make the first frame image output by the second camera consistent with the image output by the first camera, for example , only use the second white balance gain to correct it, and then gradually add the influence of the third white balance gain (for example, two white balance gains can be mixed according to a certain ratio, gradually reduce the ratio of the second white balance gain, increase The ratio of the third white balance gain, to only use the third white balance gain to correct the second image), so that the subsequent output image gradually transitions from the color of the image output by the first camera to the color transition of the image output by the second camera, Until the final presented color is the real color of the image output by the second camera, that is, it is only corrected by the third white balance gain. In this way, it is possible to avoid the need for the first camera to work all the time, and to determine the first white balance gain based on the collected first image and send it to the second camera.

In some implementations, the target pixel area may have the same field of view as the reference pixel area in the second image, where the reference pixel area may be the entire pixel area of the second image, or a partial pixel area in the second image . When performing white balance correction on the second image based on the white balance gain of the first image, the fourth white balance gain can be determined based on the first image, and then the fifth white balance gain can be determined based on the second image, according to The color deviation between the target pixel area after white balance correction using the fourth white balance gain and the reference pixel area after white balance correction using the fifth white balance gain determines the offset coefficient, which is used to characterize the acquisition of the two cameras In the image with the same field of view, there is color deviation after white balance correction, and then the fifth white balance gain can be adjusted based on the offset coefficient to obtain the sixth white balance gain, and then use the sixth white balance Gain performs white balance correction on the second image to obtain the target image and output it.

In some embodiments, the target pixel area and the reference pixel area can be pre-calibrated according to the relative positional relationship between the first camera and the second camera. The regions in the images captured by the two cameras that are completely aligned with the real scene are used as the target pixel region and the reference pixel region.

In some embodiments, when determining the fourth white balance gain based on the first image, the fourth white balance gain may be determined directly based on the RGB values of the full image of the first image. In some embodiments, in order to eliminate the influence of field of view difference when determining the fourth white balance gain, the fourth white balance gain may be determined only according to the RGB values of the target pixel area.

Likewise, for the fifth white balance gain, in some embodiments, it may be determined according to RGB values of the full image of the second image. In some embodiments, it may also be determined according to the RGB values of the reference pixel area, so as to eliminate the influence of the difference in the field of view.

Of course, after the white balance correction is performed on the image captured by the camera, further color correction is usually required to eliminate the influence of the white balance correction on the color of the image. For example, the saturation of the image after the white balance correction is often low. In order to make the color of the image area with the same field of view in the final output image of the two cameras as close as possible, when determining the offset coefficient, the influence of color correction on the image color can be considered at the same time. For example, after using the fourth white balance gain to perform white balance correction on the target pixel area, color correction may be further performed on the white balance corrected target pixel area, and after using the fifth white balance gain to perform white balance correction on the reference pixel area, The color correction can be further performed on the reference pixel area after white balance correction, and then the offset coefficient can be determined according to the color deviation between the target pixel area after color correction and the reference pixel area after color correction. By determining the offset coefficient in this way, it can be more It truly reflects the deviation between the final output image of the first camera and the final output image of the second camera, so that after correcting the second image with the white balance gain adjusted by the offset coefficient, the obtained target image is different from the first The color of the same scene in the image output by the camera is closer.

The correction matrix and color temperature for color correction of an image are also interrelated. In order to ensure the accuracy of the correction result, the correction matrix is often different under different color temperatures. Therefore, in some embodiments, when performing color correction on the target pixel area, a correction matrix may be determined based on the third color temperature corresponding to the fourth white balance gain, and then use the correction matrix to perform color correction on the target pixel area. Similarly, when performing color correction on the reference pixel area, a correction matrix may be determined based on the fourth color temperature corresponding to the fifth white balance gain, and then use the correction matrix to perform color correction on the reference pixel area.

In some embodiments, it is also possible to perform white balance correction on multiple frames of second images collected by the second camera based on the fifth white balance gain and the sixth white balance gain to obtain multiple frames of target images; so that the multiple frames output by the second camera The color of the frame target image transitions from the color corrected by the sixth white balance gain to the color corrected by the fifth white balance gain.

For example, in order to avoid the color deviation of the two frames of images output before and after camera switching, and the phenomenon of color mutation, you can first make the first frame image output by the second camera consistent with the image color output by the first camera, for example, only Use the sixth white balance gain to correct it, and then gradually add the influence of the fifth white balance gain (for example, two white balance gains can be mixed in a certain proportion, gradually reduce the proportion of the sixth white balance gain, increase the fifth white balance gain, until only the fifth white balance gain is used to correct the second image), so that the subsequent output image gradually transitions from the color of the image output by the first camera to the color of the output image of the second camera, until finally The presented color is the real color of the image output by the second camera, that is, only corrected by the sixth white balance gain. In this way, it can avoid that the first camera needs to work all the time, and the fourth white balance gain is determined based on the collected first image and sent to the second camera.

In order to further explain the image processing method of this embodiment, it will be explained in conjunction with several specific embodiments below.

Assuming that the image acquisition device includes a short-focus camera 1 and a telephoto camera 2, during the zooming process, when switching from camera 1 to camera 2 (similar to switching from camera 2 to camera 1), due to the angle of view and contrast of camera 1 and camera 2 There are differences in the response characteristics of light, resulting in a large difference in the color of the image area with the same field of view in the output image before and after the camera is switched. In order to solve the above problems, the following image processing methods are provided to make the output image color consistent before and after camera switching.

Embodiment one:

The image area with the same field of view as image 2 can be determined from image 1 collected by camera 1 based on the zoom ratio, and the white balance gain can be determined based on the image area to correct image 2, so that the two images caused by different fields of view can be eliminated. The image output by the camera is different.

As shown in Figure 4(a), it specifically includes the following steps:

(1) After the camera 1 captures the image 1, it can determine the ROI area in the image 1 based on the current zoom ratio and the positional relationship between the two cameras, wherein the ROI area and the image 2 captured by the camera 2 have the same viewing angle. field.

(2) Expand the ROI area in image 1 according to a preset ratio (such as 5%) to obtain the expanded ROI area. The expanded ROI area is slightly larger than the field of view of image 2, but the overall near.

(3) Determine white balance gain 1 and color temperature 1 based on the RGB values of the expanded ROI area. Compared with directly using the ROI area to determine the white balance gain, the result obtained by expanding the ROI area and then using it to determine the white balance gain will be more accurate.

(4) After the white balance gain 1 is determined, the white balance correction of image 1 can be performed by using the white balance gain 1, and then based on the zoom ratio, after sampling from the central area of the white balance corrected image 1, a digital zoom image is obtained and output .

(5) Send white balance gain 1 and color temperature 1 to camera 2, and camera 2 can map color temperature 1 to the color space of camera 2 based on the color mapping relationship between camera 1 and camera 2, obtain color temperature 2, and determine the color temperature White balance gain of 2 under 2.

(6) The camera 2 can also determine the white balance gain 3 and the color temperature 3 based on the RGB values of the image 2 .

(7) White balance correction can be performed on the multi-frame image 2 output by the camera in combination with white balance gain 2 and white balance gain 3, so that the color of the output optical zoom image is corrected from the color corrected by white balance gain 2 to white balance gain 3 The final color transition is consistent with the color of image 1 captured by camera 1 when the camera first switches, and then gradually transitions to the real color of image 2 captured by camera 2, so that there will be no color jump, but a slow transition .

Embodiment two:

The ROI area whose field of view is completely aligned can be selected from the images collected by the two cameras, and the offset coefficient is determined based on the color deviation of the ROI area after the camera performs balance correction on the ROI area, and the image color is adjusted using the offset coefficient. Adjust to make the color of the images output by the two cameras consistent for the same scene. As shown in Figure 4(b), it specifically includes the following steps:

(1) When camera 1 is switched to camera 2 based on the zoom ratio, ROI area 1 can be determined from image 1 collected by camera 1, and ROI area 2 can be determined from image 2 collected by camera 2, wherein ROI area 1 and ROI Region 2 has the same field of view and is a completely aligned region of the two cameras. ROI region 1 and ROI region can be pre-calibrated based on the positional relationship of the two cameras.

(2) White balance gain 1 and color temperature 1 can be determined based on the RGB value of ROI area 1, and the white balance correction of ROI area 1 can be performed using white balance gain 1, and then the color correction matrix CCM1 corresponding to color temperature 1 can be determined, and white balance correction can be performed using CCM1 ROI area 1 is further color corrected.

(3) White balance gain 2 and color temperature 2 can be determined based on the RGB value of ROI area 2, and the white balance correction of ROI area 2 can be performed using white balance gain 2, and then the color correction matrix CCM2 corresponding to color temperature 2 can be determined, and white balance correction can be performed using CCM2 ROI area 2 for further color correction.

(4) Determine the offset coefficient according to the color difference between ROI region 1 after white balance correction+color correction and ROI region 2 after white balance correction+color correction.

(5) Use the offset coefficient to correct the white balance gain 2 to obtain the white balance gain 3, and use the white balance gain 3 to correct the white balance of the image 2 collected by the camera 2 and output it, so that the image 2 and the image 1 have the same visual The image area of the field is uniform in color.

(6) In addition, for the subsequent multi-frame images output by camera 2, it can also be corrected in combination with white balance gain 2 and white balance gain 3, so that the image output by camera 2 gradually changes from the corrected color of white balance gain 3 (that is, The color of the image output by the first camera gradually transitions to the color corrected by the white balance gain 2, until finally the real color of the image captured by the camera 2 is completely presented. In this way, camera 1 can be prevented from working all the time, and the white balance gain 1 determined based on image 1 can be sent to camera 2.

Corresponding to the above method, the embodiment of the present application also provides an image processing device. As shown in FIG. computer program, the processor 51 can implement the following steps when executing the computer program:

Using the color information of the first image captured by the first camera to perform color processing on the second image captured by the second camera to obtain a target image, wherein the color information is based on the target pixel area in the first image Acquiring that the target pixel area has the same field of view as all or part of the pixel area of the second image;

In some embodiments, the processor is configured to use the color information of the first image captured by the first camera to perform color processing on the second image captured by the second camera, specifically for:

Performing white balance correction on the second image based on the white balance gain for performing white balance correction on the first image.

In some embodiments, when the processor is configured to perform white balance correction on the second image based on the white balance gain for performing white balance correction on the first image, it is specifically used to:

determining a first white balance gain based on the target pixel area;

The second white balance gain is obtained based on the color mapping relationship between the first camera and the second camera and the first white balance gain, wherein the color mapping relationship is based on the spectral response characteristic of the first camera and the first white balance gain. The spectral response characteristics of the second camera are determined;

Perform white balance correction on the second image by using the second white balance gain.

In some embodiments, the processor is configured to use the target pixel area to determine the first white balance gain, specifically for:

obtaining an extended pixel area based on a preset expansion ratio and the target pixel area, where the extended pixel area includes the target pixel area and pixel areas around the target pixel area;

The first white balance gain is determined by using RGB values of the extended pixel area.

In some embodiments, when the processor is configured to obtain the second white balance gain based on the color mapping relationship between the first camera and the second camera and the first white balance gain, it is specifically configured to:

determining a first color temperature corresponding to the first white balance gain;

determining a second color temperature based on the color mapping relationship and the first color temperature;

A white balance gain corresponding to the second color temperature is determined as the second white balance gain.

In some embodiments, the processor is further configured to:

determining a third white balance gain based on the RGB values of the second image;

Based on the second white balance gain and the third white balance gain, white balance correction is performed on multiple frames of second images collected by the second camera to obtain multiple frames of target images; so that the output of the second camera The color of the multiple frames of the target image transitions from the color corrected by the second white balance gain to the color corrected by the third white balance gain.

In some embodiments, the viewing angle of the first camera covers the viewing angle of the second camera, and the target pixel area has the same field of view as all pixel areas of the second image; or

The viewing angle of the second camera covers the viewing angle of the first camera, and the target pixel area has the same field of view as a partial pixel area of the second image.

In some embodiments, the target pixel area is determined based on a zoom factor and a relative positional relationship between the first camera and the second camera.

In some embodiments, the target pixel area has the same field of view as the reference pixel area in the second image;

When the processor is used to perform white balance correction on the second image based on the white balance gain for performing white balance correction on the first image, it is specifically used to:

determining a fourth white balance gain based on the first image;

determining a fifth white balance gain based on the second image;

determining an offset coefficient based on a color deviation between a target pixel area after white balance correction by using the fourth white balance gain and a reference pixel area after white balance correction by using the fifth white balance gain;

determining the sixth white balance gain based on the fifth white balance gain and the offset coefficient;

Perform white balance correction on the second image by using the sixth white balance gain.

In some embodiments, when the processor is configured to determine the fourth white balance gain based on the first image, it is specifically configured to:

determining the fourth white balance gain based on RGB values of the target pixel region in the first image; and/or

determining a fifth white balance gain based on the second image, comprising:

The fifth white balance gain is determined based on RGB values of the reference pixel region in the second image.

In some embodiments, the processor is configured to use the fourth white balance gain to perform white balance correction based on the color deviation between the target pixel area after white balance correction and the fifth white balance gain. When determining the offset coefficient, it is specifically used for:

performing color correction on the target pixel area after white balance correction and the reference pixel area after white balance correction;

The offset coefficient is determined based on a color deviation between the color-corrected target pixel area and the color-corrected reference pixel area.

In some embodiments, the correction matrix for performing color correction on the target pixel area is determined based on a third color temperature, and the third color temperature is determined based on the fourth white balance gain;

A correction matrix for performing color correction on the white balance corrected reference pixel area is determined based on a fourth color temperature, and the fourth color temperature is determined based on a fifth white balance gain.

In some embodiments, the target pixel area and the reference pixel area are pre-calibrated and obtained based on the relative positional relationship between the first camera and the second camera.

In some embodiments, the processor is further configured to:

Based on the fifth white balance gain and the sixth white balance gain, white balance correction is performed on multiple frames of second images collected by the second camera to obtain multiple frames of target images; so that the output of the second camera The color of the multiple frames of the target image transitions from the color corrected by the sixth white balance gain to the color corrected by the fifth white balance gain.

Further, an image acquisition device is provided in an embodiment of the present application, and the image acquisition device includes a first camera, a second camera, and the image processing apparatus according to any one of the above embodiments. The image acquisition device can hold various devices equipped with at least two cameras, such as a pan-tilt camera, a mobile phone, and a drone.

Correspondingly, the embodiments of this specification further provide a computer storage medium, where a program is stored in the storage medium, and when the program is executed by a processor, the method in any of the foregoing embodiments is implemented.

Embodiments of the present description may take the form of a computer program product embodied on one or more storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) having program code embodied therein. Computer usable storage media includes both volatile and non-permanent, removable and non-removable media, and may be implemented by any method or technology for information storage. Information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for computers include, but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridge, tape magnetic disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

As for the device embodiment, since it basically corresponds to the method embodiment, for related parts, please refer to the part description of the method embodiment. The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without creative effort.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or order between them. The term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements but also other elements not expressly listed elements, or also elements inherent in such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

The methods and devices provided by the embodiments of the present invention have been described in detail above. The principles and implementation methods of the present invention have been explained by using specific examples in this paper. The descriptions of the above embodiments are only used to help understand the methods and methods of the present invention. core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation and scope of application. In summary, the content of this specification should not be construed as limiting the present invention .

Claims

An image processing method, characterized in that the method comprises:

For the same target shooting scene, switch the first camera of the image acquisition device to the second camera to achieve optical zoom;

Using the color information of the first image captured by the first camera before the optical zoom to perform color processing on the second image captured by the second camera after the optical zoom to obtain a target image, wherein the color information is based on the first Acquiring a target pixel area in the image, where the target pixel area has the same field of view as all or part of the pixel area of the second image;

Outputting the target image, wherein the image output by the first camera before the optical zoom is in the same color as the image area in the target image having the same field of view.
The method according to claim 1, wherein the color processing of the second image captured by the second camera using the color information of the first image captured by the first camera comprises:

Performing white balance correction on the second image based on the white balance gain for performing white balance correction on the first image.
The method according to claim 2, wherein performing white balance correction on the second image based on a white balance gain for performing white balance correction on the first image comprises:

determining a first white balance gain based on the target pixel area;

The second white balance gain is obtained based on the color mapping relationship between the first camera and the second camera and the first white balance gain, wherein the color mapping relationship is based on the spectral response characteristic of the first camera and the first white balance gain. The spectral response characteristics of the second camera are determined;

Perform white balance correction on the second image by using the second white balance gain.
The method according to claim 3, wherein determining the first white balance gain based on the target pixel area comprises:

obtaining an extended pixel area based on a preset expansion ratio and the target pixel area, where the extended pixel area includes the target pixel area and pixel areas around the target pixel area;

The first white balance gain is determined by using RGB values of the extended pixel area.
The method according to claim 3, wherein obtaining the second white balance gain based on the color mapping relationship between the first camera and the second camera and the first white balance gain comprises:

determining a first color temperature corresponding to the first white balance gain;

determining a second color temperature based on the color mapping relationship and the first color temperature;

A white balance gain corresponding to the second color temperature is determined as the second white balance gain.
The method according to claim 3, further comprising:

determining a third white balance gain based on the RGB values of the second image;

Based on the second white balance gain and the third white balance gain, white balance correction is performed on multiple frames of second images collected by the second camera to obtain multiple frames of target images; so that the output of the second camera The color of the multiple frames of the target image transitions from the color corrected by the second white balance gain to the color corrected by the third white balance gain.
The method according to claim 3, wherein the viewing angle of the first camera covers the viewing angle of the second camera, and the target pixel area has the same field of view as all pixel areas of the second image; or

The viewing angle of the second camera covers the viewing angle of the first camera, and the target pixel area has the same field of view as a partial pixel area of the second image.
The method according to any one of claims 3-7, wherein the target pixel area is determined based on a zoom magnification and a relative positional relationship between the first camera and the second camera.
The method according to claim 2, wherein the target pixel area has the same field of view as the reference pixel area in the second image;

Performing white balance correction on the second image based on a white balance gain for performing white balance correction on the first image, including:

determining a fourth white balance gain based on the first image;

determining a fifth white balance gain based on the second image;

determining an offset coefficient based on the color deviation between the target pixel area after white balance correction by using the fourth white balance gain and the reference pixel area after white balance correction by using the fifth white balance gain;

determining a sixth white balance gain based on the fifth white balance gain and the offset coefficient;

Perform white balance correction on the second image by using the sixth white balance gain.
The method according to claim 9, wherein determining a fourth white balance gain based on the first image comprises:

determining the fourth white balance gain based on RGB values of the target pixel region in the first image; and/or

determining a fifth white balance gain based on the second image, comprising:

The fifth white balance gain is determined based on RGB values of the reference pixel region in the second image.
The method according to claim 9, characterized in that based on the difference between the target pixel area after white balance correction using the fourth white balance gain and the reference pixel area after white balance correction using the fifth white balance gain Color Bias determines offset factors, including:

performing color correction on the target pixel area after white balance correction and the reference pixel area after white balance correction;

The offset coefficient is determined based on a color deviation between the color-corrected target pixel area and the color-corrected reference pixel area.
The method according to claim 11, wherein the correction matrix for performing color correction on the target pixel area is determined based on a third color temperature, and the third color temperature is determined based on the fourth white balance gain;

A correction matrix for performing color correction on the white balance corrected reference pixel area is determined based on a fourth color temperature, and the fourth color temperature is determined based on a fifth white balance gain.
The method according to claim 9, wherein the target pixel area and the reference pixel area are pre-calibrated and obtained based on the relative positional relationship between the first camera and the second camera.
The method according to claim 9, characterized in that the method further comprises:

Based on the fifth white balance gain and the sixth white balance gain, white balance correction is performed on multiple frames of second images collected by the second camera to obtain multiple frames of target images; so that the output of the second camera The color of the multiple frames of the target image transitions from the color corrected by the sixth white balance gain to the color corrected by the fifth white balance gain.
An image processing device, characterized in that the device includes a processor, a memory, and a computer program stored in the memory for execution by the processor, and the processor can implement the following steps when executing the computer program:

For the same target shooting scene, switch the first camera of the image acquisition device to the second camera to achieve optical zoom;

Using the color information of the first image captured by the first camera before the optical zoom to perform color processing on the second image captured by the second camera after the optical zoom to obtain a target image, wherein the color information is based on the first Acquiring a target pixel area in the image, where the target pixel area has the same field of view as all or part of the pixel area of the second image;

Outputting the target image, wherein the image output by the first camera before the optical zoom is in the same color as the image area in the target image having the same field of view.
The device according to claim 15, wherein the processor is configured to use the color information of the first image captured by the first camera to perform color processing on the second image captured by the second camera, specifically for :

Performing white balance correction on the second image based on the white balance gain for performing white balance correction on the first image.
The device according to claim 16, wherein, when the processor is configured to perform white balance correction on the second image based on the white balance gain for performing white balance correction on the first image, it is specifically used to:

determining a first white balance gain based on the target pixel area;

The second white balance gain is obtained based on the color mapping relationship between the first camera and the second camera and the first white balance gain, wherein the color mapping relationship is based on the spectral response characteristic of the first camera and the first white balance gain. The spectral response characteristics of the second camera are determined;

Perform white balance correction on the second image by using the second white balance gain.
The device according to claim 17, wherein the processor is configured to determine the first white balance gain based on the target pixel area, specifically for:

obtaining an extended pixel area based on a preset expansion ratio and the target pixel area, where the extended pixel area includes the target pixel area and pixel areas around the target pixel area;

The first white balance gain is determined by using RGB values of the extended pixel area.
The device according to claim 17, wherein the processor is configured to obtain the second white balance gain based on the color mapping relationship between the first camera and the second camera and the first white balance gain , specifically for:

determining a first color temperature corresponding to the first white balance gain;

determining a second color temperature based on the color mapping relationship and the first color temperature;

A white balance gain corresponding to the second color temperature is determined as the second white balance gain.
The device according to claim 17, wherein the processor is further configured to:

determining a third white balance gain based on the RGB values of the second image;

Based on the second white balance gain and the third white balance gain, white balance correction is performed on multiple frames of second images collected by the second camera to obtain multiple frames of target images; so that the output of the second camera The color of the multiple frames of the target image transitions from the color corrected by the second white balance gain to the color corrected by the third white balance gain.
The device according to claim 17, wherein the viewing angle of the first camera covers the viewing angle of the second camera, and the target pixel area has the same field of view as all pixel areas of the second image; or

The viewing angle of the second camera covers the viewing angle of the first camera, and the target pixel area has the same field of view as a partial pixel area of the second image.
The device according to any one of claims 17-21, wherein the target pixel area is determined based on a zoom magnification and a relative positional relationship between the first camera and the second camera.
The device according to claim 16, wherein the target pixel area has the same field of view as the reference pixel area in the second image;

When the processor is used to perform white balance correction on the second image based on the white balance gain for performing white balance correction on the first image, it is specifically used to:

determining a fourth white balance gain based on the first image;

determining a fifth white balance gain based on the second image;

determining an offset coefficient based on a color deviation between a target pixel area after white balance correction by using the fourth white balance gain and a reference pixel area after white balance correction by using the fifth white balance gain;

determining a sixth white balance gain based on the fifth white balance gain and the offset coefficient;

Perform white balance correction on the second image by using the sixth white balance gain.
The device according to claim 23, wherein when the processor is configured to determine the fourth white balance gain based on the first image, it is specifically configured to:

determining the fourth white balance gain based on RGB values of the target pixel region in the first image; and/or

determining a fifth white balance gain based on the second image, comprising:

The fifth white balance gain is determined based on RGB values of the reference pixel region in the second image.
The device according to claim 23, wherein the processor is configured to perform white balance correction based on the target pixel area using the fourth white balance gain and the white balance correction using the fifth white balance gain When determining the offset coefficient, the color deviation of the reference pixel area is specifically used for:

performing color correction on the target pixel area after white balance correction and the reference pixel area after white balance correction;

The offset coefficient is determined based on a color deviation between the color-corrected target pixel area and the color-corrected reference pixel area.
The device according to claim 25, wherein the correction matrix for performing color correction on the target pixel area is determined based on a third color temperature, and the third color temperature is determined based on the fourth white balance gain;

A correction matrix for performing color correction on the white balance corrected reference pixel area is determined based on a fourth color temperature, and the fourth color temperature is determined based on a fifth white balance gain.
The device according to claim 23, wherein the target pixel area and the reference pixel area are pre-calibrated and obtained based on the relative positional relationship between the first camera and the second camera.
The device according to claim 23, wherein the processor is further configured to:

Based on the fifth white balance gain and the sixth white balance gain, white balance correction is performed on multiple frames of second images collected by the second camera to obtain multiple frames of target images; so that the output of the second camera The color of the multiple frames of the target image transitions from the color corrected by the sixth white balance gain to the color corrected by the fifth white balance gain.
An image acquisition device, characterized in that the image acquisition device comprises a first camera, a second camera, and the image processing device according to any one of claims 15-28.
A computer-readable storage medium, wherein the method according to any one of claims 1-14 can be realized when the computer-readable storage medium is executed.