WO2022047701A1

WO2022047701A1 - Image processing method and apparatus

Info

Publication number: WO2022047701A1
Application number: PCT/CN2020/113251
Authority: WO
Inventors: 高文良; 周游; 刘洁
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2020-09-03
Filing date: 2020-09-03
Publication date: 2022-03-10
Also published as: CN113348489A

Abstract

Provided are an image processing method and apparatus, capable of generating an image having a large field of view, improving the viewing experience of a user. The method comprises: obtaining a first image of a scene, captured by a first camera of an electronic device; obtaining a second image of the scene, captured by a second camera of the electronic device; the observation range of the first camera is larger than the observation range of the second camera; furthermore, the resolution of the first camera is lower than that of the second camera, and/or the imaging of the first camera is as a non-color image and the imaging of the second camera is as a color image; determining in said first image a first pixel region corresponding to a target object in said scene, and determining in said second image a second pixel region corresponding to a target object in the scene; on the basis of the second pixel region in the second image, performing image processing on the first pixel region in the first image, to obtain a processed first image.

Description

Image processing method and device

Copyright notice

The disclosure of this patent document contains material that is subject to copyright protection. This copyright belongs to the copyright owner. The copyright owner has no objection to the reproduction by anyone of the patent document or the patent disclosure as it exists in the official records and archives of the Patent and Trademark Office.

technical field

The present application relates to the field of image processing, and more particularly, to an image processing method and apparatus.

Background technique

There are more and more electronic devices equipped with cameras, such as drones, unmanned cars, augmented reality (VR)/(virtual reality, AR) glasses, etc. These electronic devices have greatly enriched people's lives. Life.

The camera can capture image information of the object, and transmit the captured image information to the user, and the user can operate the electronic device according to the image information. However, the field of view of the camera on the current electronic device is not very large, which restricts the user's field of view and affects the user's visual experience.

SUMMARY OF THE INVENTION

The present application provides an image processing method and device, which can generate an image with a large field of view and improve the user's visual experience.

In a first aspect, an image processing method is provided, which includes acquiring a first image captured by a first camera of an electronic device of a scene; acquiring a second image captured by a second camera of the electronic device of the scene; wherein, the The observation range of the first camera is larger than the observation range of the second camera; and the resolution of the first camera is lower than the resolution of the second camera, and/or the imaging of the first camera is an achromatic image, the image of the second camera is a color image;

determining a first pixel area corresponding to the target object in the scene in the first image, and determining a second pixel area corresponding to the target object in the scene in the second image;

Based on the second pixel area in the second image, image processing is performed on the first pixel area in the first image to obtain a processed first image.

In a second aspect, an image processing apparatus is provided, comprising: a memory for storing a computer program; a processor for invoking the computer program, and when the computer program is executed by the processor, causes the apparatus to execute The steps are as follows: obtaining a first image captured by a first camera of an electronic device on the scene; obtaining a second image captured by a second camera of the electronic device on the scene; wherein, the observation range of the first camera is greater than the observation range of the second camera; and, the resolution of the first camera is lower than the resolution of the second camera, and/or the imaging of the first camera is an achromatic image, the second camera The imaging of the camera is a color image;

A third aspect provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed, implements the method provided in the first aspect.

In a fourth aspect, there is provided a computer program product comprising instructions that, when executed by a computer, cause the computer to perform the method provided by the first aspect.

Based on the above technical solution, since the second image is a color image and/or the second image is a high-resolution image, the second image has a better visual effect, and the observation range of the first camera is larger than that of the second camera. , therefore, the field of view corresponding to the first image is larger than the field of view corresponding to the second image. The present application can fuse the second image with better visual effect and the first image with large field of view to obtain the processed first image with both large field of view and better imaging effect, so that users can have better visual experience.

Description of drawings

FIG. 1 is a schematic diagram of an application scenario provided by an embodiment of the present application.

FIG. 2 is a schematic flowchart of an image processing method provided by an embodiment of the present application.

FIG. 3 and FIG. 4 are schematic diagrams of different overlapping situations of the first image and the second image provided by the embodiments of the present application.

FIG. 5 is a schematic diagram of changes in attitude angles before and after stabilization of the first image provided by an embodiment of the present application.

FIG. 6 is a schematic diagram of a processed first image provided by an embodiment of the present application.

FIG. 7 is a schematic diagram of an image segmented using superpixels according to an embodiment of the present application.

FIG. 8 is a schematic diagram of an image mapping process provided by an embodiment of the present application.

FIG. 9 is a schematic diagram of an image edge extraction provided by an embodiment of the present application.

FIG. 10 is a schematic diagram of segmenting an edge segment according to an embodiment of the present application.

FIG. 11 is a schematic diagram of finding a line segment in a second image that matches a line in the first image according to an embodiment of the present application.

FIG. 12 is a schematic block diagram of an image processing apparatus provided by an embodiment of the present application.

detailed description

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

The electronic devices in the embodiments of the present application may be various electronic devices with cameras, such as mobile platforms, such as drones, unmanned ships, drone vehicles, and vehicles that can be driven automatically; they may also be smart wearable devices, For example, VR/AR glasses, etc.; it can also be smart terminal devices, such as mobile phones, tablet computers, etc.

The electronic device in this embodiment of the present application may be the above-mentioned movable platform or other electronic device that is communicatively connected to the wearable device. For example, the electronic device may be an unmanned aerial vehicle, or a control terminal that communicates with the unmanned aerial vehicle, such as a remote control used in conjunction with the unmanned aerial vehicle, or a mobile phone that communicates with the unmanned aerial vehicle through wired or wireless communication.

The camera in the embodiments of the present application may also be referred to as a camera, an image sensor, or the like. Examples may be RGB color cameras, infrared cameras, binocular cameras, multi-eye cameras, multi-spectral cameras, and the like.

For the convenience of description, the following description takes a movable platform as an example.

A main camera is usually set on the movable platform, and the main camera can photograph the surrounding scene of the movable platform and obtain image information of the scene during the movement of the movable platform. The removable platform can transmit the image information to the user device, and display the image on the user device. This image can be a first-person view (FPV) image, and the user can perform corresponding control operations based on the image information. For example, the movement direction, movement speed, stop position, etc. of the movable platform are controlled, or the shooting angle and exposure parameters of the main camera of the movable platform are adjusted. Specifically, this operation can be performed by the user touching a control button of the user equipment or touching the screen of the user equipment.

The main camera can be a color camera, and the color camera can capture a color image, and the color image can make the user have a more intuitive perception of the surrounding environment.

In addition, in order to further improve the user experience, the main camera can be a high-resolution camera, so that high-resolution high-definition images can be captured, so that the user has a better viewing experience of the images in the scene. Clear images help users identify objects of interest.

The main purpose of the main camera is to obtain high-resolution, vivid-color imaging. Considering the imaging cost of the camera, a camera with high imaging quality has a limited angle of view (FOV). In order to take into account high imaging and a large field of view, and form a large-format image with clear imaging and vivid colors, the design and manufacturing cost of the camera is very high. In order to integrate the cost and the user's perception, the observation range of the main camera's imaging is limited. Therefore, the operation based on the image captured by the main camera limits the user's field of view and affects the user's observation of the target object in a wider range of the scene. .

Based on this, the embodiments of the present application provide an image processing method, aiming at increasing the field of view of the image in the limited observation range of the camera, ensuring the quality of the processed image to a certain extent, and improving the user's visual experience.

After research, it is found that a perception camera for perceiving the surrounding environment can also be set on the movable platform, and this camera is mainly used to obtain sensory data around the drone. For example, binocular cameras, multi-eye cameras, infrared sensing data, etc. Usually, the movable platform can obtain the depth information and temperature information of the objects in the scene with the help of these perception cameras, and then assist in the execution of obstacle avoidance, tracking, positioning and other functions during the movement process.

The data collected by these perception cameras will be directly input into the data processing link to extract the target objects in the scene, or generate scene depth information. Perceptual cameras only need to obtain accurate depth information or temperature information. The image data collected by these perceptual cameras does not need to generate images to display to users. Therefore, the data acquisition and data processing processes of these perceptual cameras are not in electronic devices. Designed for better visual effects of images.

It is worth noting that, in order to obtain information in a scene to a greater extent, these perception cameras often have as large an observation range as possible, so that a wider range of scenes can be photographed and corresponding imaging data can be obtained. That is, the perception effect of the image captured by the perception camera is not as good as that of the image captured by the main camera, but the field of view of the camera is larger than the field of view of the main camera used for shooting.

As shown in FIG. 1 , the drone includes a main camera 110 for capturing images and a camera 120 for sensing the surrounding environment.

The resolution of the image captured by the main camera 110 is higher than the resolution of the image captured by the camera 120 , but the field of view of the camera 120 is larger than that of the main camera 110 .

Camera 120 may also be referred to as a vision sensor.

Based on the above considerations, the present application proposes to fuse the image captured by the camera 120 with the image captured by the main camera 110, and combine the respective advantages of the two cameras to generate an image with a large field of view and high resolution in some areas. .

FIG. 2 is an image processing method 200 provided by an embodiment of the present application. The method can be applied to an electronic device, and the method shown in FIG. 2 can be executed by a processor in the electronic device. The electronic device may include a first camera and a second camera, and the first camera and the second camera are two different types of cameras with different imaging effects.

For example, the observation range of the first camera is larger than the observation range of the second camera. For another example, the resolution of the first camera is lower than the resolution of the second camera. For another example, the image of the first camera is an achromatic image, and the image of the second camera is a color image.

The observation range in the embodiment of the present application can be understood as the scene area that can be photographed by the camera.

The method 200 includes steps S210-S240.

S210: Acquire a first image captured by the first camera of the scene.

S220: Acquire a second image captured by the second camera of the scene.

S230. Determine a first pixel area corresponding to the target object in the scene in the first image, and determine a second pixel area corresponding to the target object in the scene in the second image.

The first pixel area is the imaging area for the target object in the first image, and the second pixel area is the imaging area for the target object in the second image.

S240. Perform image processing on the first pixel area in the first image based on the second pixel area of the second image to obtain a processed first image.

The first camera and the second camera in the embodiment of the present application may shoot for the same scene, so as to obtain the first image and the second image respectively.

It can be understood that, because the observation ranges of the first camera and the second camera are different, the objects in the scene captured by the first camera are not exactly the same as the objects in the scene captured by the second camera. For example, since the observation range of the first camera is larger than the observation range of the second camera, there are more objects in the scene captured by the first camera than objects in the scene captured by the second camera.

For example, both the first camera and the second camera shoot at the forest, and there are imaging areas for the same trees in the first image and the second image, but the number of corresponding trees in the first image is greater than the number of trees corresponding to the second image quantity.

For the convenience of description, the difference between the first image and the second image caused by the different observation ranges of the first camera and the second camera will be referred to as the difference in the field of view of the first image and the second image. The field of view is larger than the field of view of the second image.

The first pixel area and the second pixel area are imaging areas for the same target object in the first image and the second image, respectively.

The target object is a partial scene object in the scene captured by the first camera, that is, the first pixel area is a partial area in the first image.

The target objects are all or part of the scene objects in the scene captured by the second camera, that is, the second pixel area may be the entire area in the second image, or may be a partial area of the second image.

If the second pixel area is the entire area in the second image, it means that the objects in the scene captured by the first camera include all the objects captured by the second camera.

The first pixel area and the second pixel area described above are imaging areas that objectively target the same target object. In addition, this embodiment of the present application may also determine the first pixel area and the second pixel area according to the user's selection, that is, the first pixel area and/or the second pixel area corresponding to the target object are determined based on user operations. For example, the region where the user wishes to perform image processing may be determined according to the user's click operation, and then the first pixel region and the second pixel region may be determined according to the region where the user wishes to perform image processing.

If the user only cares about the characters in the scene, he can click on the characters, and the first pixel area and the second pixel area are imaging areas corresponding to the same character in the first image and the second image, respectively.

Since the second image is a color image and/or the second image is a high-resolution image, the second image has a better visual effect, and the observation range of the first camera is larger than that of the second camera. Therefore, the first The field of view corresponding to the image is larger than the field of view corresponding to the second image. The present application can fuse the second image with better visual effect and the first image with large field of view to obtain the processed first image with both large field of view and better imaging effect, so that users can have better visual experience.

That is to say, the user can observe the imaging of objects in more scenes around the electronic device by observing the first image with a larger field of view, thereby making a decision that is beneficial to subsequent judgments. Possibly, for the image area that the user is interested in, the display information supplemented by the second image can also be used to assist in improving the visual effect of some areas in the first image, such as stronger or more realistic color, clearer imaging, etc.

For convenience of description, the first pixel area is hereinafter understood as an area in the first image that overlaps with the second image, and the second pixel area is understood as an area in the second image that overlaps with the first image.

FIG. 3 and FIG. 4 show different degrees of overlapping of the first image and the second image. Fig. 3 shows a situation where the entire area of the second image completely overlaps with a partial area of the first image, and Fig. 4 shows a situation where a partial area of the second image overlaps with a partial area of the first image.

Taking FIG. 3 as an example, the entire area of the second image 320 overlaps with a partial area of the first image 310 . In this case, the second pixel area is the entire area of the second image, and the first pixel area 311 and the second image overlapping.

As shown in FIG. 4 , a partial area of the second image 320 overlaps with a partial area of the first image 210 , in this case, the first pixel area 312 overlaps with a part of the second image 320 . Preferably, the area of the area overlapping with the first image in the second image is larger than the area of the area not overlapping, that is, the size of the second pixel area 312 is slightly smaller than the size of the second image 320 .

The camera in this embodiment of the present application may be a camera that shoots photos, or may be a camera that shoots videos. For a camera that shoots video, the first image and the second image may be a certain frame of images in the video stream.

The first image may be an image directly captured by a camera, or the first image may be an image obtained by performing image processing on an image captured by a camera.

For example, the present application may acquire the first initial image, and perform stabilization processing on the first image to obtain the first image. During the image or video shooting process, shaking cannot be avoided. In the embodiment of the present application, the image captured by the camera can be stabilized, so that the user will be more comfortable watching the stabilized image without dizziness.

In addition, during the stabilization process, the image needs to be cropped, that is, the image size after stabilization is smaller than the image size before stabilization. Therefore, in this embodiment of the present application, the first initial image can be stabilized first, and then fused with the second image, so as to ensure that the second image with high resolution and/or color information will not be cropped, and improve the Utilization of the second image.

The embodiment of the present application does not specifically limit the manner of stabilization processing, for example, a traditional video stabilization algorithm may be used for stabilization.

Generally, the gyroscope in the inertial measurement unit (IMU) is used to measure the change of the rotation angle of the sensor in the camera, and then the change is filtered more smoothly through a filter to generate a new attitude angle, and finally the image Switch to the stabilized attitude angle.

Figure 5 shows the change of the attitude angle of the sensor before and after stabilization. Among them, the dotted line represents the change of the attitude angle with time before stabilization, and the solid line represents the change of the attitude angle with time after stabilization.

It can be seen from Figure 5 that after stabilization, the change of the sensor attitude angle is more stable.

The attitude angle in this embodiment of the present application may include the pitch angle of the sensor.

Taking the drone in FIG. 1 as an example, the first camera may be the camera 120 , and the second camera may be the main camera 110 . The camera 120 and the camera 110 have their respective functions in the drone. The camera 120 is used to perceive the surrounding environment, and the images it takes will not be presented to the user; the camera 110 is used to take pictures, and the images it takes are mainly used for presentation. to users.

The present application cleverly combines the images captured by the camera 110 and the camera 120 without changing the hardware mechanism of the electronic device, and can present images with both a large field of view and a better visual effect to the user without significantly increasing the cost.

The first camera in this embodiment of the present application may be used to perceive depth information of surrounding objects, that is, the first image may be an image including depth information of the object.

The embodiments of the present application do not specifically limit the manner in which the first camera acquires the depth information. For example, the first camera can acquire depth information according to the principle of binocular vision, as shown in FIG. 1 . For another example, the first camera may acquire depth information according to the time of flight (TOF) principle.

Taking binocular vision as an example, the first camera 120 in FIG. 1 may include a first visual sensor 121 and a second visual sensor 122 . Further, the first visual image captured by the first visual sensor 121 may be acquired, the second visual image captured by the second visual sensor 122 may be acquired, and an image with depth information may be generated according to the first visual image and the second visual image.

In this embodiment of the present application, a first initial image may be generated according to the first visual image and the second visual image, and then stabilization processing is performed on the first initial image to obtain the first image. Alternatively, stabilization processing may also be performed on the first visual image and the second visual image respectively, and then the first image is generated based on the stabilized left first visual image and the second visual image.

Since the first image and the second image are generated by different cameras, there will be differences in the internal parameters, shooting positions and shooting angles of different cameras. If the images captured by the cameras are directly fused and matched, the complexity will be relatively high.

Based on this, the second image in this embodiment of the present application may be an image after pose transformation. For example, the present application may acquire a second initial image, where the second initial image may be an image directly captured by the second camera, and then convert the second initial image to the camera pose where the first image is located to generate the second image .

This embodiment of the present application may first perform pose transformation on the second initial image, the transformed second image is the same as the camera pose where the first image is located, and then the second image with the same camera pose is fused with the first image deal with. Fusion of two images with the same camera pose can greatly reduce the complexity of fusion processing.

The above description is to convert the second initial image to the camera pose where the first image is located, which is not specifically limited in this embodiment of the present application. Of course, it is also possible to convert the first image to the camera pose where the second image is located, as long as the camera poses of the two finally generated images are the same.

In this embodiment of the present application, the second initial image can be converted into the first image according to at least one of the following parameters: internal parameters of the first camera, internal parameters of the second camera, and a rotation relationship between the first image and the second initial image. On the camera pose of the image, a second image is formed.

For example, the second image may be formed by converting the second initial image to the camera pose of the first image according to the internal parameters of the first camera and the internal parameters of the second camera. In this way, the camera internal parameters corresponding to the converted second image and the first image are the same. Since the rotation relationship is not adjusted, the converted first image and the second image may still have a relative rotation relationship.

For another example, according to the internal parameters of the first camera, the internal parameters of the second camera, and the rotation relationship between the first image and the second initial image, the second initial image can be converted into the camera pose of the first image to form second image. In this way, the converted second image and the first image not only have the same internal reference, but also eliminate the relative rotation relationship. The baselines of the two images are already parallel, and the matching relationship between the two images only has a positional deviation in the horizontal direction. In this way, during the image matching process, It is only necessary to find matching pixels in the horizontal direction, which can greatly reduce the complexity of image processing.

The camera model used in this embodiment of the present application may be as follows:

Among them, [u, v, 1] ^T represents a two-dimensional (2D) pixel point in a homogeneous image coordinate system (homogeneous image coordinates).

[x _w , y _w , z _w ] ^T represents a three-dimensional (world coordinates, 3D) pixel point in the world coordinate system.

The matrix K is called the camera calibration matrix, that is, the internal parameters of each camera, referred to as internal parameters.

For a finite projective camera, the internal parameter K contains 5 parameters, as follows:

Wherein, α _x =fm _x , α _y =fm _y , f represents the focal length, and m _x and m _y represent the number of pixels per unit distance on the x and y axes. γ represents the skew parameters between the x, y axes, eg, for a charge coupled device (CCD) camera, the pixels are not square. μ and ν are the position of the optical center (principal point).

The matrix R is the rotation matrix, the matrix T is the translation matrix, and R and T are the external parameters of the camera (extrinsic matrix). Rotation and displacement transformations.

For the stabilization process, the original posture of the first image in the world coordinate system is denoted as R ₀ , and the posture of the filtered first image is denoted as R ₁ , then the rotation transformation from the original posture to the filtered posture is R ₁₀ = R ₁ R ₀ ^-1 .

For the pose transformation process, the internal parameter of the second camera is recorded as K _c , the second initial image captured by the second camera is recorded as I _c , the internal parameter of the first camera is recorded as K _g , the first image is recorded as I _g , and the first image is recorded as I g . The rotation relationship between the first image and the second initial image is denoted as R _gc .

It can be obtained from the above stabilization process that the posture of the first image after stabilization and filtering is transformed to R ₁₀ , and then the second initial image I _c is transformed to the camera position where the first image I _g is located.

where (u _c , _vc ) ^T is a point on the second initial image I _c , and (u _c ', _vc ') ^T is a point on the transformed second image I _c '.

The electronic device in the embodiment of the present application may be a device such as a movable platform.

After the processed first image is obtained in the manner described above, the user can operate the electronic device according to the processed first image. For example, the user can enter instructions. The electronic device can adjust the orientations of the first camera and the second camera according to the instruction input by the user, so that the first camera and the second camera shoot towards the same scene.

After obtaining the processed first image, the user can determine the current position information of the electronic device according to the information in the processed first image, and determine the moving direction of the electronic device according to the current position information, and input the information related to the movement The control command corresponding to the direction.

The processed first image in the embodiment of the present application can be used for FPV flight. For example, on a drone, the user can control the flight of the drone according to the processed first image, and since the field of view of the processed first image is larger, the user's field of view is wider, and the user experience will be better.

The first pixel area in the embodiment of the present application is preferably located in the middle area of the first image, that is, the overlapping area of the second image and the first image is the middle area of the first image. In this way, after the second image is fused with the first image, the generated processed first image may be an image in which the middle area is a high-resolution image, the edge area is a low-resolution image, and/or the middle area is a color image, The edge area is the image of the subcolor image. Since the user mainly focuses on the middle area when viewing the processed first image, the high resolution and/or color of the middle area can improve the user's visual experience. In addition, the edge area is mainly used to assist the user's perception of the surrounding environment. Therefore, the user does not have high requirements for the resolution and/or color of the edge area, and the low resolution and/or color information of the edge area will not affect the user's Visual experience.

In addition, the user may also adjust the shooting direction of the first camera and/or the second camera with respect to the region where the first pixel region is located in the first image. For example, as shown in FIG. 4 , if the first pixel area 312 is located at the edge area of the first image 310 instead of the middle area, the user can adjust the shooting direction of the first camera and/or the second camera so that the first pixel area located in the middle area of the first image.

Optionally, the first image is an achromatic image, and the second image is a color image. For example, the first image may be a low-resolution achromatic image, and the second image may be a high-resolution color image.

Since the first image is an achromatic image and the second image is a color image, during the fusion process, the color information of the second image can be retained. In this way, the generated processed first image is partially colored and partially black and white. Image.

In order to improve the user's visual experience, after the processed first image is generated, the embodiment of the present application may further perform color processing on the processed first image to generate an aesthetic artistic photo.

The color processing may include at least one of the following: color filling, color removal, and color retention.

Since the field of view of the second image is smaller than that of the first image, in the processed first image after fusion, half of the same object is in color and half is in black and white, which will affect the user's visual experience .

As shown in FIG. 6 , the middle area 420 in FIG. 6 is a color area, and the edge area 410 is a black and white area. Among them, the tall buildings 402 are half color and the other half are black and white; the ship 404 is half color and the other half is black and white; the road 406 is half color and the other half is black and white; the Ferris wheel 408 is half color and the other half is black and white.

Based on this, the embodiments of the present application may perform color filling (or color expansion) on the processed first image. Specifically, based on the colored areas of the same object in the processed first image, the colorless areas may be filled. color fill. For example, the processed first image can be segmented, and the same object can be segmented from the processed first image; if one part of the same object has color information and the other part has no color information, the same object can be divided according to the color information in the same object. part, fill the other part with color.

This embodiment of the present application does not specifically limit the manner of dividing the processed first image. For example, a superpixel segmentation method can be used to segment the processed first image to generate superpixels, so that adjacent pixels with similar texture, color, brightness and other characteristics form irregular pixel blocks with certain visual significance.

Using the similarity of features between pixels to group pixels, replacing a large number of pixels with a small number of superpixels to express image features, the use of superpixel segmentation can significantly reduce the number of pixels, which can greatly reduce the complexity of image post-processing .

The smaller the number of superpixels after image segmentation, the lower the complexity of image processing, but at the same time, the more features in the image are lost. This embodiment of the present application does not specifically limit the number of divided superpixels, which may be set according to actual needs.

FIG. 7 shows an image segmented by a superpixel segmentation method, wherein the number of superpixels after the segmentation of the region 504 is smaller than the number of superpixels after the segmentation of the region 502 . Therefore, the processing complexity of region 504 is lower than that of region 502 , however, the image features retained by region 502 are greater than those retained by region 504 .

The embodiments of the present application do not specifically limit the method used for superpixel segmentation. For example, the superpixel segmentation method may include at least one of the following algorithms: simple linear iterative clustering (SLIC), Graph-based, NCut, Turbopixel, Quick-shift, Graph-cut a, Graph-cut b.

Compared with other algorithms, the SLIC algorithm can significantly reduce the computational complexity and at the same time can improve the control over the size and compactness of the superpixels. Therefore, the SLIC algorithm is preferably used in this embodiment of the present application.

In order to further reduce the complexity of image processing, in this embodiment of the present application, only the same object located at the boundary between the color area and the black and white area may be segmented, and if a part of the object has color information, the color information is extended to the entire object.

As shown in FIG. 6 , the Ferris wheel 408 is located at the boundary between the color area and the black and white area, and a part of the Ferris wheel 408 has color information and the other part has no color information, then the color can be extended to the entire Ferris wheel according to the part of the color information. .

This embodiment of the present application may further perform color processing such as color filling, and/or color removal, and/or color retention, on the processed first image based on the depth information.

Specifically, since the first image has depth information, the generated processed first image also has depth information. The present application may perform color processing on the processed first image based on the depth information. For example, the same color processing can be performed on regions of the same depth in the processed first image.

As an embodiment, a user's click operation on the processed first image may be acquired, first depth information of the object at the click location may be determined, and color processing may be performed on the processed first image according to the first depth information.

For example, the target area can be determined according to the first depth information, and the target area is filled with color or only the color of the target area is retained. Wherein, the target area is connected to the object at the point selected, and the difference between the depth of the target area and the first depth information is within a first preset range.

The user can click on the object that needs to be colored, and the processor can determine the pixel points that are connected in position and similar in depth as the target area according to the depth information of the clicked place, and only display the color of the target area, thus forming an aesthetic art. According to.

As another embodiment, distance information input by the user may be acquired, and color processing is performed on the processed first image according to the distance information.

The target area can be determined according to the distance information, and the target area is filled with color or only the color of the target area is retained, wherein the difference between the depth of the target area and the distance is within a second preset range.

The user can input the distance at which the color is to be displayed, and the processor can convert the distance into depth, and then highlight the object corresponding to the depth with color, leaving the rest in black and white, thus forming an aesthetically pleasing artistic photo.

In addition, the embodiment of the present application can highlight objects, buildings, etc. at a certain distance according to the depth information, or perform auxiliary measurement, detection, etc. based on the depth information.

The embodiments of the present application do not specifically limit the manner of performing image processing on the first pixel region in the first image based on the second pixel region in the second image.

For example, the second pixel area in the second image can be used to directly replace the first pixel area in the first image. This method is simple to implement and has low processing complexity, but there will be images in the processed first image. dislocation.

For another example, the second pixel area may be mapped to the first pixel area according to the mapping relationship between the second pixel area and the first pixel area. The processed first image generated in this way is relatively beautiful, and there is no obvious image dislocation.

The following description takes the mapping method as an example.

The present application can map the second pixel area to the first pixel area according to the mapping relationship between the pixel points in the second pixel area and the pixel points in the first pixel area.

For example, a pixel point matching the pixel point in the second pixel area is found in the first pixel area, and the pixel point in the second pixel area is mapped to the position of the matching pixel point in the first pixel area.

If, for each pixel in the second pixel area, a matching pixel is searched in the first pixel area, this will undoubtedly increase the complexity of image processing. Based on this, in order to reduce the complexity of image processing, the present application only maps the edges of the first pixel area and the second pixel area, and the non-edge area can be interpolated by using an interpolation method.

Specifically, according to the mapping relationship between the edge of the second pixel area and the edge of the first pixel area, the edge of the second pixel area can be mapped to the edge of the first pixel area, and the non-edge area can use the edges on both sides Do difference mapping.

As shown in FIG. 8 , 315 , 316 , and 317 are edge areas of the

first image

310 , and 314 is a non-edge area of the first image 310 ; 325 , 326 , and 327 are edge areas of the second image, and 324 is the second image 320 non-edge areas. Also, edge 315 matches edge 325, edge 316 matches edge 326, and edge 317 matches edge 327. Therefore, edge 325 can be mapped to where edge 315 is, and edge 326 can be mapped to where edge 316 is. , which maps edge 327 to where edge 317 is located. The non-edge region 314 can be obtained by interpolation based on the information of the

edges

315 and 316 on both sides thereof.

Optionally, if only one side of the non-edge region has edge information, you can choose not to map.

The present application can determine the edge of the second pixel area and the edge of the first pixel area, and map the edge of the second pixel area to the first pixel area according to the mapping relationship between the edge of the second pixel area and the edge of the first pixel area the edge of the pixel area to generate image information of the edge area of the processed first image.

The embodiments of the present application do not specifically limit the manner of determining the edge of the second pixel region and the edge of the first pixel region.

For example, edge detection operators can be used to extract the edge of the second pixel region and the edge of the first pixel region. The edge detection operator may be, for example, a detection operator in the Canny Operator algorithm, that is, the Canny Operator algorithm may be used to extract the edges of the second image and the first pixel area.

For the Canny Operator algorithm, the parameters in the edge detection operator can be threshold parameters.

Since the exposure parameters of the first camera and the second camera are inconsistent, this will lead to differences in the edges of the two images. If the edge densities of the two images are not balanced, it will increase the complexity of edge mapping. For example, the number of edges extracted from the second image is greater than the number of edges extracted from the first pixel region, which increases the complexity of determining the mapping relationship between the edges.

In order to improve the accuracy of edge matching and reduce the complexity of matching, the present application can adjust the parameters in the edge detection operator to balance the edge densities of the second pixel area and the first pixel area.

Edge density may refer to the ratio between the number of pixels on the edges in the image and the total number of pixels in the entire image.

For the first pixel area, the edge density d _g is:

d _g = the number of pixels on the edge of the first pixel area/the total number of pixels in the first pixel area

For the second pixel area, the edge density _dc is:

d _c = the number of pixels on the edge in the second pixel area/the total number of pixels in the second pixel area

The present application can extract the edge of the second pixel area and the initial edge of the first pixel area; and adjust the parameters in the edge detection operator according to the edge density of the second pixel area and the initial edge density of the first pixel area; The parameters in the subsequent edge detection operator determine the edge of the first pixel area.

Specifically, if the difference between the initial edge density of the first pixel area and the edge density of the second pixel area is within the third preset range, the initial edge of the first pixel area may be used as the final first pixel area If the difference between the initial edge density of the first pixel area and the edge density of the second pixel area is not within the third preset range, adjust the threshold parameter in the edge detection operator, and according to the adjusted parameters, Re-extract the edge of the first pixel area, so that the difference between the edge density of the second pixel area and the edge density of the first pixel area is within the third preset range, that is, the edge detection operator can be adjusted continuously and re-extract the edge of the first pixel area until the re-extracted edge density of the first pixel area and the edge density of the second pixel area are within a third preset range.

The above description is to re-extract the edge of the first pixel area so that the difference between the edge density of the first pixel area and the edge density of the second pixel area is within the third preset range. It is not limited to this, as long as it is ensured that the difference between the edge densities between the two images is within the third preset range. For example, it is also possible to keep the edge of the first pixel area unchanged, and re-extract the edge of the second pixel area, so that the difference between the edge density of the second pixel area and the edge density of the first pixel area is within the third preset range Inside.

Taking the Canny Operator algorithm as an example, the default threshold parameters [t _l , t _n ] can be used first to extract the edge of the second pixel area and the edge of the first pixel area. Taking FIG. 9 as an example, the left side is the second pixel area, and the right side is the extracted edge of the second pixel area.

If the edge densities of the second pixel area and the first pixel area are unbalanced, the following process can be used to adjust.

(1) If d _g <d _c -ε, reduce the threshold parameter [t _l , t _n ], re-extract the edge of the first pixel area, and recalculate the edge density d _g of the first pixel area.

(2) If d _g >d _c +ε, increase the threshold parameter [t _l , t _n ], re-extract the edge of the first pixel area, and recalculate the edge density d _g of the first pixel area.

(3) Until d _c -ε<d _g <d _c +ε, the above process is stopped.

Among them, ε represents the tolerance of the edge density difference between the two images, and its size is an engineering experience value, which can be adjusted as needed. 1/ε represents the similarity of the edge density of the two images.

According to the mapping relationship between the edge of the second pixel region and the edge of the first pixel region, the edge of the second pixel region is mapped to the edge of the first pixel region to generate the processed The image information of the edge area of the first image may include: determining a first edge segment and a second edge segment, the first edge segment is the edge segment on the edge of the first pixel area, and the second edge segment is the second pixel area. According to the mapping relationship between the first edge segment and the second edge segment, the second edge segment is mapped to the region where the first edge segment is located, so as to generate the edge region of the processed first image. image information.

Since the extracted edge may be multiple edge segments, rather than one continuous edge, the present application can perform edge matching according to the mapping relationship between multiple edge segments in the two figures.

In addition, in order to further reduce the complexity in the edge matching process, the present application may segment the extracted edge, for example, into multiple line segments, and then perform matching according to the segmented line segments.

Figure 10 shows schematic views before and after edge segmentation. The left image is the edge extracted according to the edge extraction algorithm, and the right image is the edge after segmentation.

There may be bifurcated edge segments in the extracted edges of the image, for example, an edge segment similar to the one in the left image where three line segments meet at one point. If the edge matching is performed directly with the forked edge segment, it will increase the difficulty of matching.

In this application, the extracted edge can be divided into multiple line segments, that is, into multiple segments with two endpoints and a connecting line. Because the structure of the line segment is simple, and compared with the edge segment before the segment, the pixels on the line segment are also smaller. Therefore, matching with line segments can improve the efficiency of matching and reduce the difficulty of matching.

The line segment in the embodiment of the present application may be a straight line segment, or may also be a curve segment, which is not specifically limited.

Specifically, the present application can segment the first edge segment to obtain the first line segment set; segment the second edge segment to obtain the second line segment set; determine the line segment and the second line segment in the first line segment set The mapping relationship between the line segments in the set; according to the mapping relationship between the line segments in the first line segment set and the line segments in the second set, the line segments in the second line segment set are mapped to the line segments in the first line segment set , to generate image information of the edge region of the processed first image.

Further, the segmented line segments can be deleted, and some line segments that do not meet the requirements can be removed, and then the line segment matching can be performed. For example, a line segment whose length is less than the first preset value in the line segment set may be eliminated, because it is not meaningful to match a line segment whose length is too small, and processing resources will be wasted.

For the first edge segment, the present application may segment the first edge segment to obtain a first initial edge segment set; and then remove line segments whose length is less than the first preset value in the first initial edge segment set to obtain the first line segment collection.

For the second edge segment, the present application may segment the second edge segment to obtain a second initial edge segment set; and then remove line segments whose length is less than the second preset value in the second initial edge segment set to obtain the second line segment gather.

Wherein, the first preset value and the second preset value may be equal.

When performing line segment matching, the line segments in the first line segment set and the second line segment set may be determined according to the matching degree between the pixels on the line segments in the first line segment set and the pixels on the line segments in the second line segment set The mapping relationship between the line segments in .

This embodiment of the present application does not specifically limit the algorithm used in the matching process between the first line segment set and the second line segment set.

For example, the matching between the first set of line segments and the second set of line segments may be performed using a normalized cross-correlation (NCC) algorithm.

Assume that the first line segment set is E _g ={l ₁ ,l ₂ ,l ₃ ,...,l _n }, and the second line segment set is E _c ={m ₁ ,m ₂ ,m ₃ ,...,m _n }.

For each pixel point p _i1 , p _i2 ,...,p _in on the _i -th line segment li in the set E _g , the corresponding matching points q _i1 , q _i2 ,...,q in the set E _c are searched by NCC matching _in . If the matching point q belongs to a certain line segment m _j , the matching degree of the line segment _li and the line segment m _j is increased by 1. The matching points corresponding to each point on the line segment _{li are repeatedly calculated, so as to find the line segment m j} _that best matches the line segment _li .

The embodiment of the present application does not specifically limit the manner of determining the line segment m _j that matches the line segment _li .

As an example, if the ratio between the number of pixels matching the line segment _{li in the line segment m j} _and the total number of pixels on the line segment m _j is greater than a preset threshold, it can be determined that the line segment m _j and the line segment _li match.

As another example, according to the voting principle, the line segment with the largest number of pixel points matching the line segment _li in the set E _c may be determined as the line _segment matching the line segment li, or the line segment matching the line segment li in the set E _c may be determined as the line segment matching the line segment _li . The line segment with the highest proportion of pixel points is determined as the line _segment matching the line segment li, or the line segment with the proportion of the pixel points matching the line segment _li in the set E _c that is greater than the preset threshold is determined as the line _segment matching the line segment li. .

For example, if l ₁ and m ₁ have 2 pixels matching, m ₂ has 15 pixels matching, and m ₃ has 3 pixels matching, then according to the voting principle, m ₂ and l ₁ matching pixels At most, that is, m ₂ matches l ₁ the most, so m ₂ matches l ₁ .

Or, assuming that the preset threshold is 50%, since the proportion of pixels matching m ₂ and l ₁ is 15/(2+15+3)=75%, 75%>50%, then m ₂ matches l ₁ .

According to the above description, since the first image and the second image have already aligned the internal parameters, in the matching process, it is only necessary to search for the corresponding matching line segment in the horizontal direction.

Taking FIG. 11 as an example, the left side is the first image, and the right side is the second image. For the line segment in the first image, compare with the line segment in the same row in the second image, and find the line segment in the second image that matches the line segment in the first image.

FIG. 11 is a schematic diagram of finding a matching line segment for the line segment 602 in the first image. According to the voting principle described above, since the number of matching pixels in the line segment 604 in the second image is the largest, that is, the line segment 604 has the highest matching score, the line segment 604 is the line segment that matches the line segment 602 .

The method embodiments of the present application are described in detail above with reference to FIG. 1 to FIG. 11 , and the apparatus embodiments of the present application are described below with reference to FIG. 12 . The previous part of the method embodiment.

FIG. 12 is an image processing apparatus 700 provided by an embodiment of the present application. The apparatus 700 may include a memory 710 and a processor 720 .

The memory 710 is used to store computer programs.

The processor 720 is configured to invoke the computer program, and when the computer program is executed by the processor, cause the apparatus to perform the following steps:

acquiring a first image captured by the first camera of the electronic device on the scene;

acquiring a second image obtained by photographing the scene by a second camera of the electronic device;

Wherein, the observation range of the first camera is larger than the observation range of the second camera; and the resolution of the first camera is lower than the resolution of the second camera, and/or the first camera The imaging of the second camera is an achromatic image, and the imaging of the second camera is a color image;

Optionally, in some embodiments, the target object is all or part of the scene objects in the scene captured by the second camera.

Optionally, in some embodiments, the first pixel area and/or the second pixel area corresponding to the target object is determined based on a user operation.

Optionally, in some embodiments, when the computer program is executed by the processor, it causes the apparatus to perform the following steps: acquiring a first initial image captured by the first camera of the scene; Stabilization processing is performed on the first initial image to obtain the first image.

Optionally, in some embodiments, the first camera includes a first visual sensor and a second viewing angle sensor, and when the computer program is executed by the processor, the apparatus causes the apparatus to perform the following steps: acquiring the obtaining a first visual image captured by the first visual sensor on the scene; acquiring a second visual image captured by the second visual sensor on the scene; according to the first visual image and the second visual image image, generating the first initial image with depth information.

Optionally, in some embodiments, when the computer program is executed by the processor, it causes the apparatus to perform the following steps: acquiring a second initial image captured by the second camera of the scene; Converting the second initial image to the camera pose of the first image to generate the second image.

Optionally, in some embodiments, when the computer program is executed by the processor, it causes the apparatus to perform the following steps: according to the internal parameters of the first camera, the internal parameters of the second camera, and the rotation relationship between the first image and the second initial image, and converting the second initial image to the camera pose of the first image to generate the second image.

Optionally, in some embodiments, when the computer program is executed by the processor, it causes the apparatus to perform the following steps: receiving an instruction input by a user; adjusting the first instruction according to the instruction input by the user The shooting directions of the camera and the second camera are such that the first camera and the second camera shoot toward the same scene.

Optionally, in some embodiments, the electronic device is an unmanned aerial vehicle, and the processed first image is used for FPV flight from a first-person main perspective.

Optionally, in some embodiments, the first pixel area is located in a middle area of the first image.

Optionally, in some embodiments, the first image is an achromatic image, and the second image is a color image.

Optionally, in some embodiments, when the computer program is executed by the processor, it causes the device to perform the following steps: using a superpixel segmentation device to segment the processed first image; if If a part of the same object after segmentation has color information and the other part has no color information, then another part of the same object is filled with color according to the part of the same object that has color information.

Optionally, in some embodiments, the achromatic image includes depth information of an object, and when the computer program is executed by the processor, causes the apparatus to perform the following steps: according to the depth information of the object, Color processing is performed on the processed first image, and the color processing includes at least one of the following: color filling, color removal, and color retention.

Optionally, in some embodiments, when the computer program is executed by the processor, the computer program causes the apparatus to perform the following step: performing the same color processing on regions of the same depth in the processed first image .

Optionally, in some embodiments, when the computer program is executed by the processor, it causes the apparatus to perform the following steps: acquiring a user's clicking operation on the processed first image; determining a point first depth information of the selected object; color processing is performed on the processed first image according to the first depth information.

Optionally, in some embodiments, when the computer program is executed by the processor, it causes the apparatus to perform the following step: determining a target area according to the first depth information, wherein the target area is the same as the The objects at the selected point are connected, and/or the difference between the depth of the target area and the first depth information is within a first preset range; and color processing is performed on the target area.

Optionally, in some embodiments, when the computer program is executed by the processor, it causes the apparatus to perform the following steps: acquiring distance information input by a user; The first image is color processed.

Optionally, in some embodiments, when the computer program is executed by the processor, it causes the apparatus to perform the following step: determining a target area according to the distance information, wherein the depth of the target area is the same as that of the target area. The difference between the distances is within a second preset range; and color processing is performed on the target area.

Optionally, in some embodiments, the computer program, when executed by the processor, causes the apparatus to perform the steps of: determining the edge of the second pixel area and the edge of the first pixel area; According to the mapping relationship between the edge of the second pixel region and the edge of the first pixel region, the edge of the second pixel region is mapped to the edge of the first pixel region to generate the processed The image information of the edge region of the first image.

Optionally, in some embodiments, the computer program, when executed by the processor, causes the apparatus to perform the step of: extracting the edge of the second pixel area and the initial edge of the first pixel area ; According to the edge density of the second pixel area and the initial edge density of the first pixel area, adjust the parameters in the edge detection operator; According to the parameters in the edge detection operator after the adjustment, determine the first The edge of a pixel area.

Optionally, in some embodiments, when the computer program is executed by the processor, it causes the apparatus to perform the following steps: at the edge density of the second pixel area and the initial If the difference between the edge densities is not within the third preset range, adjust the parameters in the edge detection operator; the first pixel area is determined according to the parameters in the adjusted edge detection operator The edges of The difference between is within the third preset range.

Optionally, in some embodiments, when the computer program is executed by the processor, it causes the apparatus to perform the step of: determining a first edge segment and a second edge segment, the first edge segment being the an edge segment on the edge of the first pixel region, and the second edge segment is an edge segment on the edge of the second pixel region; according to the mapping between the first edge segment and the second edge segment relationship, the second edge segment is mapped to the region where the first edge segment is located, so as to generate image information of the edge region of the processed first image.

Optionally, in some embodiments, when the computer program is executed by the processor, it causes the apparatus to perform the following steps: segment the first edge segment to obtain a first line segment set; The second edge is segmented to obtain a second line segment set; the mapping relationship between the line segments in the first line segment set and the line segments in the second line segment set is determined; according to the first line segment set The mapping relationship between the line segments in the second line segment set and the line segments in the second line segment set, the line segments in the second line segment set are mapped to the positions of the line segments in the first line segment set to generate the processing The image information of the edge area of the first image after.

Optionally, in some embodiments, when the computer program is executed by the processor, it causes the apparatus to perform the following steps: segment the first edge segment to obtain a first initial set of line segments; culling The first line segment set is obtained from the line segments whose length is less than the first preset value in the first initial line segment set.

Optionally, in some embodiments, when the computer program is executed by the processor, it causes the apparatus to perform the following steps: segment the second edge segment to obtain a second initial set of line segments; culling The second line segment set is obtained from the line segments whose length is less than the second preset value in the second initial line segment set.

Optionally, in some embodiments, when the computer program is executed by the processor, the computer program causes the apparatus to perform the following steps: The matching degree between the pixels and the pixels on the line segments in the second line segment set determines the mapping relationship between the line segments in the first line segment set and the line segments in the second line segment set.

Optionally, in some embodiments, when the computer program is executed by the processor, it causes the apparatus to perform the following steps: according to the image information of the edge region of the processed first image, use an interpolation method Image information of the non-edge region of the processed first image is generated.

The present application also provides an electronic device or system, where the electronic device or system may include the image processing apparatuses of the various embodiments of the present application.

The present application also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer causes the computer to execute the method of the above method embodiment.

The present application also provides a computer program product comprising instructions, the instructions, when executed by the computer, cause the computer to execute the method of the above method embodiment.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server, or data center Transmission to another website site, computer, server, or data center by wire (eg, coaxial cable, optical fiber, digital subscriber line) or wireless (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes an integration of one or more available media. The usable media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, digital video disc (DVD)), or semiconductor media (eg, solid state disk (SSD)), etc. .

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

An image processing method, comprising:

acquiring a first image captured by the first camera of the electronic device on the scene;

acquiring a second image obtained by photographing the scene by a second camera of the electronic device;

Wherein, the observation range of the first camera is larger than the observation range of the second camera; and the resolution of the first camera is lower than the resolution of the second camera, and/or the first camera The imaging of the second camera is an achromatic image, and the imaging of the second camera is a color image;

determining a first pixel area corresponding to the target object in the scene in the first image, and determining a second pixel area corresponding to the target object in the scene in the second image;

Based on the second pixel area in the second image, image processing is performed on the first pixel area in the first image to obtain a processed first image.
The method according to claim 1, wherein the target objects are all or part of the scene objects in the scene captured by the second camera.
The method according to claim 1 or 2, wherein the first pixel area and/or the second pixel area corresponding to the target object is determined based on a user operation.
The method according to any one of claims 1-3, wherein the acquiring the first image captured by the first camera of the electronic device on the scene comprises:

acquiring a first initial image captured by the first camera of the scene;

Stabilization processing is performed on the first initial image to obtain the first image.
The method according to claim 4, wherein the first camera comprises a first visual sensor and a second viewing angle sensor, and the acquiring a first initial image captured by the first camera on the scene, include:

acquiring a first visual image captured by the first visual sensor of the scene;

acquiring a second visual image captured by the second visual sensor on the scene;

Based on the first visual image and the second visual image, the first initial image with depth information is generated.
The method according to any one of claims 1-5, wherein the acquiring a second image obtained by photographing the scene by a second camera of the electronic device comprises:

acquiring a second initial image captured by the second camera of the scene;

Converting the second initial image to the camera pose of the first image to generate the second image.
The method according to claim 6, wherein the converting the second initial image to the camera pose of the first image to form the second image comprises:

Converting the second initial image to the first the camera pose of an image to generate the second image.
The method according to any one of claims 1-7, wherein the method further comprises:

Receive instructions entered by the user;

According to the instruction input by the user, the shooting directions of the first camera and the second camera are adjusted, so that the first camera and the second camera shoot toward the same scene.
The method according to any one of claims 1-8, wherein the electronic device is an unmanned aerial vehicle, and the processed first image is used for FPV flight from a first-person main perspective.
The method according to any one of claims 1-9, wherein the first pixel area is located in a middle area of the first image.
The method according to any one of claims 1-10, wherein the first image is an achromatic image, and the second image is a color image.
The method of claim 11, wherein the method comprises:

using a superpixel segmentation method to segment the processed first image;

If a part of the same object after segmentation has color information and the other part has no color information, then another part of the same object is filled with color according to the part of the same object that has color information.
The method according to claim 11 or 12, wherein the achromatic image includes depth information of the object, and the method further comprises:

According to the depth information of the object, color processing is performed on the processed first image, and the color processing includes at least one of the following: color filling, color removal, and color retention.
The method according to claim 13, wherein the performing color processing on the processed first image according to the depth information of the object comprises:

The same color processing is performed on regions of the same depth in the processed first image.
The method according to claim 13 or 14, wherein the method further comprises:

obtaining a user's click operation on the processed first image;

determining the first depth information of the object at the point selected;

Color processing is performed on the processed first image according to the first depth information.
The method according to claim 15, wherein the performing color processing on the processed first image according to the first depth information comprises:

According to the first depth information, a target area is determined, wherein the target area is connected to the object at the point selected, and/or the difference between the depth of the target area and the first depth information is within the first preset range;

Color processing is performed on the target area.
The method according to claim 13 or 14, wherein the method further comprises:

Get the distance information entered by the user;

Color processing is performed on the processed first image according to the distance information.
The method according to claim 17, wherein the performing color processing on the processed first image according to the distance information comprises:

determining a target area according to the distance information, wherein the difference between the depth of the target area and the distance is within a second preset range;

Color processing is performed on the target area.
The method according to any one of claims 1-18, wherein the first pixel region in the first image is performed based on the second pixel region of the second image. Image processing to obtain the processed first image, including:

determining the edge of the second pixel area and the edge of the first pixel area;

According to the mapping relationship between the edge of the second pixel region and the edge of the first pixel region, the edge of the second pixel region is mapped to the edge of the first pixel region to generate the processed The image information of the edge region of the first image.
The method of claim 19, wherein the method further comprises:

extracting the edge of the second pixel area and the initial edge of the first pixel area;

Adjust parameters in the edge detection operator according to the edge density of the second pixel area and the initial edge density of the first pixel area;

Determine the edge of the first pixel area according to the parameters in the adjusted edge detection operator.
The method according to claim 20, wherein the adjusting parameters in the edge detection operator according to the edge density of the second pixel region and the initial edge density of the first pixel region, comprising:

In the case that the difference between the edge density of the second pixel area and the initial edge density of the first pixel area is not within a third preset range, adjusting the parameters in the edge detection operator;

The determining the edge of the first pixel region according to the parameters in the adjusted edge detection operator includes:

According to the parameters in the adjusted edge detection operator, re-extract the edge of the first pixel area until the difference between the edge density of the second pixel area and the edge density of the first pixel area within the third preset range.
The method according to any one of claims 19-21, wherein the second pixel area is mapped according to the mapping relationship between the edge of the second pixel area and the edge of the first pixel area. The edge of the pixel area is mapped to the edge of the first pixel area to generate image information of the edge area of the processed first image, including:

Determine a first edge segment and a second edge segment, the first edge segment is an edge segment on the edge of the first pixel area, and the second edge segment is an edge segment on the edge of the second pixel area ;

According to the mapping relationship between the first edge segment and the second edge segment, the second edge segment is mapped to the region where the first edge segment is located, so as to generate the processed image of the first image. Image information for edge regions.
The method according to claim 22, wherein the second edge segment is mapped to the first edge segment according to a mapping relationship between the first edge segment and the second edge segment where the image information of the edge region of the processed first image is generated, including:

segmenting the first edge segment to obtain a first line segment set;

segmenting the second edge to obtain a second line segment set;

determining the mapping relationship between the line segments in the first line segment set and the line segments in the second line segment set;

According to the mapping relationship between the line segments in the first line segment set and the line segments in the second line segment set, map the line segments in the second line segment set to where the line segments in the first line segment set are located position to generate image information of the edge region of the processed first image.
The method according to claim 23, wherein the segmenting the first edge segment to obtain the first line segment set comprises:

Segmenting the first edge segment to obtain a first initial line segment set;

The line segment whose length is less than the first preset value in the first initial line segment set is eliminated to obtain the first line segment set.
The method according to claim 23, wherein the segmenting the second edge segment to obtain the second line segment set comprises:

Segmenting the second edge segment to obtain a second initial line segment set;

The second line segment set is obtained by eliminating the line segments whose length is less than the second preset value in the second initial line segment set.
The method according to any one of claims 23-25, wherein the determining the mapping relationship between the line segments in the first line segment set and the line segments in the second line segment set comprises:

Determine the line segment in the first line segment set and the second line segment according to the matching degree between the pixel on the line segment in the first line segment set and the pixel on the line segment in the second line segment set The mapping relationship between the line segments in the collection.
The method according to any one of claims 19-26, wherein the first pixel region in the first image is performed based on the second pixel region of the second image. Image processing to obtain the processed first image, including:

According to the image information of the edge area of the processed first image, image information of the non-edge area of the processed first image is generated by interpolation.
An image processing device, comprising:

memory for storing computer programs;

a processor, configured to invoke the computer program, and when the computer program is executed by the processor, cause the apparatus to perform the following steps:

acquiring a first image captured by the first camera of the electronic device on the scene;

acquiring a second image obtained by photographing the scene by a second camera of the electronic device;

Wherein, the observation range of the first camera is larger than the observation range of the second camera; and the resolution of the first camera is lower than the resolution of the second camera, and/or the first camera The imaging of the second camera is an achromatic image, and the imaging of the second camera is a color image;

determining a first pixel area corresponding to the target object in the scene in the first image, and determining a second pixel area corresponding to the target object in the scene in the second image;

Based on the second pixel area in the second image, image processing is performed on the first pixel area in the first image to obtain a processed first image.
The apparatus according to claim 28, wherein the target objects are all or part of the scene objects in the scene captured by the second camera.
The apparatus according to claim 28 or 29, wherein the first pixel area and/or the second pixel area corresponding to the target object is determined based on a user operation.
The apparatus according to any one of claims 28-30, wherein when the computer program is executed by the processor, the apparatus is caused to perform the following steps:

acquiring a first initial image captured by the first camera of the scene;

Stabilization processing is performed on the first initial image to obtain the first image.
The apparatus of claim 31, wherein the first camera includes a first visual sensor and a second viewing angle sensor, and when the computer program is executed by the processor, the apparatus causes the apparatus to perform the following steps:

acquiring a first visual image captured by the first visual sensor of the scene;

acquiring a second visual image captured by the second visual sensor on the scene;

Based on the first visual image and the second visual image, the first initial image with depth information is generated.
The apparatus according to any one of claims 28-32, wherein when the computer program is executed by the processor, the apparatus is caused to perform the following steps:

acquiring a second initial image captured by the second camera on the scene;

Converting the second initial image to the camera pose of the first image to generate the second image.
The apparatus of claim 33, wherein when the computer program is executed by the processor, it causes the apparatus to perform the following steps:

Converting the second initial image to the first the camera pose of an image to generate the second image.
The apparatus according to any one of claims 28-34, wherein when the computer program is executed by the processor, the apparatus is caused to perform the following steps:

Receive instructions entered by the user;

According to the instruction input by the user, the shooting directions of the first camera and the second camera are adjusted, so that the first camera and the second camera shoot toward the same scene.
The apparatus according to any one of claims 28-35, wherein the electronic device is a drone, and the processed first image is used for FPV flight from a first-person main perspective.
The device according to any one of claims 28-36, wherein the first pixel area is located in a middle area of the first image.
The apparatus according to any one of claims 28-37, wherein the first image is an achromatic image, and the second image is a color image.
The apparatus of claim 38, wherein when the computer program is executed by the processor, it causes the apparatus to perform the following steps:

Using a superpixel segmentation device to segment the processed first image;

If a part of the same object after segmentation has color information and the other part has no color information, then another part of the same object is filled with color according to the part of the same object that has color information.
The apparatus according to claim 38 or 39, wherein the achromatic image includes depth information of an object, and when the computer program is executed by the processor, the apparatus causes the apparatus to perform the following steps:

According to the depth information of the object, color processing is performed on the processed first image, and the color processing includes at least one of the following: color filling, color removal, and color retention.
The apparatus of claim 40, wherein when the computer program is executed by the processor, the apparatus is caused to perform the following steps:

The same color processing is performed on regions of the same depth in the processed first image.
The apparatus according to claim 40 or 41, wherein when the computer program is executed by the processor, the apparatus is caused to perform the following steps:

obtaining a user's click operation on the processed first image;

determining the first depth information of the object at the point selected;

Color processing is performed on the processed first image according to the first depth information.
The apparatus of claim 42, wherein the computer program, when executed by the processor, causes the apparatus to perform the following steps:

According to the first depth information, a target area is determined, wherein the target area is connected to the object at the point selected, and/or the difference between the depth of the target area and the first depth information is within the first preset range;

Color processing is performed on the target area.
The apparatus according to claim 40 or 41, wherein when the computer program is executed by the processor, the apparatus is caused to perform the following steps:

Get the distance information entered by the user;

Color processing is performed on the processed first image according to the distance information.
The apparatus of claim 44, wherein, when executed by the processor, the computer program causes the apparatus to perform the following steps:

determining a target area according to the distance information, wherein the difference between the depth of the target area and the distance is within a second preset range;

Color processing is performed on the target area.
The apparatus according to any one of claims 28-45, wherein when the computer program is executed by the processor, the apparatus is caused to perform the following steps:

determining the edge of the second pixel area and the edge of the first pixel area;

According to the mapping relationship between the edge of the second pixel region and the edge of the first pixel region, the edge of the second pixel region is mapped to the edge of the first pixel region to generate the processed The image information of the edge region of the first image.
The apparatus of claim 46, wherein, when executed by the processor, the computer program causes the apparatus to perform the following steps:

extracting the edge of the second pixel area and the initial edge of the first pixel area;

Adjust parameters in the edge detection operator according to the edge density of the second pixel area and the initial edge density of the first pixel area;

Determine the edge of the first pixel area according to the parameters in the adjusted edge detection operator.
The apparatus of claim 47, wherein, when executed by the processor, the computer program causes the apparatus to perform the following steps:

In the case that the difference between the edge density of the second pixel area and the initial edge density of the first pixel area is not within a third preset range, adjusting the parameters in the edge detection operator;

The determining the edge of the first pixel region according to the parameters in the adjusted edge detection operator includes:

According to the parameters in the adjusted edge detection operator, the edge of the first pixel area is re-extracted until the difference between the edge density of the second pixel area and the edge density of the first pixel area within the third preset range.
The apparatus according to any one of claims 45-48, wherein when the computer program is executed by the processor, the apparatus is caused to perform the following steps:

Determine a first edge segment and a second edge segment, the first edge segment is an edge segment on the edge of the first pixel area, and the second edge segment is an edge segment on the edge of the second pixel area ;

According to the mapping relationship between the first edge segment and the second edge segment, the second edge segment is mapped to the region where the first edge segment is located, so as to generate the processed image of the first image. Image information for edge regions.
The apparatus of claim 49, wherein when the computer program is executed by the processor, it causes the apparatus to perform the following steps:

segmenting the first edge segment to obtain a first line segment set;

segmenting the second edge to obtain a second line segment set;

determining the mapping relationship between the line segments in the first line segment set and the line segments in the second line segment set;

According to the mapping relationship between the line segments in the first line segment set and the line segments in the second line segment set, map the line segments in the second line segment set to where the line segments in the first line segment set are located position to generate image information of the edge region of the processed first image.
The apparatus of claim 50, wherein when the computer program is executed by the processor, the apparatus is caused to perform the following steps:

Segmenting the first edge segment to obtain a first initial set of line segments;

The first line segment set is obtained by eliminating line segments whose length is less than a first preset value in the first initial line segment set.
The apparatus of claim 51, wherein when the computer program is executed by the processor, the apparatus is caused to perform the following steps:

segmenting the second edge segment to obtain a second initial line segment set;

Line segments whose lengths are less than a second preset value in the second initial line segment set are eliminated to obtain the second line segment set.
The apparatus according to any one of claims 50-52, wherein when the computer program is executed by the processor, the apparatus is caused to perform the following steps:

Determine the line segment in the first line segment set and the second line segment according to the matching degree between the pixel on the line segment in the first line segment set and the pixel on the line segment in the second line segment set The mapping relationship between the line segments in the collection.
The apparatus according to any one of claims 46-53, wherein when the computer program is executed by the processor, the apparatus is caused to perform the following steps:

According to the image information of the edge area of the processed first image, image information of the non-edge area of the processed first image is generated by interpolation.
A computer-readable storage medium, characterized in that a computer program is stored thereon, and when the computer program is executed, the method according to any one of claims 1 to 27 is implemented.