CN116168064A

CN116168064A - Image processing method, device, electronic equipment and storage medium

Info

Publication number: CN116168064A
Application number: CN202111417972.7A
Authority: CN
Inventors: 戴向东
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2021-11-25
Filing date: 2021-11-25
Publication date: 2023-05-26

Abstract

The embodiment of the application discloses an image processing method, an image processing device, electronic equipment and a storage medium, wherein the method comprises the following steps: matching the image content of the first image and the second image in the target area to obtain a matching relation corresponding to the target area; each frame of image in the first image and the second image comprises at least two areas, each area included in the first image corresponds to each area included in the second image, and the at least two areas are divided according to the depth value of the pixel point in each area; the target area is any one area of the at least two areas; and carrying out image registration on the first image and the second image according to the matching relation corresponding to the at least two areas respectively. By implementing the embodiment of the application, the accuracy of image registration can be improved.

Description

Image processing method, device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of image technologies, and in particular, to an image processing method, an image processing device, an electronic device, and a storage medium.

Background

Image registration may align one image with another image, which may be used for a variety of image processing such as video analysis, pattern recognition, object tracking, etc. However, in practice, it has been found that the current image registration method still has a problem of low accuracy.

Disclosure of Invention

The embodiment of the application discloses an image processing method, an image processing device, electronic equipment and a storage medium, which can improve the accuracy of image registration.

The embodiment of the application discloses an image processing method, which comprises the following steps: matching the image content of the first image and the second image in the target area to obtain a matching relation corresponding to the target area; each frame of image in the first image and the second image comprises at least two areas, each area included in the first image corresponds to each area included in the second image, and the at least two areas are divided according to the depth value of the pixel point in each area; the target area is any one area of the at least two areas; and carrying out image registration processing on the first image and the second image according to the matching relation corresponding to the at least two areas respectively. .

An embodiment of the application discloses an image processing apparatus, which is characterized by comprising: the matching module is used for matching the image content of the first image and the second image in the target area to obtain a matching relationship corresponding to the target area; each frame of image in the first image and the second image comprises at least two areas, each area included in the first image corresponds to each area included in the second image, and the at least two areas are divided according to the depth value of the pixel point in each area; the target area is any one area of the at least two areas; and the registration module is used for carrying out image registration processing on the first image and the second image according to the matching relation corresponding to the at least two areas respectively.

The embodiment of the application discloses an electronic device, which comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor realizes any one of the image processing methods disclosed in the embodiment of the application.

The embodiment of the application discloses a computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements any one of the image processing methods disclosed in the embodiments of the application.

Compared with the related art, the embodiment of the application has the following beneficial effects:

the electronic device may divide the first image and the second image into at least two areas according to the depth values, and the divided areas match image contents of the first image and the second image, so as to obtain matching relationships respectively corresponding to the areas. Wherein, the matching relation corresponding to each region can be used for registering the first image and the second image. The image content matching in the subareas can improve the matching accuracy, and is beneficial to improving the accuracy of image registration.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of an image processing circuit according to one embodiment of the disclosure;

FIG. 2 is a method flow diagram of an image processing method according to one embodiment of the disclosure;

FIG. 3A is an exemplary diagram of feature point matching of a foreground region as disclosed by one embodiment;

FIG. 3B is an exemplary diagram of feature point matching for a background region in accordance with one embodiment disclosure;

FIG. 3C is an exemplary diagram of full-graph feature point matching in the related art;

FIG. 4 is a method flow diagram of another image processing method disclosed in one embodiment;

FIG. 5A is a diagram illustrating a comparison of registration effects of a foreground region according to one embodiment of the disclosure;

FIG. 5B is a diagram illustrating a comparison of registration effects of a background region according to one embodiment of the disclosure;

FIG. 6 is an exemplary diagram of a person identification of a first image in accordance with one embodiment disclosure;

FIG. 7 is a method flow diagram of another image processing method disclosed in one embodiment;

fig. 8 is a schematic structural view of an image processing apparatus according to an embodiment of the present disclosure;

fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the disclosure.

Detailed Description

The following description of the technical solutions in the embodiments of the present application will be made clearly and completely with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.

It should be noted that the terms "comprising" and "having" and any variations thereof in the embodiments and figures herein are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed steps or elements but may include other steps or elements not listed or inherent to such process, method, article, or apparatus.

Image registration refers to a process of determining a translational and rotational relationship between two frames of images of the same subject, and spatially aligning the two frames of images based on the translational and rotational relationship.

In the related art, image registration is mainly performed by feature point matching of the whole image. That is, when the image registration scheme in the related art performs feature point matching, the whole image is taken as a search range, and feature points matched with each other are identified in two different images, so that a mapping matrix corresponding to a full image is estimated according to the identified feature point pairs, and mapping transformation is performed on all pixel points included in the image according to the mapping matrix, so as to realize registration of the two images.

The full-image feature point matching refers to a search area which is directly matched by taking the whole image as the feature point without distinguishing the image area where the feature point is in the process of matching the feature point. The registration method performed by the full-image feature point matching can achieve better effects in some scenes, but still has the problem of insufficient precision in other scenes.

For example, when the subject is a portrait, if the person to be photographed stands at a position far from the imaging device, the images obtained by photographing at this time are registered based on the matching of the feature points of the whole image, so that a more accurate registration effect can be obtained. If the photographer stands at a position closer to the imaging device, the captured image is registered based on the full-image feature point matching at this time, and the registration accuracy is lowered.

The embodiment of the application discloses an image processing method, an image processing device, electronic equipment and a storage medium, which can improve the accuracy of image registration. The following will describe in detail.

Referring to fig. 1, fig. 1 is a schematic diagram of an image processing circuit according to an embodiment of the disclosure. The image processing circuit can be applied to electronic equipment such as smart phones, smart tablets, smart watches and the like, but is not limited to the electronic equipment. As shown in fig. 1, the image processing circuit may include an imaging device (camera) 110, an attitude sensor 120, an image memory 130, an image signal processing (Image Signal Processing, ISP) processor 140, a logic controller 150, and a display 160.

The image processing circuitry includes an ISP processor 140 and control logic 150. Image data captured by imaging device 110 is first processed by ISP processor 140, where ISP processor 140 analyzes the image data to capture image statistics that may be used to determine one or more control parameters of imaging device 110. Imaging device 110 may include one or more lenses 112 and an image sensor 114. The image sensor 114 may include a color filter array (e.g., bayer filters), and the image sensor 114 may obtain light intensity and wavelength information captured by each imaging pixel and provide a set of RAW image data (RAW image data) that may be processed by the ISP processor 140. The attitude sensor 120 (e.g., tri-axis gyroscope, hall sensor, accelerometer, etc.) may provide acquired image processing parameters (e.g., anti-shake parameters) to the ISP processor 140 based on the type of attitude sensor 120 interface. The attitude sensor 120 interface may employ an SMIA (Standard Mobile Imaging Architecture ) interface, other serial or parallel camera interfaces, or a combination of the above.

In addition, the image sensor 114 may also send raw image data to the gesture sensor 120, the gesture sensor 120 may provide raw image data to the ISP processor 140 based on the gesture sensor 120 interface type, or the gesture sensor 120 may store raw image data in the image memory 130.

The ISP processor 140 processes the raw image data on a pixel-by-pixel basis in a variety of formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and ISP processor 140 may perform one or more image processing operations on the raw image data, collecting statistical information about the image data. Wherein the image processing operations may be performed with the same or different bit depth precision.

ISP processor 140 may also receive image data from image memory 130. For example, the gesture sensor 120 interface sends the raw image data to the image memory 130, where the raw image data in the image memory 130 is provided to the ISP processor 140 for processing. Image memory 130 may be part of a memory device, a storage device, or a separate dedicated memory within an electronic device, and may include DMA (Direct Memory Access ) features.

Upon receiving raw image data from the image sensor 114 interface or from the pose sensor 120 interface or from the image memory 130, the ISP processor 140 may perform one or more image processing operations, such as temporal filtering. The processed image data may be sent to image memory 130 for additional processing before being displayed. The ISP processor 140 receives the processed data from the image memory 130 and processes the processed data for image data in one or more color spaces, such as in the original domain and YUV, RGB, YCbCr. The image data processed by ISP processor 140 may be output to display 160 for viewing by a user and/or further processing by a graphics engine or GPU (Graphics Processing Unit, graphics processor). In addition, the output of ISP processor 140 may also be sent to image memory 130, and display 160 may read image data from image memory 130. In one embodiment, image memory 130 may be configured to implement one or more frame buffers.

The statistics determined by ISP processor 140 may be sent to control logic 150. For example, the statistics may include image sensor 114 statistics such as vibration frequency of gyroscope, auto-exposure, auto-white balance, auto-focus, flicker detection, black level compensation, lens 112 shading correction, etc. Control logic 150 may include a processor and/or microcontroller that executes one or more routines (e.g., firmware) that may determine control parameters of imaging device 110 and control parameters of ISP processor 140 based on the received statistics. For example, the control parameters of the imaging device 110 may include attitude sensor 120 control parameters (e.g., gain, integration time for exposure control, anti-shake parameters, etc.), camera flash control parameters, camera anti-shake displacement parameters, lens 112 control parameters (e.g., focal length for focusing or zooming), or a combination of these parameters. The ISP control parameters may include gain levels and color correction matrices for automatic white balancing and color adjustment (e.g., during YUV processing), as well as lens 112 shading correction parameters.

In one embodiment, ISP processor 140 may obtain two different frames of images, a first image and a second image, respectively, from imaging device 110. Wherein the first image and the second image may be two different images including the same photographic subject.

The first image and the second image may be two frames of images taken by the imaging device 110 continuously in time; alternatively, the first image and the second image may be two frames of images obtained by the imaging device 110 photographing the same object from different photographing angles; alternatively, the first image and the second image may be two frames of images obtained when the imaging device 110 performs tracking shooting on a certain moving object, which is not particularly limited.

After acquiring the first image and the second image, the ISP processor 140 may match image contents in the target area in the first image and the second image to obtain a matching relationship corresponding to the target area. Each frame of image in the first image and the second image comprises at least two areas, and the areas are divided according to the depth values of pixel points in each area; each region included in the first image corresponds to each region included in the second image, and the target region is any one of at least two regions.

After obtaining the matching relationship corresponding to each region, the ISP processor 140 may perform image registration processing on the first image and the second image according to the matching relationship corresponding to each region.

Referring to fig. 2, fig. 2 is a flow chart illustrating a method of image processing according to an embodiment of the present disclosure, and the method may be applied to any of the foregoing electronic devices. As shown in fig. 2, the method may include the steps of:

210. and matching the image contents of the first image and the second image in the target area to obtain a matching relationship corresponding to the target area.

The first image and the second image include the same subject, but the image coordinates of the same subject in the first image and the second image are different due to the difference in photographing time or photographing angle. Thus, in the first image and the second image, the same image content on different image coordinates may be image content that matches each other. The image content may include feature points or image blocks, and is not limited in particular.

Each frame of the first image and the second image may be divided into at least two regions based on the depth values of the pixels. The depth value corresponding to the pixel point can be used for representing the physical distance between the shooting object corresponding to the pixel point and the imaging device.

For example, the first image may include a foreground region and a background region, and a depth value corresponding to a pixel point in the foreground region may be smaller than a depth value corresponding to a pixel point in the background region. When the first image is an image obtained by photographing a portrait by the imaging device, the foreground region may be a portrait region in which the portrait is located, and the background region may be the rest of the image region other than the portrait region.

For example, the first image may also include a foreground region, a near region, and a far region, and the depth value corresponding to each pixel point in the foreground region may be smaller than the depth value corresponding to the near region, and the depth value corresponding to the near region may be smaller than the depth value corresponding to the far region. When the first image is an image obtained by photographing a portrait by the imaging device, the foreground region may be a portrait region in which the portrait is located, the near-view region may be a background plate set up when photographing the portrait, and the far-view region may be a forest behind the background plate.

Since the second image and the first image may include the same photographic subject, the second image may also be divided into at least two regions similarly to the first image, and the respective regions included in the second image may correspond to the respective regions included in the first image, respectively.

The aforementioned target region may be any one of at least two regions included in the first image. For example, the target region may be a foreground region or a background region. The electronic device may obtain a matching relationship corresponding to each target area after matching the image content for the target area.

For example, if the first image and the second image include a foreground region and a background region, respectively, referring to fig. 3A and 3B, fig. 3A is an exemplary diagram of feature point matching of a foreground region as disclosed in one embodiment, and fig. 3B is an exemplary diagram of feature point matching of a background region as disclosed in one embodiment.

If the target region is a foreground region, an example of performing pixel point matching in the foreground region of the first image and the second image may be as shown in fig. 3A, and the electronic device may identify a feature point pair 310 that matches each other in the foreground region of the first image and the second image. If the target area is a background area, an example of performing pixel matching in the background areas of the first image and the second image may be as shown in fig. 3B, and the electronic device may identify the feature point pair 320 that matches each other in the background areas of the first image and the second image.

In the embodiment of the present application, the electronic device may perform image content matching in different image areas, and obtain matching relationships corresponding to the different image areas. The matching relation corresponding to each region can comprise a plurality of characteristic point pairs, and each characteristic point pair can comprise two characteristic points matched with each other; alternatively, the matching relationship corresponding to each region may include a plurality of image block pairs, and each image block pair may include two image blocks that match each other, which is not limited in detail.

In the embodiment of the application, the image can be divided into areas in advance according to the depth values of the pixel points, and the image contents are respectively matched in different image areas, so that the pixel points with larger depth values in the first image are matched with the pixel points with larger depth values in the second image, and the pixel points with smaller depth values in the first image are matched with the pixel points with smaller depth values in the second image, thereby improving the matching accuracy of the image contents.

220. And performing image registration on the first image and the second image by using a matching relation corresponding to each of the at least two areas.

The matching relationship corresponding to at least two regions respectively can be used for image registration, and the matching relationship corresponding to each region can be used for indicating the offset of each region between two frames of images, namely the translation and rotation relationship of each region between two frames of images.

The electronic equipment can register the first image by utilizing the matching relation corresponding to each region respectively, so that the first image is aligned with the second image by taking the second image as a reference; or, the electronic device may register the second image by using the matching relationship corresponding to each region, so that the second image is aligned with the first image based on the first image, which is not limited in detail.

It will be appreciated that in a scene where the difference in image depth is large, the offset of the image regions corresponding to different depth values may also be different between two frames of images, and the difference is large. For example, when the imaging apparatus photographs at a fixed photographing angle, the moving of the portrait is made closer to the imaging apparatus, and the shift amount of the portrait region with a smaller depth value between two frame images is larger, but the shift amount of the background region with a smaller depth value between two frame images is smaller. If the offset is calculated only by a matching relationship corresponding to the full graph, the calculated offset may be greatly different from the offset actually corresponding to the different image areas, thereby reducing the accuracy of image registration.

In the embodiment of the application, the electronic device may calculate the offset of each region between two frames of images according to the matching relationship corresponding to each region divided based on the depth values, so as to perform image transformation processing on each image region based on the offset corresponding to each region, thereby improving the accuracy of image registration.

In one embodiment, the first image may be a reference image and the second image may be an image to be registered, the image to be registered being aligned with the reference image with respect to the reference image. The offset may be represented by a mapping matrix. Therefore, the electronic device can determine the mapping matrix corresponding to each region according to the matching relation corresponding to each region in the at least two regions; and performing image transformation processing on each image area in the second image by using a mapping matrix corresponding to each area so that the second image is aligned with the first image by taking the first image as a reference.

The mapping matrix corresponding to each region may include a homography matrix. Taking feature point matching as an example, if the electronic device identifies at least four sets of feature point pairs that are matched with each other in each region, a homography matrix may be calculated according to the identified feature point pairs.

It can be seen that after the image processing method disclosed in the embodiment of the present application is executed, the electronic device may obtain a plurality of matching relationships corresponding to different image areas, where each matching relationship corresponding to an area may include a plurality of feature point pairs that are matched with each other or a plurality of image blocks that are matched with each other. In the related art, if the feature point matching is performed based on the full graph without distinguishing the image areas, only one matching relationship corresponding to the full graph can be obtained. For example, referring to fig. 3C, fig. 3C is an exemplary diagram of full-graph feature point matching in the related art.

Therefore, in the embodiment of the present application, the electronic device may calculate two or more mapping matrices based on two or more different matching relationships, and only one mapping matrix corresponding to the full graph may be calculated based on the matching of the feature points of the full graph. The calculation of the mapping matrix of the sub-region can reduce the offset calculation error caused by the calculation of the mapping matrix based on the matching of the feature points of the full graph, and is beneficial to improving the registration accuracy of the registration image obtained after final registration.

Therefore, by implementing the image processing method disclosed by the embodiment of the application, the electronic equipment performs image content matching in different image areas and performs image registration according to the matching relations corresponding to the different image areas, so that the matching accuracy can be improved when the depth difference of the pixel points in the image is large, the accuracy of image registration based on the matching relations can be improved, and the image registration is more accurate. And moreover, matching errors of image contents can be compatible to a certain extent, and the influence of the matching errors on image registration is reduced.

In some embodiments, the electronic device may utilize the registered images to further perform one or more image processing operations such as image synthesis, background blurring, moving object tracking, and the like. The higher precision of the registered image is beneficial to improving the accuracy of a series of image processing operations based on the registered image, for example, the definition of image synthesis can be improved.

In some embodiments, each frame of image in the first image and the second image may be divided into two regions, namely, a foreground region and a background region, and the image processing method disclosed in the embodiments of the present application will be described based on the following.

Referring to fig. 4, fig. 4 is a schematic flow chart of another image processing method disclosed in an embodiment, and the method can be applied to any of the foregoing electronic devices. As shown in fig. 4, the method may include the steps of:

410. and matching the image content of the first image and the second image in the foreground region, and determining a first mapping matrix corresponding to the foreground region according to the matching relation corresponding to the foreground region.

420. And matching the image contents of the first image and the second image in the background area, and determining a second mapping matrix corresponding to the background area according to the matching relation corresponding to the background area.

The embodiment of matching the image content in the foreground region and the background region can be referred to as the embodiment of the foregoing embodiment step 210, and the following description will be omitted.

It should be noted that, the foregoing

steps

410 and 420 do not have a necessarily sequential relationship logically. In other embodiments, the electronic device may perform step 420 first and then perform step 410; alternatively, the electronic device may perform step 410 and step 420 simultaneously.

430. And multiplying the first image coordinates of the foreground region in the second image with the first mapping matrix to obtain the foreground region of the registered image.

The result of the multiplication of the first image coordinates and the first mapping matrix may be the image coordinates of the foreground region in the registered image.

440. And multiplying the second image coordinates of the background area in the second image with a second mapping matrix to obtain the background area of the registered image.

The result of the multiplication of the second image coordinates and the second mapping matrix may be the image coordinates of the background area in the registered image.

It can be seen that the aforementioned registration image may be obtained by performing an image transformation process on the second image, and the registration image is aligned with the first image with reference to the first image.

It should be noted that, step 430 may be performed after step 410, and step 440 may be performed after step 420; step 430 and step 440 do not have a logical relationship, and step 440 may be performed before step 430 or simultaneously.

Therefore, the electronic device can respectively match the image contents of the foreground region and the background region, and perform image transformation on the second image to be registered based on the matching relationship respectively corresponding to the foreground region and the background region, so that the transformed registered image can be accurately and highly accurately aligned with the first image.

Referring to fig. 5A, fig. 5A is a diagram illustrating a comparison of registration effects of a foreground region according to one embodiment. As shown in fig. 5A, the first image 510 is used as a reference image, the second image is registered based on a registration method of full-image matching to obtain a registered image 520, and the second image is registered based on the image processing method disclosed in the foregoing embodiment to obtain a registered image 530.

As shown in fig. 5A, a partial image is taken at the same position of the foreground region in the first image 510, the registration image 520, and the registration image 530, resulting in a foreground region 510a in the first image 510, a foreground region 520a in the registration image 520, and a foreground region 530a in the registration image 530.

As shown in fig. 5A, there is a more significant difference between the image position of the pixel i in the foreground region 510a and the image position of the pixel i in the foreground region 520 a; and the image position of pixel i in the foreground region 510a tends to coincide with the image position of pixel i in the foreground region 530 a.

Referring to fig. 5B, fig. 5B is a diagram illustrating a comparison of registration effects of a background area according to an embodiment. The first image 510, the registration image 520, and the registration image 530 in fig. 5B are the same as those in fig. 5A, and the following description will be omitted.

As shown in fig. 5B, a partial image is taken at the same position of the background areas of the first image 510, the registration image 520, and the registration image 530, resulting in a background area 510B in the first image 510, a background area 520B in the registration image 520, and a background area 530B in the registration image 530.

As shown in fig. 5B, there is a more significant difference between the image position of the pixel j in the background area 510B and the image position of the pixel j in the background area 520B; and the image position of the pixel j in the background area 510b tends to coincide with the image position of the pixel j in the background area 530b.

The higher the accuracy of image registration, the more closely the image locations of the mutually matched pixels on the different images are consistent. Therefore, by implementing the image processing method disclosed by the embodiment of the application, the accuracy of image registration can be improved.

In one embodiment, the electronic device may divide the image areas in the first image and the second image prior to performing step 310. It should be noted that, although the image area is divided according to the depth values of the pixels in each area, the electronic device does not necessarily need to perform depth estimation on the captured image to restore the depth values of the pixels. The following provides two alternative ways of dividing the image area, but is not limited thereto.

Further, since the respective areas included in the first image and the second image correspond to each other, the electronic apparatus may perform operations shown in the following embodiments on the first image and the second image, respectively, and may perform operations shown in the following embodiments on only one of the images, and may perform the operations shown in the following embodiments on the other image, according to the divided image areas.

The following describes two image area division modes using the first image as an example.

As an alternative implementation manner, the first image and the second image may include a portrait, and the electronic device may perform portrait identification on the first image to obtain a portrait area where the portrait is located. In general, in an image of a subject photographed with a portrait, the portrait is often a foreground region. Accordingly, the electronic device may determine a portrait region of the first image as a foreground region of the first image, and determine remaining image regions of the first image other than the portrait region as a background region of the first image.

The electronic device may perform portrait identification through portrait identification algorithms based on feature extraction or based on machine learning portrait Matting (Matting), portrait segmentation, and the like, which is not particularly limited.

Referring to fig. 6, for exemplary purposes, fig. 6 is an exemplary diagram of a person identification of a first image according to one embodiment. As shown in fig. 6, after the first image 610 is subjected to the portrait matting, a portrait matting result 620 corresponding to the first image 610 may be obtained. The image matting result 620 may be a binary image, and in the image matting result 620, a pixel point (black) with a pixel value of 0 may be used to indicate a portrait area, and a pixel point (white) with a pixel value of 1 may be used to indicate a background area.

The embodiment of determining the foreground region and the background region in the second image based on the portrait identification may refer to the embodiment of determining the foreground region and the background region in the first image based on the portrait identification, and will not be described in detail.

As can be seen, in the embodiment of the present application, the foreground regions of the first image and the second image may be portrait regions determined by portrait identification; the first image is subjected to portrait identification, so that a foreground area of the first image can be determined; the image recognition of the second image may determine a foreground region of the second image.

Accordingly, the background areas of the first image and the second image may be image areas other than portrait areas. The background area of the first image may be an image area of the first image other than the portrait area; the background area of the second image may be an image area of the second image other than the portrait area.

As another alternative embodiment, the electronic device may perform depth estimation on the image to divide the image area according to the restored depth values.

The electronic device can perform monocular depth estimation on the first image through a machine learning algorithm such as a convolutional neural Network (Convolutional Neural Network, network), a U-shaped Network (U-Net) and the like; alternatively, the electronic device may perform binocular depth estimation using the first image and the second image, and recover the depth value of each pixel point in the first image by means of parallax between the first image and the second image.

The electronic device may identify pixels in the first image having a depth value less than a first depth threshold as pixels of the foreground region, to obtain the foreground region of the first image. The first depth threshold may be set with reference to an actual service requirement, and is not specifically limited. Or,

The electronic device may also identify, as pixels of the background area, pixels in the first image having a depth value greater than or equal to the second depth threshold, to obtain the background area of the first image. The second depth threshold may also be set with reference to an actual service requirement, and the second depth threshold may be the same as the first depth threshold, or the second depth threshold may also be greater than the first depth threshold, which is not specifically limited.

Optionally, the electronic device may identify a foreground region of the first image according to the first depth threshold; after the foreground region is identified, the rest of the image regions except the foreground region in the first image may be determined as the background region, or the background region of the first image may be identified according to the second depth threshold.

Optionally, the electronic device may identify a background area of the first image according to the second depth threshold; after the background region is identified, the rest of the image regions except the background region in the first image may be determined as the foreground region, or the foreground region of the first image may be identified according to the first depth threshold.

The embodiment of determining the foreground region and the background region in the second image based on the depth estimation may refer to the embodiment of determining the foreground region and the background region in the first image by the depth estimation, which will not be described in detail below.

It can be seen that the depth values of the pixels in the first image and the second image can be obtained by depth estimation. The foreground regions of the first image and the second image may be image regions in which the depth value of the pixel point is less than a first depth threshold; and/or, the background areas of the first image and the second image may be image areas where the depth value of the pixel point is smaller than the second depth threshold.

It can be seen that in the foregoing embodiment, the electronic device may perform image region division on the first image and the second image by means of image recognition or depth estimation. The image area dividing method based on portrait identification can rapidly distinguish the foreground area and the background area of the image, reduce the calculated amount and improve the algorithm instantaneity. The image dividing method based on the depth estimation can accurately distinguish the foreground area and the background area of the image when the shooting object of the image is a non-human object, and is beneficial to expanding the applicable scene of the image processing method disclosed by the embodiment of the application.

In the image processing method disclosed in the embodiment of the present application, the electronic device matches the image content in the target area, which may include matching of feature points or matching of image blocks. The following description will be made separately.

In one embodiment, the electronic device may perform feature point matching in the target areas of the first image and the second image, and may include the steps of:

the electronic equipment extracts the characteristic points of the first image and the second image respectively so as to extract the first characteristic points of the first image and the second characteristic points of the second image. The feature point extraction may be performed in the first image and the second image, without distinguishing the image areas.

For example, the electronic device may perform feature point extraction on the first image and the second image through a feature point detection algorithm such as a quick selection key point algorithm (Features from Accelerated Segments Test, FAST), a Scale-invariant feature transform algorithm (Scale-Invariant Feature Transform, SIFT), or the like.

After the electronic device performs region division on the first image and the second image based on any one of the image region division modes, the electronic device may identify a third feature point located in the first image target region from the first feature points, and identify a fourth feature point located in the second image target region from the second feature points. And matching the third characteristic point with the fourth characteristic point to obtain a characteristic point matching relation corresponding to the target area. In addition, the electronic equipment can synchronously perform feature point matching and image region division, so that time consumption can be reduced, and the image registration efficiency can be improved.

For example, the electronic device may describe the third feature point and the fourth feature point by a BRIEF (Binary Robust Independent Elementary Features) feature description algorithm. The matching of feature points may be converted into computing a Hamming (Hamming) distance between two feature descriptors, where the Hamming distance between feature descriptors corresponding to two feature points that match each other is minimal.

The feature point matching algorithm that combines FAST and BRIEF is commonly referred to as the ORB (Oriented Fast and Rotated Brief) feature matching algorithm. The ORB algorithm has the advantages of being fast, free of noise and image transformation to a certain extent, and the like. On the basis of image content matching in the subareas, the ORB feature matching algorithm is combined to match the image feature points, so that the accuracy of image content matching can be further improved, and the accuracy of image registration is further improved.

Referring to fig. 7, fig. 7 is a schematic flow chart of another image processing method according to an embodiment. As shown in fig. 7, the first image 710 and the second image 720 may be subjected to ORB-based feature point extraction, respectively, resulting in a first feature point 730 of the first image 710 and a second feature point 740 of the second image 720.

At the same time, the electronic device may perform a portrait matting on the first image 710 to obtain a portrait matting result 750. The portrait matting result 750 may be used to distinguish foreground and background regions in the first and second images.

The electronic device may identify a third feature point 730a belonging to the foreground region and a third feature point 730b belonging to the background region in combination with the first feature point 730 and the portrait matting result 750. And, the electronic device may identify a fourth feature point 740a belonging to the foreground region and a fourth feature point 750b belonging to the background region in combination with the second feature point 740 and the portrait matting result 750.

The electronic device matches the third feature point 730a belonging to the foreground region with the fourth feature point 740a belonging to the foreground region to obtain a feature point matching relationship corresponding to the foreground region; the third feature point 730b belonging to the background area and the fourth feature point 740b belonging to the background area are matched to obtain a feature point matching relationship corresponding to the background area.

The electronic device may further calculate a first mapping matrix 760 corresponding to the foreground region according to the feature point matching relationship corresponding to the foreground region, and calculate a second mapping matrix 770 corresponding to the background region according to the feature point matching relationship corresponding to the background region. The first mapping matrix 760 is multiplied by the first image coordinates of the foreground region of the second image and the second mapping matrix 770 is multiplied by the second image coordinates of the background region of the second image to obtain a registered image 780.

In some embodiments, there may be fewer feature points in the first image and the second image. For example, when the texture features of an image are small, the image includes a large number of patches of pure colors, it is easy to bring difficulty to extraction and matching of feature points. In order to solve the problem caused by too few feature points, the electronic device may perform image block matching in the target areas of the first image and the second image, and may include the following steps:

the electronic equipment respectively performs image blocking in a target area of the first image to obtain a plurality of first image blocks in the target area of the first image and a plurality of second image blocks in the target area of the second image; and matching the first image block with the second image block to obtain an image block matching relation corresponding to the target area.

The electronic device may divide the target area of the first image into a plurality of first image blocks of the same size, and divide the target area of the second image into a plurality of second image blocks of the same size. The size of the first image block and the size of the second image block may also be the same.

After the image is segmented, the electronic device may match each of the first image block and the second image block based on a square difference metric or the like, where the square difference between the first image block and the second image block that match each other is the smallest.

The image content is matched based on image segmentation, so that a more accurate matching relationship can be calculated under a scene with fewer characteristic points, and the problem of reduced image registration accuracy caused by the reduction of the characteristic points can be solved.

Referring to fig. 8, fig. 8 is a schematic structural diagram of an image processing apparatus according to an embodiment, and the image processing apparatus may be applied to any of the foregoing electronic devices. As shown in fig. 8, the image processing apparatus 800 may include: a matching module 810 and a registration module 820.

The matching module 810 is configured to match image contents of the first image and the second image in the target area, so as to obtain a matching relationship corresponding to the target area; each frame of image in the first image and the second image comprises at least two areas, each area included in the first image corresponds to each area included in the second image, and the at least two areas are divided according to the depth values of pixel points in each area; the target area is any one of at least two areas;

the registration module 820 may be configured to perform image registration processing on the first image and the second image according to a matching relationship corresponding to at least two regions respectively.

In one embodiment, registration module 820 may include: matrix calculation unit and transformation unit.

The matrix calculation unit is used for determining a mapping matrix corresponding to each region according to the matching relation corresponding to each region in at least two regions;

the transformation unit is used for respectively carrying out image transformation processing on each region in the second image by utilizing a mapping matrix corresponding to each region to obtain a registration image; the registration image is aligned with the first image with reference to the first image.

In one embodiment, the at least two regions include a foreground region and a background region, wherein a depth value corresponding to a pixel in the foreground region is smaller than a depth value corresponding to a pixel in the background region; a mapping matrix corresponding to each of at least two regions, comprising: a first mapping matrix corresponding to the foreground region; and a second mapping matrix corresponding to the background region;

the transformation unit is further used for multiplying the first image coordinates of the foreground region in the second image with the first mapping matrix to obtain a foreground region of the registered image; and multiplying the second image coordinates of the background area in the second image with the second mapping matrix to obtain the background area of the registered image.

In one embodiment, the image processing apparatus 800 may further include: and a determining module.

The determining module may be configured to perform image recognition on the first image before the matching module 810 matches the image content of the first image and the second image in the target area to obtain a matching relationship corresponding to the target area, so as to obtain a portrait area where the portrait is located; and determining the portrait region as a foreground region of the first image, and determining the rest of the image regions except the portrait region as a background region of the first image.

In one embodiment, the foregoing determining module may be further configured to perform depth estimation on the first image to obtain a depth value of each pixel point in the first image before the matching module 810 matches the image contents of the first image and the second image in the target area to obtain a matching relationship corresponding to the target area; and identifying the pixel points with the depth values of the pixel points in the first image smaller than the first depth threshold as the pixel points of the foreground region to obtain the foreground region of the first image; and/or identifying the pixel points with the depth values of the pixel points in the first image being greater than or equal to the second depth threshold as the pixel points of the background area, and obtaining the background area of the first image.

In one embodiment, the image content includes: feature points; the matching module 810 may include: an extraction unit and a matching unit.

The extraction unit is used for extracting the characteristic points of the first image and the second image respectively so as to extract the first characteristic points of the first image and the second characteristic points of the second image; and identifying a third feature point located in the first image target area from the first feature points, and identifying a fourth feature point located in the second image target area from the second feature points;

and the matching unit can be used for matching the third characteristic point with the fourth characteristic point to obtain a characteristic point matching relation corresponding to the target area.

In one embodiment, the image content includes: an image block; the foregoing matching module 810 may further include: and a blocking unit.

The block dividing unit can be used for dividing the image in the target area of the first image respectively to obtain a plurality of first image blocks in the target area of the first image and a plurality of second image blocks in the target area of the second image;

and the matching unit is also used for matching the first image block and the second image block to obtain an image block matching relation corresponding to the target area.

Therefore, by implementing the image processing device disclosed by the embodiment of the application, the image content can be respectively matched in different image areas, and when the depth difference of the pixel points in the image is large, the matching accuracy is improved, so that the accuracy of image registration based on the matching relationship can be improved, and the image registration is more accurate. And moreover, matching errors of image contents can be compatible to a certain extent, and the influence of the matching errors on image registration is reduced.

Referring to fig. 9, fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the disclosure. As shown in fig. 9, the electronic device 900 may include:

a memory 910 storing executable program code;

a processor 920 coupled with the memory 910;

the processor 920 invokes executable program codes stored in the memory 910 to implement any one of the image processing methods disclosed in the embodiments of the present application.

It should be noted that, the mobile terminal shown in fig. 9 may further include components that are not shown, such as a power supply, an input key, a camera, a speaker, a screen, an RF circuit, a Wi-Fi module, a bluetooth module, and a sensor, which are not described in detail in this embodiment.

The embodiment of the application discloses a computer readable storage medium storing a computer program, wherein the computer program, when executed by a processor, causes the processor to implement any one of the image processing methods disclosed in the embodiments of the application.

The embodiments of the present application disclose a computer program product comprising a non-transitory computer readable storage medium storing a computer program, which when executed by a processor implements any of the image processing methods disclosed in the embodiments of the present application.

It should be appreciated that reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. Those skilled in the art will also appreciate that the embodiments described in the specification are all alternative embodiments and that the acts and modules referred to are not necessarily required in the present application.

In various embodiments of the present application, it should be understood that the size of the sequence numbers of the above processes does not mean that the execution sequence of the processes is necessarily sequential, and the execution sequence of the processes should be determined by the functions and internal logic thereof, and should not constitute any limitation on the implementation process of the embodiments of the present application.

The units described above as separate components may or may not be physically separate, and components shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.

In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units described above, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer-accessible memory. Based on such understanding, the technical solution of the present application, or a part contributing to the prior art or all or part of the technical solution, may be embodied in the form of a software product stored in a memory, including several requests for a computer device (which may be a personal computer, a server or a network device, etc., in particular may be a processor in the computer device) to perform part or all of the steps of the above-mentioned method of the various embodiments of the present application.

Those of ordinary skill in the art will appreciate that all or part of the steps of the various methods of the above embodiments may be implemented by a program that instructs associated hardware, the program may be stored in a computer readable storage medium including Read-Only Memory (ROM), random access Memory (Random Access Memory, RAM), programmable Read-Only Memory (Programmable Read-Only Memory, PROM), erasable programmable Read-Only Memory (Erasable Programmable Read Only Memory, EPROM), one-time programmable Read-Only Memory (OTPROM), electrically erasable programmable Read-Only Memory (EEPROM), compact disc Read-Only Memory (Compact Disc Read-Only Memory, CD-ROM) or other optical disk Memory, magnetic disk Memory, tape Memory, or any other medium that can be used for carrying or storing data that is readable by a computer.

The foregoing describes in detail an image processing method, apparatus, electronic device and storage medium disclosed in the embodiments of the present application, and specific examples are applied herein to illustrate the principles and embodiments of the present application, where the foregoing examples are only used to help understand the method and core idea of the present application. Meanwhile, as those skilled in the art will have modifications in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.

Claims

1. An image processing method, the method comprising:

matching the image content of the first image and the second image in the target area to obtain a matching relation corresponding to the target area; each frame of image in the first image and the second image comprises at least two areas, each area included in the first image corresponds to each area included in the second image, and the at least two areas are divided according to the depth value of the pixel point in each area; the target area is any one area of the at least two areas;

and carrying out image registration on the first image and the second image according to the matching relation corresponding to the at least two areas respectively.

2. The method according to claim 1, wherein the performing image registration on the first image and the second image according to the matching relationship corresponding to the at least two regions respectively includes:

determining a mapping matrix corresponding to each region according to the matching relation corresponding to each region in the at least two regions;

respectively carrying out image transformation processing on each region in the second image by using a mapping matrix corresponding to each region to obtain a registration image; the registration image is aligned with the first image with reference to the first image.

3. The method of claim 1 or 2, at least two regions of the first and second images each comprising a foreground region and a background region, the depth value of a pixel within the foreground region being less than the depth value of a pixel within the background region.

4. A method according to claim 3, wherein the foreground region of the first image and the second image is a portrait region determined by portrait identification; the background areas of the first image and the second image are image areas other than portrait areas.

5. A method according to claim 3, wherein the foreground region of the first and second images is an image region having pixel points with depth values less than a first depth threshold; and/or the background areas of the first image and the second image are image areas with depth values of pixel points greater than or equal to a second depth threshold;

the depth values of the pixel points in the first image and the second image are obtained through depth estimation.

6. The method of any one of claims 1 to 2, 4 to 5, wherein the image content comprises: feature points; the matching the image content of the first image and the second image in the target area to obtain a matching relationship corresponding to the target area includes:

Extracting feature points of a first image and a second image respectively to extract the first feature points of the first image and the second feature points of the second image;

identifying a third characteristic point in the first image target area from the first characteristic points, and identifying a fourth characteristic point in the second image target area from the second characteristic points;

and matching the third characteristic point with the fourth characteristic point to obtain a characteristic point matching relation corresponding to the target area.

7. The method of any one of claims 1 to 2, 4 to 5, wherein the image content comprises: an image block; the matching the image content of the first image and the second image in the target area to obtain a matching relationship corresponding to the target area includes:

image blocking is carried out in the target area of the first image respectively to obtain a plurality of first image blocks in the target area of the first image and a plurality of second image blocks in the target area of the second image;

and matching the first image block with the second image block to obtain an image block matching relation corresponding to the target area.

8. An image processing apparatus, comprising:

the matching module is used for matching the image content of the first image and the second image in the target area to obtain a matching relationship corresponding to the target area; each frame of image in the first image and the second image comprises at least two areas, each area included in the first image corresponds to each area included in the second image, and the at least two areas are divided according to the depth value of the pixel point in each area; the target area is any one area of the at least two areas;

and the registration module is used for carrying out image registration processing on the first image and the second image according to the matching relation corresponding to the at least two areas respectively.

9. An electronic device comprising a memory and a processor, the memory having stored therein a computer program which, when executed by the processor, causes the processor to implement the method of any of claims 1 to 7.

10. A computer readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, implements the method according to any one of claims 1 to 7.