WO2022227394A1

WO2022227394A1 - Image processing method and apparatus, and device, storage medium and program

Info

Publication number: WO2022227394A1
Application number: PCT/CN2021/120905
Authority: WO
Inventors: 吴佳飞; 李亘杰; 张广程
Original assignee: 上海商汤智能科技有限公司
Priority date: 2021-04-28
Filing date: 2021-09-27
Publication date: 2022-11-03
Also published as: KR20220149514A; CN113139947A

Abstract

An image processing method and apparatus, and a device, a storage medium and a program. The method comprises: according to an image to be processed and a guide image of the image to be processed, determining linear estimation parameters of a plurality of image positions of the image to be processed (S11); according to the guide image and the linear estimation parameters of the plurality of image positions, obtaining a structure map of the image to be processed (S12); and fusing the structure map with a texture map of the image to be processed, so as to obtain an image processing result of the image to be processed (S13), wherein a linear estimation parameter of any image position in the image to be processed is associated with a regional variance value and a regional entropy value of a first image region in the guide image, and the first image region comprises an image region in the guide image, which region takes the image position as the center and has a preset size. By means of the method, an image processing effect can be improved.

Description

Image processing method, apparatus, equipment, storage medium and program

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application claims the priority of the Chinese patent application number 202110469110.2 filed on April 28, 2021, the applicant is Shanghai SenseTime Intelligent Technology Co., Ltd. The entirety of the application is incorporated by reference into this application.

technical field

The present disclosure relates to the field of computer vision technology, and in particular, to an image processing method, apparatus, device, storage medium, and program.

Background technique

In various image processing tasks and computer vision tasks (such as image dehazing, dark light enhancement, image stitching, etc.), it is often necessary to filter images to achieve corresponding processing. Among them, edge protection filtering is one of the most commonly used image filtering techniques.

At present, there are two methods of edge protection filtering: local filtering and global optimization. The global optimization has better processing effect, but the calculation complexity is too high and the efficiency is low. Difference.

SUMMARY OF THE INVENTION

The embodiment of the present disclosure proposes an image processing technical solution.

An embodiment of the present disclosure provides an image processing method, the method is executed by an electronic device, and the method includes:

determining linear estimation parameters of multiple image positions of the to-be-processed image according to the to-be-processed image and the guide image of the to-be-processed image;

obtaining a structure diagram of the to-be-processed image according to the guide image and the linear estimation parameters of the multiple image positions;

fusing the structure map with the texture map of the to-be-processed image to obtain an image processing result of the to-be-processed image,

Wherein, the linear estimation parameter of any image position in the image to be processed is associated with the regional variance value and the regional entropy value of the first image area in the guide image, and the first image area includes the guide image. In the image, the image area with the image position as the center and the preset size.

In some embodiments of the present disclosure, the determining linear estimation parameters of multiple image positions of the to-be-processed image according to the to-be-processed image and the guide image of the to-be-processed image includes: for the to-be-processed image For any image position, determine a first image area and a second image area corresponding to the image position, and the second image area includes an image of a preset size centered on the image position in the to-be-processed image area; respectively determine the area parameters of the first image area and the second image area, the area parameters include the area variance value, the area entropy value and the first area mean value of the pixels in the first image area , the second area mean value of the pixels in the second image area, and the third area mean value of the fusion area of the first image area and the second image area; Linear estimation parameters for each image location. In this way, the linear estimation parameter of the image position can be determined based on the area parameter of the guide image area corresponding to the image position.

In some embodiments of the present disclosure, the determining the area parameters of the first image area and the second image area respectively includes: fusing the first image area and the second image area to obtain a fusion area; determine the third area mean of the pixels in the fusion area. In this way, the regional mean values of the regions can be fused to realize the association between the guide image and the image to be processed.

In some embodiments of the present disclosure, the determining, respectively, the area parameters of the first image area and the second image area includes: determining, according to a luminance histogram of the guide image, a The occurrence probability of each pixel point in the guide image; the region entropy value is determined according to the occurrence probability of each pixel point in the first image region. In this way, the regional entropy value of the image region can be obtained for subsequent calculation of linear estimation parameters.

In some embodiments of the present disclosure, the linear estimation parameters include a first parameter and a second parameter, and the determining the linear estimation parameters for the plurality of image positions according to the region parameters includes: according to the region parameters difference value, the regional entropy value, the first regional average value, the second regional average value, and the third regional average value to determine the first parameter of the image position; according to the first regional average value, the The second area mean and the first parameter determine the second parameter of the image position. In this way, two linear estimation parameters of any image position can be determined respectively, and the linear estimation parameters are associated with the regional variance value and the regional entropy value, which can improve the image processing effect.

In some embodiments of the present disclosure, the method further includes: performing linear filtering on the to-be-processed image to obtain a guide image of the to-be-processed image. In this way, information such as noise in the image to be processed is filtered out in the guide image, so that the information in the image is smoother, so as to guide the subsequent processing of the image to be processed.

In some embodiments of the present disclosure, the method further includes: decomposing the image to be processed to obtain a texture map of the image to be processed. In this way, an edge protection filtering algorithm can be used to decompose the to-be-processed image into a base layer and a detail layer, and the detail layer can be used as the texture map of the to-be-processed image, so that the image processing result can be obtained by using the texture map subsequently.

The following descriptions of the effects of the apparatuses, electronic devices, etc. refer to the descriptions of the above-mentioned image processing methods.

Embodiments of the present disclosure provide an image processing apparatus, including:

a parameter determination module, configured to determine linear estimation parameters of multiple image positions of the to-be-processed image according to the to-be-processed image and a guide image of the to-be-processed image;

a structure diagram acquisition module, configured to obtain the structure diagram of the to-be-processed image according to the guide image and the linear estimation parameters of the positions of the multiple images;

A result determination module, configured to fuse the structure map with the texture map of the to-be-processed image to obtain an image processing result of the to-be-processed image,

In some embodiments of the present disclosure, the parameter determination module includes:

An area determination sub-module configured to determine, for any image position of the image to be processed, a first image area and a second image area corresponding to the image position, the second image area including the image to be processed , the image area with the image position as the center and the preset size;

a region parameter determination sub-module, configured to respectively determine the region parameters of the first image region and the second image region, the region parameters including the region variance value and the region entropy of the pixels in the first image region value and the first area mean value, the second area mean value of the pixels in the second image area, and the third area mean value of the fusion area of the first image area and the second image area;

The linear parameter determination submodule is configured to determine linear estimation parameters of the plurality of image positions according to the region parameters.

In some embodiments of the present disclosure, the region parameter determination submodule is configured to fuse the first image region and the second image region to obtain a fusion region; The average of the three regions.

In some embodiments of the present disclosure, the region parameter determination submodule is configured to determine, according to a brightness histogram of the guide image, the occurrence probability of each pixel in the first image region in the guide image; The region entropy value is determined according to the occurrence probability of each pixel in the first image region.

In some embodiments of the present disclosure, the linear estimation parameter includes a first parameter and a second parameter, and the linear parameter determination sub-module is configured to An area mean value, the second area mean value and the third area mean value determine the first parameter of the image position; according to the first area mean value, the second area mean value and the first parameter, determine the second parameter of the image location.

In some embodiments of the present disclosure, the apparatus further includes: a linear filtering module configured to perform linear filtering on the to-be-processed image to obtain a guide image of the to-be-processed image.

In some embodiments of the present disclosure, the apparatus further includes: an image decomposition module configured to decompose the to-be-processed image to obtain a texture map of the to-be-processed image.

An embodiment of the present disclosure provides an electronic device, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to invoke the instructions stored in the memory to execute the above method.

Embodiments of the present disclosure provide a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the foregoing method is implemented.

An embodiment of the present disclosure also provides a computer program, the computer program includes computer-readable codes, and when the computer-readable codes are executed in an electronic device, a processor of the electronic device executes the above method.

In the embodiment of the present disclosure, the linear estimation parameters can be determined according to the image to be processed and the guide image; the structure diagram of the image to be processed can be obtained according to the guide image and the linear estimation parameters; the image processing result can be obtained by fusing the structure diagram and the texture image. The estimated parameters are related to the regional variance value and the regional entropy value, which can better maintain the sharpness of the image edge and improve the image processing effect.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings.

Description of drawings

The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the technical solutions of the present disclosure.

FIG. 1 shows a flowchart of an image processing method according to an embodiment of the present disclosure;

FIG. 2 shows a schematic diagram of a system architecture to which an image processing method according to an embodiment of the present disclosure can be applied;

FIG. 3 shows a block diagram of an image processing apparatus according to an embodiment of the present disclosure;

FIG. 4 shows a block diagram of an electronic device according to an embodiment of the present disclosure;

FIG. 5 shows a block diagram of an electronic device according to an embodiment of the present disclosure.

Detailed ways

Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numbers in the figures denote elements that have the same or similar functions. While various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" in this article is only an association relationship to describe the associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, A and B exist at the same time, and A and B exist independently B these three cases. In addition, the term "at least one" herein refers to any combination of any one of the plurality or at least two of the plurality, for example, including at least one of A, B, and C, and may mean including from A, B, and C. Any one or more elements selected from the set of B and C.

Multiple or multiple in the embodiments of the present disclosure may refer to at least two or at least two, respectively.

In addition, in order to better illustrate the present disclosure, numerous specific details are set forth in the following detailed description. It will be understood by those skilled in the art that the present disclosure may be practiced without certain specific details. In some embodiments, methods, means, components and circuits well known to those skilled in the art have not been described in detail so as not to obscure the subject matter of the present disclosure.

Guided filtering is a local filtering algorithm of edge protection filtering technology. ) and a linear regression model are fitted, and regularization constraints are added to the linear regression model to reduce the overfitting problem. However, the weight of the regularization constraint is fixed, which is prone to artifacts in edge regions.

According to the image processing method of the embodiment of the present disclosure, the linear estimation parameter of the image position can be determined based on the regional variance value and the regional entropy value of the guide image region corresponding to the image position, and the automatic adjustment can be performed by the entropy-variance joint adaptive processing method. Regularize the weight of the constraint item, thereby reducing the possible artifacts in the edge area of the image, thereby improving the processing effect of the image.

FIG. 1 shows a flowchart of an image processing method according to an embodiment of the present disclosure. As shown in FIG. 1 , the image processing method includes:

In step S11, according to the to-be-processed image and the guide image of the to-be-processed image, linear estimation parameters of multiple image positions of the to-be-processed image are determined;

In step S12, a structure diagram of the to-be-processed image is obtained according to the guide image and the linear estimation parameters of the multiple image positions;

In step S13, the structure map and the texture map of the to-be-processed image are fused to obtain an image processing result of the to-be-processed image,

In some embodiments of the present disclosure, the image processing method may be executed by an electronic device such as a terminal device or a server, and the terminal device may be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless Phones, personal digital assistants (Personal Digital Assistants, PDAs), handheld devices, computing devices, vehicle-mounted devices, wearable devices, etc., the method can be implemented by the processor calling the computer-readable instructions stored in the memory. Alternatively, the method may be performed by a server.

In some embodiments of the present disclosure, the image to be processed may be any image, such as a scene image collected by an image acquisition device, an image downloaded from a network, and the like. The image processing tasks for the images to be processed can be of any category, such as image dehazing, dark light enhancement, contrast enhancement, tone mapping, high dynamic range HDR imaging, etc. Specific categories are not limited.

In some embodiments of the present disclosure, an edge protection filtering algorithm can be used to decompose the image to be processed into two layers, called a base layer and a detail layer, where the base layer includes the overall structural information of the image (such as the position and layout of objects in the image, etc. ), the detail layer includes the texture detail information of the image (for example, the texture, direction, etc. of the object in the image).

In this case, the image to be processed X(p) can be expressed as:

X(p)=Z(p)+D(p) Formula (1);

In formula (1), Z(p) can represent the base layer; D(p) can represent the detail layer; p can represent any image position in the image. In the case of image processing using guided filtering, the base layer can be estimated by linear transformation of the guided image:

In formula (2),

may represent the linear transformation estimate of the base layer, G(p) may represent the guide image; a _p' and b _p' may represent the linear estimated parameters of the linear regression model at the image position p.

In some embodiments of the present disclosure, the guide image is used to filter out target components in the image to be processed (eg, to filter out noise in the image to be processed). Among them, linear filtering (such as block filtering, mean filtering, or Gaussian filtering, etc.) can be performed on the image to be processed to obtain a guide image; it is also possible to select an image corresponding to the image processing task based on the category of the image processing task (for example, the image that has been darkened light-enhanced image) as a guide image; the image to be processed itself can also be used as a guide image. The present disclosure does not limit the source and processing method of the guide image.

In some embodiments of the present disclosure, in step S11, the linear estimation parameters at each image position of the image to be processed may be determined respectively according to the image to be processed and the guide image. where the linearly estimated parameters a _p' and b _p' for the image position p can be calculated by the following linear regression model:

In formula (3), Ωζ ₁ (p') can represent a rectangular window (for example, 3, 5, 7, 11, 17, etc.) with the image position p as the center and the side length ζ _1. The rectangular window Including K=ζ ₁ ×ζ ₁ image positions, p' represents any pixel in the rectangular window;

can represent regularization constraints used to prevent overfitting problems;

It can represent the weight of the regularization constraint, λ represents the regularization coefficient (for example, the value is 0.1, 0.01, etc.); Γ _G (p') can be called "entropy-variance joint adaptive factor", which is used to automatically adjust the regularization the size of the coefficient.

In some embodiments of the present disclosure, Γ _G (p') can be represented as:

In formula (4),

can represent the cross entropy-variance factor of the rectangular window Ωζ ₁ (p') in the guide image; N can represent the number of pixels in the guide image; i can represent any position in the guide image;

may represent the cross-entropy-variance factor of the rectangular window corresponding to position i in the guide image. According to formula (4), Γ _G (p') is

The normalized processing result.

In some embodiments of the present disclosure, the cross-entropy-variance factor

can be expressed as:

In formula (5),

It can represent the regional variance value (or called local variance value) of the pixel point within the rectangular window Ωζ ₁ (p') in the guide image;

It can represent the regional entropy value (or local entropy value) of the pixel points within the rectangular window Ωζ ₁ (p′) in the guide image.

Substituting formulas (4) and (5) into formula (3), the linear estimated parameters a _p' and b _p' of the image position p can be obtained, and the obtained linear estimated parameters a _p' and b _p' are the same as the guide The regional variance value and the regional entropy value of the rectangular window Ωζ ₁ (p′) in the image, which may be referred to as the first image region, are associated. The first image area includes an image area of a preset size (eg, 3×3, 5×5, etc.) centered on the image position p in the guide image.

For areas near the edge of the image, the area variance value is usually much larger than the area entropy value; for smooth areas inside the image (such as texture detail areas or background areas with noise), the area entropy value is usually larger than the area variance value value. Therefore, the factor Γ _G (p') for edge regions will be larger than the factor Γ _G (p') for smooth regions. In this case, halo artifacts near edges and over-smoothing on details will be reduced, resulting in better edge sharpness and improved image processing.

In some embodiments of the present disclosure, in step S12, a structure diagram of the to-be-processed image may be obtained according to the guide image and linear estimation parameters of multiple image positions. That is, after obtaining the linear estimation parameters a _p' and b _p' of the image position p, the linear estimation result of the image position p can be calculated according to formula (2). By separately processing each image position, a linear estimation result of the entire image to be processed can be obtained, that is, the base layer of the image to be processed. The base layer includes the overall structure information of the image, which can be called the structure diagram of the image to be processed.

In the foregoing description, an edge protection filtering algorithm can be used to decompose the image to be processed into a base layer and a detail layer. The detail layer includes texture detail information of the image, which may be called a texture map of the image to be processed.

In some embodiments of the present disclosure, in step S13, the structure map of the image to be processed and the texture map may be fused to obtain an image processing result of the image to be processed.

In some embodiments of the present disclosure, the structure map and the texture map may be directly summed to obtain a processed image as an image processing result.

In some embodiments of the present disclosure, different weights may be set for the structure map and the texture map according to the category of the image processing task (for example, in the image enhancement task, the weights of the structure map and the texture map are set to 1 and 1 respectively). 2, in order to achieve the enhancement of image texture details). Furthermore, the weighted summation is performed on the structure map and the texture map to obtain a processed image, which is used as the image processing result. The present disclosure does not limit the fusion method of the structure map and the texture map.

The entropy-variance joint adaptive factor introduced in the embodiment of the present disclosure can better distinguish the edge region of the image from the smooth region inside the image, and also has better anti-noise capability. Therefore, the image processing result obtained in step S13 can not only retain the detail information of the original image, but also reduce the halo artifacts near the edge of the image, and improve the sharpness of the edge of the image.

For example, when applied to image enhancement tasks such as contrast enhancement, the image processing result obtained in step S13 has better visual quality and quantitative performance.

According to the embodiments of the present disclosure, the linear estimation parameters can be determined according to the image to be processed and the guide image; the structure diagram of the image to be processed can be obtained according to the guide image and the linear estimation parameters; the image processing result can be obtained by fusing the structure diagram and the texture image. The estimated parameters are associated with the regional variance value and the regional entropy value, which can better maintain the sharpness of the image edge, thereby improving the image processing effect.

FIG. 2 shows a schematic diagram of a system architecture to which an image processing method according to an embodiment of the present disclosure can be applied; as shown in FIG. 2 , the system architecture includes an image acquisition terminal 201 , a network 202 and a processing terminal 203 . In order to support an exemplary application, the image acquisition terminal 201 and the processing terminal 203 establish a communication connection through the network 202, the image acquisition terminal 201 reports the image to be processed to the processing terminal 203 through the network 202, and the processing terminal 203 responds to the image to be processed. First, According to the to-be-processed image and the guide image of the to-be-processed image, the linear estimation parameters of the multiple image positions of the to-be-processed image are determined; secondly, the structure diagram of the to-be-processed image is obtained according to the guide image and the linear estimation parameters of the multiple image positions; then The structure map and the texture map of the image to be processed are fused to obtain an image processing result of the image to be processed. Finally, the processing terminal 203 uploads the image processing result to the network 202 and sends it to the image acquisition terminal 201 through the network 202 .

As an example, the image acquisition terminal 201 may include an image acquisition device, and the processing terminal 203 may include a visual processing device or a remote server with visual information processing capability. Network 202 may employ wired or wireless connections. Wherein, when the processing terminal 203 is a visual processing device, the image acquisition terminal 201 can communicate with the visual processing device through a wired connection, such as data communication through a bus; when the processing terminal 203 is a remote server, the image acquisition terminal 201 can Data exchange with remote server through wireless network.

Alternatively, in some scenarios, the image acquisition terminal 201 may be a vision processing device with a video acquisition module, or a host with a camera. At this time, the image processing method of the embodiment of the present disclosure may be executed by the image acquisition terminal 201 , and the above-mentioned system architecture may not include the network 202 and the control terminal 203 .

The image processing method according to the embodiment of the present disclosure will be described below.

In some embodiments of the present disclosure, before step S11, the image processing method according to the embodiment of the present disclosure may further include: performing linear filtering on the to-be-processed image to obtain a guide image of the to-be-processed image.

That is, the guide image of the image to be processed can be obtained by means of linear filtering, and the information such as noise in the image to be processed is filtered from the guide image, so that the information in the image is smoother, so as to guide the subsequent processing of the image to be processed.

The manner of linear filtering may be, for example, block filtering, mean filtering, Gaussian filtering, etc., which is not limited in the present disclosure.

In step S11, linear estimation parameters of multiple image positions of the to-be-processed image may be determined according to the to-be-processed image and the guide image. In some embodiments of the present disclosure, step S11 may include:

For any image position of the image to be processed, determine a first image area and a second image area corresponding to the image position, the second image area includes the image to be processed, and the image position is center, image area of preset size;

Determine the area parameters of the first image area and the second image area respectively, the area parameters include the area variance value, the area entropy value and the first area mean value of the pixels in the first image area, so the second area mean of the pixels in the second image area, and the third area mean of the fusion area of the first image area and the second image area;

Based on the region parameters, linear estimation parameters for the image position are determined.

In some embodiments of the present disclosure, for any image position in the image to be processed, a window area corresponding to the image position may be determined first, including a first image area and a second image area. The first image area includes an image area of a preset size centered on the image position in the guide image; the second image area includes an image area of a preset size centered on the image position in the image to be processed. The preset size may be set to, for example, 3×3, 5×3, 7×7, 11×11, 17×17, etc. The present disclosure does not limit the specific value of the preset size.

In some embodiments of the present disclosure, in the case where the image position is near the edge of the image, if the first image area and the second image area exceed the image range of the guide image and the image to be processed, the part beyond the image range may be Zero padding.

In some embodiments of the present disclosure, the area parameters of the first image area and the second image area may be determined separately in order to calculate the linear estimation parameters a _p' and b _p' of the image position p. Wherein, the area parameters include the area variance value, the area entropy value and the first area mean of the pixels in the first image area, the second area mean of the pixels in the second image area, and the first image area and the second area mean The third regional mean of the fused regions of the image regions.

In some embodiments of the present disclosure, the area variance value of the pixel points in the first image area, that is, in formula (5)

can be expressed as:

In formula (6),

Can represent the regional variance of pixels in the rectangular window _Ωζ1(p ' ₎ corresponding to the image position p in the guide image; N can represent the number of pixels in the guide image; j can represent any image position in the guide image;

It can represent the regional variance of the pixels in the rectangular window corresponding to the image position j in the guide image. According to formula (6), it can be known that,

that is

The normalized processing result.

In some embodiments of the present disclosure, the step of respectively determining the area parameters of the first image area and the second image area may include:

According to the brightness histogram of the guide image, determine the occurrence probability of each pixel in the first image area in the guide image;

The region entropy value is determined according to the occurrence probability of each pixel in the first image region.

Among them, the regional entropy value of the pixels in the first image region, that is, in formula (5)

can be expressed as:

In formula (7), K can represent the number of pixels in the first image area;

It can represent the occurrence probability of the kth pixel in the first image area.

In some embodiments of the present disclosure, the guide image may be processed to obtain a brightness histogram of the guide image, and the present disclosure does not limit the specific processing manner.

In some embodiments of the present disclosure, according to the brightness histogram, the brightness value of each pixel in the first image area can be determined, and the probability of occurrence in the guide image; and then the brightness value of the first image area can be calculated by formula (7). Area entropy value.

In this way, the regional entropy value of the image region can be obtained for subsequent calculation of linear estimation parameters.

In some embodiments of the present disclosure, the pixel values in the first image area and the pixel values in the second image area may be averaged respectively to obtain the first area mean value and the second area mean value in the area parameters.

Fusing the first image area and the second image area to obtain a fusion area;

A third area mean of the pixels in the fusion area is determined.

That is, the pixel values at the corresponding positions of the first image area and the second image area can be dot-multiplied to obtain a fusion area; the pixel values in the fusion area are averaged to obtain the third area mean value in the area parameters.

In this way, the regional mean values of the regions can be fused to realize the association between the guiding image and the image to be processed.

In some embodiments of the present disclosure, the linear estimation parameters include a first parameter a _p' and a second parameter b _p' . Wherein, the step of determining the linear estimation parameter of the image position according to the area parameter may include:

determining the first parameter of the image position according to the regional variance value, the regional entropy value, the first regional mean, the second regional mean and the third regional mean;

A second parameter of the image position is determined based on the first regional mean, the second regional mean, and the first parameter.

As shown above, by substituting formulas (4) to (6) into formula (3), the linear estimation parameters a _p' and b _p' of the image position can be obtained, which are expressed as:

In formulas (8) and (9),

can represent the mean value of the first region,

It can represent the mean value of the second area;

It can represent the third area mean; the operator ° can represent the dot product.

In some embodiments of the present disclosure, according to the regional variance value and the regional entropy value, the factor Γ _G (p') can be obtained based on formula (4); according to the factor Γ _G (p'), the first regional mean

second area mean

and the third regional mean

That is, the first parameter a _p' of the image position can be determined based on the formula (8); then, according to the first area mean

second area mean

and the first parameter a _p' , the second parameter b _p' of the image position can be determined based on the formula (9).

In this way, two linear estimation parameters of any image position can be determined respectively, and the linear estimation parameters are associated with the regional variance value and the regional entropy value, which can improve the image processing effect.

In some embodiments of the present disclosure, the above processing methods can be used to process each image position in the image to obtain linear estimation parameters of the entire image to be processed; in step S12, according to the guiding image and the linear estimation parameters, the Based on formula (2), the structure diagram of the image to be processed is obtained by calculation.

In some embodiments of the present disclosure, the image processing method according to the embodiments of the present disclosure may further include:

The to-be-processed image is decomposed to obtain a texture map of the to-be-processed image.

As mentioned above, an edge protection filtering algorithm can be used to decompose the image to be processed into a base layer and a detail layer, and the detail layer can be used as a texture map of the image to be processed.

In some embodiments of the present disclosure, the image to be processed may also be directly filtered to remove information such as noise in the image to be processed, and to retain detailed texture information in the image to be processed, to obtain a texture map of the image to be processed. The filtering method may be linear reducible filtering, and the present disclosure does not limit the specific filtering method.

In some embodiments of the present disclosure, after the texture map of the image to be processed is obtained, in step S13, the structure map of the image to be processed and the texture map may be fused to obtain an image processing result of the image to be processed.

In some embodiments of the present disclosure, the structure map and the texture map may be directly summed to obtain a processed image as an image processing result. Different weights can also be set for the structure map and texture map according to the category of the image processing task (for example, in the image enhancement task, the weights of the structure map and texture map are set to 1 and 2, respectively). The weighted summation of the structure map and the texture map is performed to obtain the processed image as the image processing result. The present disclosure does not limit the fusion method of the structure map and the texture map.

According to the image processing method of the embodiment of the present disclosure, an extended guided filtering method is proposed, which can determine the linear estimation parameter of the image position based on the regional variance value and the regional entropy value of the guided image region corresponding to the image position; - The variance joint adaptive processing method is used to automatically adjust the weight of the regularization constraint item of the linear estimation parameter, and the regularization weight can be adjusted to different degrees according to different image structures, thereby overcoming the fixed regularization weight in guided filtering. artifact problem.

The image processing method according to the embodiment of the present disclosure can be applied to the fields of artificial intelligence, image processing, machine vision, etc., to realize image processing such as image dehazing, dark light enhancement, contrast enhancement, tone mapping, high dynamic range HDR imaging, image stitching, etc. It can reduce the halo artifacts near the edge of the image and the over-smoothing problem on the details, and can better maintain the edge sharpness and improve the image processing effect.

It can be understood that the above-mentioned method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle and logic. Those skilled in the art can understand that, in the above method of the specific embodiment, the specific execution order of each step should be determined by its function and possible internal logic.

In addition, the present disclosure also provides image processing apparatuses, electronic devices, computer-readable storage media, and programs, all of which can be used to implement any image processing method provided by the present disclosure. For the corresponding technical solutions and descriptions, refer to the corresponding records in the Methods section. ,No longer.

FIG. 3 shows a block diagram of an image processing apparatus according to an embodiment of the present disclosure. As shown in FIG. 3 , the apparatus includes:

The parameter determination module 31 is configured to determine linear estimation parameters of multiple image positions of the to-be-processed image according to the to-be-processed image and the guide image of the to-be-processed image;

The structure diagram obtaining module 32 is configured to obtain the structure diagram of the to-be-processed image according to the guide image and the linear estimation parameters of the positions of the multiple images;

The result determination module 33 is configured to fuse the structure map with the texture map of the to-be-processed image to obtain an image processing result of the to-be-processed image,

In some embodiments of the present disclosure, the parameter determination module 31 includes:

In some embodiments, the functions or modules included in the apparatuses provided in the embodiments of the present disclosure may be used to execute the methods described in the above method embodiments. For specific implementation, reference may be made to the descriptions of the above method embodiments. For brevity, here No longer.

Embodiments of the present disclosure further provide a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the foregoing method is implemented. Computer-readable storage media can be volatile or non-volatile computer-readable storage media.

An embodiment of the present disclosure further provides an electronic device, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to invoke the instructions stored in the memory to execute the above method.

An embodiment of the present disclosure also provides a computer program, the computer program includes computer-readable codes, and when the computer-readable codes are executed in an electronic device, the processor of the electronic device executes the above method.

Embodiments of the present disclosure also provide another computer program product, including computer-readable codes, or a non-volatile computer-readable storage medium carrying computer-readable codes, when the computer-readable codes are processed in an electronic device When running in the device, the processor in the electronic device executes the above method.

The electronic device may be provided as a terminal, server or other form of device.

FIG. 4 shows a block diagram of an electronic device 800 according to an embodiment of the present disclosure. For example, electronic device 800 may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc. terminal.

4, an electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812 , sensor component 814 , and communication component 816 .

The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to perform all or some of the steps of the methods described above. Additionally, processing component 802 may include one or more modules that facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

Memory 804 is configured to store various types of data to support operation at electronic device 800 . Examples of such data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like. The memory 804 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (Electrically Erasable) Erasable Programmable Read Only Memory, EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (Read Only Memory, ROM) ), magnetic memory, flash memory, magnetic disk or optical disk.

Power supply assembly 806 provides power to various components of electronic device 800 . Power supply components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800 .

Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action. In some embodiments, the multimedia component 808 includes a front-facing camera and/or a rear-facing camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each of the front and rear cameras can be a fixed optical lens system or have focal length and optical zoom capability.

Audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a microphone (Microphone, MIC) configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a calling mode, a recording mode, and a voice recognition mode. The received audio signal may be further stored in memory 804 or transmitted via communication component 816 . In some embodiments of the present disclosure, the audio component 810 further includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, a button, or the like. These buttons may include, but are not limited to: home button, volume buttons, start button, and lock button.

Sensor assembly 814 includes one or more sensors for providing status assessment of various aspects of electronic device 800 . For example, the sensor assembly 814 can detect the on/off state of the electronic device 800, the relative positioning of the components, such as the display and the keypad of the electronic device 800, the sensor assembly 814 can also detect the electronic device 800 or one of the electronic device 800 Changes in the position of components, presence or absence of user contact with the electronic device 800 , orientation or acceleration/deceleration of the electronic device 800 and changes in the temperature of the electronic device 800 . Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. Sensor assembly 814 may also include a light sensor, such as a Complementary Metal Oxide Semiconductor (CMOS) or Charge-Coupled Device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (Wi-Fi), a second-generation mobile communication technology (2-Generation, 2G) or a third-generation mobile communication technology (3rd-Generation, 3G), or their combination. In one exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BitTorrent, BT) technology and other technology to achieve.

In an exemplary embodiment, the electronic device 800 may be implemented by one or more Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processor Device (Digital Signal Processor Device) , DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic component implementation, used to perform the above method.

In an exemplary embodiment, a non-volatile computer-readable storage medium, such as a memory 804 comprising computer program instructions executable by the processor 820 of the electronic device 800 to perform the above method is also provided.

FIG. 5 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. 5, the electronic device 1900 includes a processing component 1922, which may include one or more processors, and a memory resource, represented by memory 1932, for storing instructions, such as applications, executable by the processing component 1922. An application program stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Additionally, the processing component 1922 is configured to execute instructions to perform the above-described methods.

The electronic device 1900 may also include a power supply assembly 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an I/O interface 1958. The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as a Microsoft server operating system (Windows Server ^™ ), a graphical user interface based operating system (Mac OS X ^™ ) introduced by Apple, a multi-user multi-process computer operating system (Unix ^™ ), Free and Open Source Unix-like Operating System (Linux ^™ ), Open Source Unix-like Operating System (FreeBSD ^™ ) or the like.

In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as memory 1932 comprising computer program instructions executable by processing component 1922 of electronic device 1900 to perform the above-described method.

The present disclosure may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present disclosure.

A computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), ROM, EPROM or flash memory, SRAM, portable compact disk read only Memory (Compact Disc-Read Only Memory, CD-ROM), Digital Video Disc (DVD), memory sticks, floppy disks, mechanical coding devices, such as punched cards or recessed protrusions on which instructions are stored structure, and any suitable combination of the above. Computer-readable storage media, as used herein, are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.

The computer readable program instructions described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

Computer program instructions for carrying out the operations of the present disclosure may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or in one or Source or object code written in any combination of programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as the "C" language or similar programming languages The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or Wide Area Network (WAN), or it can be connected to an external A computer (eg, using an Internet service provider to connect through the Internet). In some embodiments, electronic circuits, such as programmable logic circuits, FPGAs, or programmable logic arrays, are personalized by utilizing state information of computer-readable program instructions ( Programmable Logic Arrays, PLA), electronic circuits that can execute computer-readable program instructions to implement various aspects of the present disclosure.

Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , resulting in means for implementing the functions/acts specified in one or more blocks of the flowchart and/or block diagrams. These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium storing the instructions includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more functions for implementing the specified logical function(s) executable instructions. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions.

The computer program product can be specifically implemented by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. Wait.

Various embodiments of the present disclosure have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the various embodiments, the practical application or improvement over the technology in the marketplace, or to enable others of ordinary skill in the art to understand the various embodiments disclosed herein.

Industrial Applicability

Embodiments of the present disclosure relate to an image processing method, apparatus, device, storage medium, and program. The method includes: determining multiple image positions of the to-be-processed image according to the to-be-processed image and a guide image of the to-be-processed image The linear estimation parameters of the image to be processed are obtained; the structure diagram of the image to be processed is obtained according to the linear estimation parameters of the guide image and the positions of the multiple images; the structure diagram and the texture map of the image to be processed are fused to obtain the The image processing result of the image to be processed, wherein the linear estimation parameter of any image position in the image to be processed is associated with the regional variance value and the regional entropy value of the first image region in the guide image, so The first image area includes an image area with a preset size centered on the image position in the guide image.

Claims

An image processing method, the method being performed by an electronic device, the method comprising:

determining linear estimation parameters of multiple image positions of the to-be-processed image according to the to-be-processed image and the guide image of the to-be-processed image;

obtaining a structure diagram of the to-be-processed image according to the guide image and the linear estimation parameters of the multiple image positions;

fusing the structure map with the texture map of the to-be-processed image to obtain an image processing result of the to-be-processed image,

Wherein, the linear estimation parameter of any image position in the image to be processed is associated with the regional variance value and the regional entropy value of the first image area in the guide image, and the first image area includes the guide image. In the image, the image area with the image position as the center and the preset size.
The method according to claim 1, wherein the determining linear estimation parameters of multiple image positions of the to-be-processed image according to the to-be-processed image and the guide image of the to-be-processed image comprises:

For any image position of the image to be processed, determine a first image area and a second image area corresponding to the image position, the second image area includes the image to be processed, and the image position is center, image area of preset size;

Determine the area parameters of the first image area and the second image area respectively, the area parameters include the area variance value, the area entropy value and the first area mean value of the pixels in the first image area, so the second area mean of the pixels in the second image area, and the third area mean of the fusion area of the first image area and the second image area;

Based on the region parameters, linear estimation parameters for the plurality of image locations are determined.
The method according to claim 2, wherein the determining the area parameters of the first image area and the second image area respectively comprises:

Fusing the first image area and the second image area to obtain a fusion area;

A third area mean of the pixels in the fusion area is determined.
The method according to claim 2, wherein the determining the area parameters of the first image area and the second image area respectively comprises:

According to the brightness histogram of the guide image, determine the occurrence probability of each pixel in the first image area in the guide image;

The region entropy value is determined according to the occurrence probability of each pixel in the first image region.
The method according to claim 2, wherein the linear estimation parameter includes a first parameter and a second parameter, and the determining the linear estimation parameter of the plurality of image positions according to the region parameter comprises:

determining the first parameter of the image position according to the regional variance value, the regional entropy value, the first regional mean, the second regional mean and the third regional mean;

A second parameter of the image position is determined based on the first regional mean, the second regional mean, and the first parameter.
The method according to any one of claims 1 to 5, wherein the method further comprises:

Linear filtering is performed on the to-be-processed image to obtain a guide image of the to-be-processed image.
The method according to any one of claims 1 to 6, wherein the method further comprises:

The to-be-processed image is decomposed to obtain a texture map of the to-be-processed image.
An image processing device, comprising:

A parameter determination module, configured to determine linear estimation parameters of multiple image positions of the to-be-processed image according to the to-be-processed image and the guide image of the to-be-processed image;

a structure diagram acquisition module, configured to obtain the structure diagram of the to-be-processed image according to the guide image and the linear estimation parameters of the positions of the multiple images;

A result determination module, configured to fuse the structure map with the texture map of the to-be-processed image to obtain an image processing result of the to-be-processed image,

Wherein, the linear estimation parameter of any image position in the image to be processed is associated with the regional variance value and the regional entropy value of the first image area in the guide image, and the first image area includes the guide image. In the image, the image area with the image position as the center and the preset size.
An electronic device comprising:

processor;

memory for storing processor-executable instructions;

wherein the processor is configured to invoke the memory-stored instructions to perform the method of any one of claims 1-7.
A computer-readable storage medium having computer program instructions stored thereon, the computer program instructions implementing the method of any one of claims 1 to 7 when executed by a processor.
A computer program comprising computer readable codes, when the computer readable codes are run in an electronic device, the processor of the electronic device executes the code for implementing any one of claims 1 to 7 The image processing method described in one item.